Test Management for Jira: Zephyr vs TestFLO

TestFLO

5 November 2019 • 30 min read

    Introducing a dedicated test management tool into the company marks a milestone in the process of improving the testing area of the software development life cycle (SDLC). But the multitude of available solutions makes that choice quite challenging. If we use Jira in the area of ​​project management, we can expand its functionality with dedicated apps for test management.

    Implementing the testing process in Jira brings numerous benefits. You can read more about them here. Once you decide to use an app for Jira, you’ll have at least a few interesting options to choose from. In this article, we compare two such tools: TestFLO and Zephyr; a clash of David with Goliath.

    How we developed our comparison

    The purpose of this comparison is to analyze both tools in the context of their test management capabilities. To this end, we created a comparative table with 94 points broken down into 11 categories. It includes popular functionalities in the field of test management and omits the functions specific to a given solution.

    When designing the comparison, we adopted the following assumptions:

    • We take into account only popular features that are important for the user
    • We assess each category separately
    • Points marked with ⚠ indicate partial or limited functionality and are treated as uncredited
    • We analyze solutions with their extensions (for example, TestFLO Automation or ZAPI)

    You can download the full comparison table here.

    Onboarding

    When starting our work with a new tool, the entry threshold becomes an important aspect. It refers to how quickly we are able to learn to navigate the tool efficiently. Zephyr is very good in this respect because we can start testing in just a few minutes after installation and without reading the documentation. The app doesn’t require any initial configuration and is ready to use immediately.

    TestFLO also provides default settings that allow users to run the tool quickly. But to make the most of it, we need to carry out a more detailed configuration. The add-on also offers a lot of mechanisms that can seem a little overwhelming at first glance.

    Test Design

    Test design is one of the basic steps in the test process. Testers create many test scenarios and must be able to do it efficiently. That’s why testers appreciate TestFLO’s features such as reusable preconditions, group test creation, or a folder tree for categorizing the created tests.

    Zephyr doesn’t perform so well in this area. The tool doesn’t allow adding preconditions, not to mention reusing them. We also can’t group test steps or import steps from external files. An interesting mechanism here is the reusability of test steps, which allows importing them from another test.

    Zephyr TestFLO
    Test Design 33% 75%
    Preconditions
    Reusable preconditions
    Reusable steps
    Create groups of steps
    Clone steps
    Create test sets
    Create tests in bulk
    Tests import
    Steps import
    Test versioning
    Folders for tests
    Clone Tests

    Test Planning

    Having created our tests, we want to plan their performance. In this aspect, the tools present similar possibilities but through completely different mechanisms. In Zephyr, test cycles are used for this. They are provided by the app and can be categorized in the folder structure. Planning is based on the versioning mechanism in Jira.

    In the case of TestFLO, we deal with a new issue type, the Test Plan. Thanks to this approach, we can use Jira functionalities such as workflow, custom fields, and many others. Test Plans can be marked using the version from a Jira project, but it’s not necessary. In addition, Test Cases accumulated on the Test Plan can be grouped, and that’s something Zephyr doesn’t allow.

    Zephyr TestFLO
    Test Planning 83% 83%
    Reusable tests
    Create groups of tests
    Set the order of tests
    Copy test plans/cycles
    Assign tests to users
    Folders for test plans/cycles

    Test execution

    Performing tests is an activity that takes a lot of time for testers, so the tool should provide support in this area as well. TestFLO and Zephyr have similar capabilities when performing tests, but the former re-implements the execution in the form of a Test Case issue, while the latter has its own object for this purpose.

    In TestFLO, it’s much more convenient to use statuses on test steps, because changing the status requires only one click, and we can also set statuses for all steps or a group of steps with a single action. Zephyr only allows changing the status of each step individually.

    Zephyr TestFLO
    Test Execution 70% 80%
    Change steps’ statuses in bulk
    Change statuses for a group of steps
    Create defects from test executions
    Create defects from steps
    Link existing defects to test executions
    Link existing defects to steps
    Add comments to steps
    Testing in cycles/iterations
    Registering test execution time
    Ad-hoc test execution

    Reporting

    A good test management tool needs to include solid reporting capabilities. TestFLO provides six built-in reports based on the Fix Version/s field or saved JQL filters, which makes them flexible. In addition, the implementation of test objects as issues gives you the opportunity to use the Issue Navigator, as well as built-in Jira gadgets on dashboards for reporting purposes.

    Zephyr offers only one report: the Traceability report. Also, we can use the built-in dashboard which summarizes the tests carried out throughout the project. When creating your own dashboard, you can use dedicated gadgets. However, keep in mind that these gadgets are useful because we can’t use Jira’s standard gadgets to report on test executions and cycles.

    Zephyr TestFLO
    Reporting 50% 67%
    Built-in reports
    Flexible filtering in reports
    Jira gadgets  ✘
    Issue Navigator reporting (reporting fields)
    Printable export (PDF, HTML etc.)
    Built-in dashboard

    Requirement management

    Requirements form the foundation for writing test scenarios. That’s why it’s important to support them with test management tools. TestFLO allows defining the types of issues that serve as requirements for a given project with tests. Such requirements gain new panels with all the defect issues and test objects, as well as the possibility of using such operations as test creation or Test Plan from the level of the requirement.

    Zephyr offers limited support in this regard. The requirement can be any issue; defining this element is impossible. This means users have an additional test creation action on every issue type in the project. It’s also not possible to associate a requirement with a test cycle or create a cycle from the requirement level.

    Zephyr TestFLO
    Requirement Management 33% 67%
    Built-in requirement issue types
    Requirement issue types setting
    Folders for requirements
    Link requirements with tests
    Link requirements with test plans/cycles
    Create tests from requirements
    Create test plans/cycles from requirements
    Import requirements from Word files
    Import requirements from CSV files

    Traceability

    If we want easily available information about the relationships between test objects, requirements, and defects, we should consider TestFLO. The app offers high visibility between almost every object that participates in the testing process. Flexible panels and various custom fields provide transparent information.

    The same cannot be said for Zephyr, where users have only basic information about the relationship between, for example, a test and its execution, or execution and a defect. Unfortunately, users also gain very little information about the relationship with the requirements, because they can only trace the relationship between the requirement and the test.

    Zephyr TestFLO
    Traceability 50% 90%
    Requirements and tests
    Requirements and test plans/cycles
    Requirements and test executions
    Requirements and defects
    Tests and test executions
    Tests and test plans/cycles
    Tests and defects
    Test plans/cycles and test executions
    Test plans/cycles and defects
    Test executions and defects

    Customization

    Sometimes the test process offered by the app out-of-the-box may not meet our needs fully. Then the tool’s configuration options come into play. In this area, TestFLO is the undisputed leader, allowing users to modify workflows, implement optional fields, configure the Steps field, or create their own flexible panels. And that’s only a fraction of all the app’s possibilities. The mere use of Jira issues as test objects offers a huge scope for configuration.

    Zephyr strongly focuses on the plug’n’play approach and built-in continuous testing process. The configuration is basic and limited to defining statuses or additional fields on the test executions.

    Zephyr TestFLO
    Customization 27% 100%
    Configurable workflows
    Jira workflow post functions support
    Additional workflow post functions
    Steps columns configuration
    Steps statuses configuration
    Test statuses configuration
    Customizable issue panels
    Additional custom fields
    Additional test plan setting
    Per project configuration
    Inactive test statuses setting

    Permissions and restrictions

    Permission support is another strong point of TestFLO. The app supports Jira permission schemes but also introduces its own permissions module. Besides, we can use additional conditions and validators to model the workflow better. This support of permissions and configurability are of great value for implementing advanced test processes.

    Zephyr offers fewer options. Although it implements its own permission as an extension of Jira’s permission schemes, it doesn’t support Jira’s standard permission schemes. That way, a user who, for example, doesn’t have the permission to edit the issue can change the test steps on it.

    Zephyr TestFLO
    Permissions & restrictions 29% 100%
    Jira workflow conditions support
    Additional workflow conditions
    Jira workflow validators support
    Additional workflow validators
    Workflow properties support
    Built-in permissions module
    Jira permission schemes support

    Integrations & migrations

    In terms of integration with other tools, Zephyr offers a powerful REST API called ZAPI. It introduces many useful methods to use in your scripts for testing purposes. It’s worth mentioning the tool’s integration with Confluence, which is used by a dedicated app. A big downside here is that you can’t export tests.

    TestFLO provides a REST API, but also extends Jira’s API to support the fields provided by the app. Users can easily export the tests and contents of the Steps field to a file. Unfortunately, integration with Confluence is limited as not all TestFLO fields are supported, and there are no dedicated macros.

    Zephyr TestFLO
    Integrations & migrations 57% 57%
    Built-in REST API
    Jira REST API support
    Confluence integration
    Tests export
    Steps export
    Test plans/cycles export
    Test executions export

    Test automation

    Automatic testing support is offered at a similar level in TestFLO and Zephyr. Both apps integrate with Jenkins and Bamboo, although they use different methods for accomplishing that. Zephyr uses add-ons for these CIs, while TestFLO provides a dedicated JAR for sending tests to Jira.

    The biggest difference is the ability to support XML file formats. In the case of Zephyr, we can use TestNG, NUnit, or JUnit, while TestFLO only supports the last format. An interesting option available in TestFLO is the ability to run the build directly from Jira, after which the test results are automatically imported.

    Zephyr TestFLO
    Test Automation 63% 63%
    Integration with Jenkins
    Integration with Bamboo
    Run a build directly from Jira
    Linking test executions with builds
    JUnit XML importing
    TestNG XML importing
    NUnit XML importing
    Cucumber support

    Pricing

    This is not the most important feature of the software, but it can sometimes make a difference. TestFLO is much more favorable here in every Jira user tier compared to Zephyr. The table below shows the price list of each app with its paid extensions.

    Zephyr + Zephyr Blueprints for Confluence TestFLO + TestFLO Automation
    25 users 1000$ 350$
    50 users 2000$ 900$
    100 users 4000$ 1700$
    250 users 8000$ 3400$
    500 users 11000$ 5600$
    2000 users 16000$ 8700$
    10000 users 20000$ 10800$
    Unlimited 26000$ 16000$

    Key takeaway

    You will find at least a few good tools for test management in Jira on the Atlassian Marketplace. Since their selection isn’t easy, we compared two tools: TestFLO and Zephyr. To do this reliably, we designed a detailed comparison table with functions that are important from the point of view of test management.

    So here’s our conclusion:

    Zephyr

    The comparison showed that Zephyr is a tool that is easier to implement. It has a built-in, permanent test process and is intuitive enough to enable users to start using it without referring to the documentation. A noteworthy feature of Zephyr is the extensive REST API in the form of ZAPI and good integration with Confluence through the extension Zephyr Blueprints for Confluence. 

    Unfortunately, the weaknesses of this tool lie in the basic areas of test management. Zephyr doesn’t offer a dedicated solution for saving preconditions or structuring tests. Reporting is mainly based on dashboards, and the tool offers only one built-in report. Configuration options are also very limited.

    TestFLO

    TestFLO, on the other hand, is a tool that has a fairly high entry threshold. Users need to invest some time in designing their processes and use the tool effectively. We could say the same about Jira. TestFLO supports the entire test management process by introducing the folder structure for tests, the functional Steps field, and six useful reports. Attention should also be paid to the extensive support of requirements and transparency regarding the relations between various objects. 

    The biggest advantage of the tool, however, is its configurability. TestFLO allows designing various and even the most advanced test processes. The implementation of test objects in the form of issues, the use of custom fields, optional post functions, and workflow conditions, as well as the built-in authorization module and authorization support in Jira, make TestFLO’s customization possibilities really outstanding.

    [contact-form-7 404 "Not Found"]