Test Management for Jira: Zephyr vs TestFLO

Introducing a dedicated test management tool into the company marks a milestone in the process of improving the testing area of the software development life cycle (SDLC). But the multitude of available solutions makes that choice quite challenging. If we use Jira in the area of ​​project management, we can expand its functionality with dedicated apps for test management.

Implementing the testing process in Jira brings numerous benefits. You can read more about them here. Once you decide to use an app for Jira, you’ll have at least a few interesting options to choose from. In this article, we compare two such tools: TestFLO and Zephyr; a clash of David with Goliath.

How we developed our comparison

The purpose of this comparison is to analyze both tools in the context of their test management capabilities. To this end, we created a comparative table with 94 points broken down into 11 categories. It includes popular functionalities in the field of test management and omits the functions specific to a given solution.

When designing the comparison, we adopted the following assumptions:

• We take into account only popular features that are important for the user
• We assess each category separately
• Points marked with ⚠ indicate partial or limited functionality and are treated as uncredited
• We analyze solutions with their extensions (for example, TestFLO Automation or ZAPI)

You can download the full comparison table here.

Onboarding

When starting our work with a new tool, the entry threshold becomes an important aspect. It refers to how quickly we are able to learn to navigate the tool efficiently. Zephyr is very good in this respect because we can start testing in just a few minutes after installation and without reading the documentation. The app doesn’t require any initial configuration and is ready to use immediately.

TestFLO also provides default settings that allow users to run the tool quickly. But to make the most of it, we need to carry out a more detailed configuration. The add-on also offers a lot of mechanisms that can seem a little overwhelming at first glance.

Test Design

Test design is one of the basic steps in the test process. Testers create many test scenarios and must be able to do it efficiently. That’s why testers appreciate TestFLO’s features such as reusable preconditions, group test creation, or a folder tree for categorizing the created tests.

Zephyr doesn’t perform so well in this area. The tool doesn’t allow adding preconditions, not to mention reusing them. We also can’t group test steps or import steps from external files. An interesting mechanism here is the reusability of test steps, which allows importing them from another test.

Test Planning

Having created our tests, we want to plan their performance. In this aspect, the tools present similar possibilities but through completely different mechanisms. In Zephyr, test cycles are used for this. They are provided by the app and can be categorized in the folder structure. Planning is based on the versioning mechanism in Jira.

In the case of TestFLO, we deal with a new issue type, the Test Plan. Thanks to this approach, we can use Jira functionalities such as workflow, custom fields, and many others. Test Plans can be marked using the version from a Jira project, but it’s not necessary. In addition, Test Cases accumulated on the Test Plan can be grouped, and that’s something Zephyr doesn’t allow.

Test execution

Performing tests is an activity that takes a lot of time for testers, so the tool should provide support in this area as well. TestFLO and Zephyr have similar capabilities when performing tests, but the former re-implements the execution in the form of a Test Case issue, while the latter has its own object for this purpose.

In TestFLO, it’s much more convenient to use statuses on test steps, because changing the status requires only one click, and we can also set statuses for all steps or a group of steps with a single action. Zephyr only allows changing the status of each step individually.

Reporting

A good test management tool needs to include solid reporting capabilities. TestFLO provides six built-in reports based on the Fix Version/s field or saved JQL filters, which makes them flexible. In addition, the implementation of test objects as issues gives you the opportunity to use the Issue Navigator, as well as built-in Jira gadgets on dashboards for reporting purposes.

Zephyr offers only one report: the Traceability report. Also, we can use the built-in dashboard which summarizes the tests carried out throughout the project. When creating your own dashboard, you can use dedicated gadgets. However, keep in mind that these gadgets are useful because we can’t use Jira’s standard gadgets to report on test executions and cycles.

Requirement management

Requirements form the foundation for writing test scenarios. That’s why it’s important to support them with test management tools. TestFLO allows defining the types of issues that serve as requirements for a given project with tests. Such requirements gain new panels with all the defect issues and test objects, as well as the possibility of using such operations as test creation or Test Plan from the level of the requirement.

Zephyr offers limited support in this regard. The requirement can be any issue; defining this element is impossible. This means users have an additional test creation action on every issue type in the project. It’s also not possible to associate a requirement with a test cycle or create a cycle from the requirement level.

Traceability

If we want easily available information about the relationships between test objects, requirements, and defects, we should consider TestFLO. The app offers high visibility between almost every object that participates in the testing process. Flexible panels and various custom fields provide transparent information.

The same cannot be said for Zephyr, where users have only basic information about the relationship between, for example, a test and its execution, or execution and a defect. Unfortunately, users also gain very little information about the relationship with the requirements, because they can only trace the relationship between the requirement and the test.

Customization

Sometimes the test process offered by the app out-of-the-box may not meet our needs fully. Then the tool’s configuration options come into play. In this area, TestFLO is the undisputed leader, allowing users to modify workflows, implement optional fields, configure the Steps field, or create their own flexible panels. And that’s only a fraction of all the app’s possibilities. The mere use of Jira issues as test objects offers a huge scope for configuration.

Zephyr strongly focuses on the plug’n’play approach and built-in continuous testing process. The configuration is basic and limited to defining statuses or additional fields on the test executions.

Permissions and restrictions

Permission support is another strong point of TestFLO. The app supports Jira permission schemes but also introduces its own permissions module. Besides, we can use additional conditions and validators to model the workflow better. This support of permissions and configurability are of great value for implementing advanced test processes.

Zephyr offers fewer options. Although it implements its own permission as an extension of Jira’s permission schemes, it doesn’t support Jira’s standard permission schemes. That way, a user who, for example, doesn’t have the permission to edit the issue can change the test steps on it.

Integrations & migrations

In terms of integration with other tools, Zephyr offers a powerful REST API called ZAPI. It introduces many useful methods to use in your scripts for testing purposes. It’s worth mentioning the tool’s integration with Confluence, which is used by a dedicated app. A big downside here is that you can’t export tests.

TestFLO provides a REST API, but also extends Jira’s API to support the fields provided by the app. Users can easily export the tests and contents of the Steps field to a file. Unfortunately, integration with Confluence is limited as not all TestFLO fields are supported, and there are no dedicated macros.

Test automation

Automatic testing support is offered at a similar level in TestFLO and Zephyr. Both apps integrate with Jenkins and Bamboo, although they use different methods for accomplishing that. Zephyr uses add-ons for these CIs, while TestFLO provides a dedicated JAR for sending tests to Jira.

The biggest difference is the ability to support XML file formats. In the case of Zephyr, we can use TestNG, NUnit, or JUnit, while TestFLO only supports the last format. An interesting option available in TestFLO is the ability to run the build directly from Jira, after which the test results are automatically imported.

Pricing

This is not the most important feature of the software, but it can sometimes make a difference. TestFLO is much more favorable here in every Jira user tier compared to Zephyr. The table below shows the price list of each app with its paid extensions.

Key takeaway

You will find at least a few good tools for test management in Jira on the Atlassian Marketplace. Since their selection isn’t easy, we compared two tools: TestFLO and Zephyr. To do this reliably, we designed a detailed comparison table with functions that are important from the point of view of test management.

So here’s our conclusion:

Zephyr

The comparison showed that Zephyr is a tool that is easier to implement. It has a built-in, permanent test process and is intuitive enough to enable users to start using it without referring to the documentation. A noteworthy feature of Zephyr is the extensive REST API in the form of ZAPI and good integration with Confluence through the extension Zephyr Blueprints for Confluence. 

Unfortunately, the weaknesses of this tool lie in the basic areas of test management. Zephyr doesn’t offer a dedicated solution for saving preconditions or structuring tests. Reporting is mainly based on dashboards, and the tool offers only one built-in report. Configuration options are also very limited.

TestFLO

TestFLO, on the other hand, is a tool that has a fairly high entry threshold. Users need to invest some time in designing their processes and use the tool effectively. We could say the same about Jira. TestFLO supports the entire test management process by introducing the folder structure for tests, the functional Steps field, and six useful reports. Attention should also be paid to the extensive support of requirements and transparency regarding the relations between various objects. 

The biggest advantage of the tool, however, is its configurability. TestFLO allows designing various and even the most advanced test processes. The implementation of test objects in the form of issues, the use of custom fields, optional post functions, and workflow conditions, as well as the built-in authorization module and authorization support in Jira, make TestFLO’s customization possibilities really outstanding.