Test Management for Jira: Zephyr vs TestFLO
5 November 2019 • 23 min read
Updated on July 6th, 2021 by Tatiana Glazkova
Introducing a dedicated test management tool into the company marks a milestone in the process of improving the testing area of the software development life cycle (SDLC). But the multitude of available solutions makes that choice quite challenging. If we use Jira in the area of project management, we can expand its functionality with dedicated apps for test management.
Implementing the testing process in Jira brings numerous benefits. You can read more about them here. Once you decide to use an app for Jira, you’ll come across a few interesting options to choose from. In this article, let’s compare two possible options, such as TestFLO – Test Management for Jira and Zephyr Squad – Test Management for Jira.
Issues to compare
We’d like to analyze both applications in the context of their test management capabilities. Below, you’ll find comparative tables with 94 points separated into 11 categories. It includes popular functionalities in the field of test management and omits the functions specific to a given solution.
When making a comparison, we adopted the following assumptions:
- we take into account only popular features that are important for a user;
- we assess each category separately;
- points marked with ⚠ indicate partial or limited functionality and are treated as uncredited;
- we analyze solutions with their test automation module.
You can download the full comparison table here.
Onboarding
When starting our work with a new tool, the entry threshold becomes an important aspect. It refers to how quickly we are able to learn to navigate the tool efficiently. We can start testing in just a few minutes after installation and without reading the documentation in Zephyr Squad. The app doesn’t require any initial configuration and is ready to be used immediately, in case your team is familiar with Jira.
TestFLO also provides default settings that allow users to run the tool quickly. After installation, we can select a project in which we automatically enable TestFLO with default components. If necessary, we can carry out a more detailed configuration. There are a lot of options in the app to make your testing processes highly customized.
Test Design
Test design is one of the basic steps in the test process. Testers create many test scenarios and must be able to do it efficiently.
View of Test Steps (edit mode) in TestFLO
This is one of the reasons why testers appreciate TestFLO’s features such as reusable preconditions, group test creation, or a folder tree for categorizing the created tests.
View of Test Steps (edit mode) in Zephyr
Zephyr Squad doesn’t perform so well in this area. The tool doesn’t allow adding preconditions, not to mention reusing them. We also can’t group test steps or import steps from external files. An interesting mechanism here is the reusability of test steps, which allows importing them from another test.
Test Planning
Having created our tests, we want to plan their performance. In this aspect, the tools present similar possibilities but through completely different mechanisms. In Zephyr Squad, there are test cycles that are provided by the app and can be categorized in the folder structure. Planning is based on the versioning mechanism in Jira.
In the case of TestFLO, we deal with a new issue type, the Test Plan. Thanks to this approach, we can use Jira functionalities such as workflow, custom fields, and many others. Test Plans can be marked using the version from a Jira project, but it’s not necessary. In addition, Test Cases accumulated on the Test Plan can be grouped, and that’s something Zephyr Squad doesn’t allow.
Test execution
Performing tests is an activity that takes a lot of time for testers, so the tool should provide support in this area as well. TestFLO and Zephyr Squad have similar capabilities when performing tests, but the former re-implements the execution in the form of a Test Case issue, while the latter has its own object for this purpose.
In TestFLO, it’s much more convenient to use statuses on test steps, because changing the status requires only one click, and we can also set statuses for all steps or a group of steps with a single action. Zephyr Squad only allows changing the status of each step individually.
Reporting
A good test management tool needs to include solid reporting capabilities. TestFLO provides six built-in reports based on the Requirements, Test Plans, Test Case statuses, and the progress of Test Case steps. You can filter data using the Fix Version/s field, JQL queries and saved filters, which makes them flexible. In addition, the implementation of test objects as issues gives you the opportunity to use the Issue Navigator, as well as built-in Jira gadgets on dashboards for reporting purposes.
Requirement Traceability Report in TestFLO
Zephyr Squad offers four reports based on Test Case execution and Defects.
Traceability Report in Zephyr
There is a built-in dashboard that summarizes the tests carried out throughout the project. Creating your own dashboard, you can use dedicated gadgets that are very useful, as you can’t use Jira’s standard gadgets to report on test executions and cycles.
Requirement management
Requirements form the foundation for writing test scenarios. That’s why it’s important to support them with test management tools. TestFLO allows defining the types of issues that serve as requirements for a given project with tests. Such requirements gain new panels with all the defect issues and test objects, as well as the possibility of using such operations as test creation or Test Plan from the level of the requirement.
Zephyr Squad offers limited support in this regard. The requirement can be any issue; defining this element is impossible. This means users have an additional test creation action on every issue type in the project. It’s also not possible to associate a requirement with a test cycle or create a cycle from the requirement level.
Customization
Sometimes, the test process offered by the app out-of-the-box may not meet our needs fully. Then the tool’s configuration options come into play. In this area, TestFLO is the undisputed leader, allowing users to modify workflows, implement optional fields, configure the Steps field, or create their own flexible panels. And that’s only a fraction of all the app’s possibilities. The mere use of Jira issues as test objects offers huge scope for configuration.
View of Test Steps configuration in TestFLO
Zephyr Squad strongly focuses on the Plug’n’Play approach and built-in continuous testing process. The configuration is basic and limited to defining statuses or additional fields on the test executions.
Permissions and restrictions
Permission support is another strong point of TestFLO. The app supports Jira permission schemes but also introduces its own permissions module. Besides, we can use additional conditions and validators to model the workflow better. This support of permissions and configurability are of great value for implementing advanced test processes.
Zephyr Squad offers fewer options. Although it implements its own permission as an extension of Jira’s permission schemes, it doesn’t support Jira’s standard permission schemes. That way, a user who, for example, doesn’t have permission to edit the issue can change the test steps on it.
Integrations & migrations
In terms of integration with other tools, Zephyr Squad offers a powerful REST API called ZAPI. It introduces many useful methods to use in your scripts for testing purposes. It’s worth mentioning the tool’s integration with Confluence, which is used by a dedicated app. A big downside here is that you can’t export tests.
TestFLO provides a REST API, but also extends Jira’s API to support the fields and operations provided by the app. Users can easily export the tests and contents of the Steps field to a file. Integration with Confluence is limited as not all TestFLO fields are supported but there is a dedicated macro with the view of the Test Plan Iteration Report. Through integration with Configuration Manager for Jira, TestFLO provides you with an option to properly manage changes in your testing processes. When the changes are approved in the Jira test environment, they can be automatically pushed into your Jira production.
Test automation
Automatic testing support is offered at a similar level in TestFLO and Zephyr Squad. Both apps integrate with Jenkins and Bamboo and have dedicated plugins for CI Servers.
The biggest difference is that in TestFLO you can run the build directly from Jira, after which the test results are automatically imported. You can also view all these tests on a dedicated page (list of all active and finished automated test runs that users requested from a given Test Plan). In the case of supported formats, both Zephyr Squad and TestFLO support JUnit and TestNG XML test results.
Pricing
This is not the most important feature of the software, but it can sometimes make a difference. TestFLO is much more favorable here in every Jira user tier compared to Zephyr Squad. The table below shows the price list* of each app with its paid extensions. Please, note that prices are indicated on a yearly basis for all applications that are used on the hosting of the Data Center.
Key takeaway
All in all, the comparison showed that Zephyr Squad is a weaker tool in the basic areas of test management. It doesn’t offer a dedicated solution for saving preconditions or structuring tests. Reporting is mainly based on dashboards. The offered built-in reports (4) have no extensive filtering options, so they are not very informative. Configuration options are also very limited.
Zephyr Squad is a tool that is easier to implement. It has a built-in, permanent test process and is intuitive enough to enable users to start using it without referring to the documentation. A noteworthy feature of Zephyr Squad is the extensive REST API in the form of ZAPI and good integration with Confluence through the extension Zephyr Blueprints for Confluence.
As for TestFLO, it is a very customizable test management app with default settings to run the tool quickly. As in Jira, after installation, its users can have detailed configurations of their projects to use the tool effectively. TestFLO supports the entire test management process by introducing the folder structure for tests, the functional Steps field, and six useful reports. Attention should also be paid to the extensive support of requirements and transparency regarding the relations between various objects.
The biggest advantage of the tool, however, is its configurability. TestFLO allows designing various and even the most advanced test processes. The implementation of test objects in the form of issues, the use of custom fields, optional post functions, and workflow conditions, as well as the built-in authorization module and authorization support in Jira, make TestFLO’s customization possibilities really outstanding.
To get more information about TestFLO, follow the link to the Atlassian Marketplace. You can also book a live demo via Calendly, to see the app in action.
You can also read more on bringing test management process inside Jira on our blog: