RTM by Deviniti vs TM4J by Adaptavist on a race against bugs

Apps, Atlassian, Requirements & Test Management

17 September 2019 • 25 min read

    If you ask your testers about the challenges they have to deal with every day, the majority of them will admit that race against time is one of the most problematic. At some point, the software development team finishes coding, and the release date is coming closer and closer. Managers are watching the testers’ back, requiring to predict and prevent all the possible bugs, but silently hoping they won’t find anything to fix. Fortunately, there are some tools which support the collaboration between developers and testers. One of them is Jira, a well-known project management software. The Atlassian suite upgrades workflow organization and makes it more efficient. But we can’t forget that testing teams have their own, special needs. Can Jira meet the expectations of even the most demanding projects? It turns out that indeed, it can! You’ll be surprised when you discover the miracles Jira Software with the right extension can do for your test management. 

    Why test inside Jira?

    The first answer which comes to mind is: because the time of your team members matters to you. It would be a waste not to benefit from the solution which can easily speed up testing-relation actions by structuring and prioritizing tasks. Besides, embedded testing will always impact the final product in a good way. Jira along with a dedicated test management app allows each stakeholder to monitor the process step by step, keep track of implemented changes, and see the reports anytime it’s necessary. The last example confirms a little bonus point: in addition to preventing bugs or duplicating work, testing in Jira also supports avoiding these famous meetings that should’ve been an e-mail.

    Tests performed in Jira not only economize your team’s time but also your company’s money. As the majority of people working in the software development industry already knows Jira, you won’t have to provide any additional training before the actual work starts. Jira apps are simply cheaper than external testing tools, and they still offer all the functionalities or even more. Considering the prices and the wide range of features, test management inside Jira seems to be the right way to go. Here’s a quick summary of what Atlassian product can do to improve your testing teams’ efficiency.

    Preparation of the competitors

    Requirements and Test Management for Jira (RTM) by Deviniti and Test Management for Jira (TM4J) by Adaptavist are two opponents in a race where the goal is elimination of bugs from your software. Each app has strong and weak points. The key to your success is choosing the right one, and that’s what we’re here to help you with. But first, before we start going through the pros and cons, it’s a good idea to analyze your individual needs. There are some factors which we believe should be taken into consideration first:

    • the size of your team;
    • the number of requirements;
    • the general complexity of the project;
    • the risk of unexpected changes; 
    • the project management approach.

    Gathering this kind of information is definitely worth your while, as it will help determine which conditions will be the best for your project and which features you really want. For example, if you have less than 100 people in the team, we recommend you using Jira Cloud instead of Server. Not only because it’s cheaper, but also more suited for smaller companies. The number of requirements constitutes an important question as well. If your project requires many of them, you’ll need a tool which provides its users with embedded requirements management. With this in mind, even if unexpected changes occur, you and your team will be well prepared.

    If you already got your starting point specified, we can now move to the actual comparison. We’ll look at the most common testing challenges and see how the tools selected for our comparison can help deal with them.

    Ready, set, go! 

    Stage 1: Requirements management 

    The testing process should be treated as a whole. It’s not possible to execute complete functional tests without well-managed requirements. The more complex the project, the more important this initial stage is. It leads to the conclusion that integrating requirements into the process is a good way to track the relations between them and the other testing objects. Requirements and Test Management for Jira (RTM) makes it possible. Thanks to our app, you can see at a glance if the requirements are safely covered by test cases throughout the process and make sure nothing’s omitted. Additionally, RTM for Jira lets us observe which requirements generate most bugs during the execution.

    There are different types of requirements, and structuring them well is related directly to the stakeholders’ and customers’ expectations. The best way to do this is through Jira issue types. With RTM, you can not only categorize your requirements but also organize them in a transparent tree with folders and subfolders. On the other hand, TM4J doesn’t provide its users with dedicated issue types nor any kind of structure for requirements, which makes effective planning of the further steps practically impossible.

    RTM has four dedicated issue types for requirements to choose from RTM has four dedicated issue types for requirements to choose from 

    Stage 2: Intuitive process

    This circuit starts well for both our runners. After a quick installation, typical for the majority of Jira testing apps, here comes the time for setting up Permissions. At this stage, Adaptavist’s and Deviniti’s tools give us many customization possibilities. TM4J offers even a built-in permissions module, which we can divide the roles and define who’ll be able to implement what changes throughout your project development, regarding separately each object. It’s also the only Jira built-in test management app on the market which provides the possibility of versioning test cases and creating large libraries for the issues ordered by labels. But this is where its advantages end, as later on, the testing process in Adaptavist’s tool gets more complicated. The test cases can be attached directly to the executions, which undermines the sense of creating test plans and inevitably brings chaos into the process.

    Most of the objects in TM4J by Adaptavist are displayed as a list, which may not pass the test when there are many elements.Most of the objects in TM4J by Adaptavist are displayed as a list, which may not pass the test when there are many elements. Image source: Software Testing News 

    RTM for Jira guides us through our testing from end to end. We start by analyzing requirements, then move on to creating test cases and assign them relatedly. Prepared and organized objects are ready to be scheduled into test plans. This way the process gets maximally standardized, and we know exactly which steps or objects caused potential defects. It’s much easier to fix something when we know exactly where it comes from. What’s more, our app ensures all the stakeholders are up-to-date, as each stage and change is easy to track thanks to a high-grained relations view.

    Thanks to well-described relations in RTM for Jira, your QA team will know exactly where the defects come from and be able to fix them right awayThanks to well-described relations in RTM for Jira, your QA team will know exactly where the defects come from and be able to fix them right away

    Stage 3: Transparent relations between the objects 

    As RTM for Jira is originally made to bring the whole software project into a single tool, it enables full traceability. And we’re not talking only about requirements and test cases. As the successive testing stages result from each other, it’s important that everyone involved in the process could keep up with the dependencies. Of course, by everyone involved we mean not only testers, but also analysts, software developers, product managers, and designers. The main goal of RTM is to provide its users with all the benefits of embedded testing, so it gives a wide range of relations tracking options. We can observe direct and indirect links between the elements on each issue screen on the Relations tab. There are also two transparent reports which display connections in an understandable way: the Traceability Matrix and the Requirement Coverage. The first one allows verifying correlations between any two baselined issue types using many-to-many relationship comparison. The second is generated as a table and focuses on requirements and their coverage by test cases, test plans, executions, and defects if they occur during the process.

    Requirement Coverage Report in RTM allows tracking the relations between requirements and testing objects

    Requirement Coverage Report in RTM allows tracking the relations between requirements and testing objects

    This time Adaptavist’s TM4J falls behind quite significantly. Inside the app, requirements can’t be directly connected to other testing objects, as they’re basically separated from the process. This is the main reason why their relations with related test cases can be set only through web links. Adaptavist’s test management tool for Jira offers the Traceability and Coverage reports too, but their design is rather simple and seems incomplete.

    TM4J's Coverage Report shows only basic information. Image source: Adaptavist App Documentation

    TM4J’s Coverage Report shows only basic information. Image source: Adaptavist App Documentation 

    Stage 4: Automated testing support

    TM4J takes the lead. The tool makes it possible to enter the code through built-in REST API. It also integrates with CI tools like Jenkins or Bamboo, supports writing tests in Cucumber and enables importing data from JUnit XML. This is a strong point of Adaptavist’s app, as automated testing is getting very popular these days, and more and more teams decide on facilitating their work with it.

    TM4J enables adding scripts through REST API.

    TM4J enables adding scripts through REST API. Image source: Adaptavist App Documentation 

    As to RTM for Jira, it focuses mainly on manual testing which is why it doesn’t allow to run scripts, nor integrates with automation servers. The good news is we’re currently working on it, and you can expect automation features to be added in the near future.

    Stage 5: Customizable test results reports 

    RTM for Jira is back on track! Even though most of the testing tools include built-in test reports, they usually present a completely different approach. Requirements and Test Management for Jira features four fully flexible reports, all ready to be exported to a .pdf or a .csv file. Two of them were described in the paragraph concerning relations. The other two are Test Execution and Test Case Execution reports. The first one shows the overall progress of our Test Plans, and the second goes in-depth for each test case. Both can be set up with filters, so we can select the data only for a particular ProjectTest PlanExecution, Environment, TCE Created Date, or TCE Assignee. The reports are generated as transparent charts with detailed tables below. With RTM, the risk of misunderstanding between the stakeholders decreases to a minimum!

    Transparent test result reports show the progress of testing and are ready to export to a .pdf or .csv file

    Transparent test result reports show the progress of testing and are ready to export to a .pdf or .csv file 

    Reporting in Adaptavist’s TM4J is possible, but not simple and definitely not intuitive. The tool provides various kinds of the Test Execution report. Instead of customizing our results with filters, we’re obliged to choose their display per parameter as separate reports. The forms of generated reports are also hard to read and will require an additional explanation from one group of team members to another.

    In Adaptavist's tool, there are many versions of the Test Execution report instead of filters.

    In Adaptavist’s tool, there are many versions of the Test Execution report instead of filters. Image source: Adaptavist App Documentation 

    And the winner is… 

    In this race, Requirements and Test for Jira (RTM) evidently emerges as a winner. From the very beginning, almost to the end, our app leaves its opponent far behind. It tripped only when it came to automated testing support, which is still yet to come, but in the end, it came back in glory. All the other features, like requirements management, reports, and relations-tracking solutions support your testing at a high level. What’s more, RTM for Jira is now available for Jira Server as well as for Jira Cloud, and it looks identical on both hostings, so in case of a need for migration, it can be performed smoothly.

    TM4J tries to catch up with the flexible Permissions settings and the automated testing support, but for most teams it still won’t be enough. Although Adaptavist’s app has all the functionalities necessary to perform your test management inside Jira, the clarity of the whole process leaves a lot to be desired.

    If you’ve decided to test our app, RTM for Jira, take a free 30-day trial from the Atlassian Marketplace. You can also book a live demo via Calendly to see the app in action. We also invite you to read more on bringing test management process inside Jira on Deviniti blog: 

    [contact-form-7 404 "Not Found"]