8 common test automation challenges


30 August 2017 • 16 min read

    Test automation is the rage right now in Agile development environments. And for a good reason. Automation works great for executing smoke or regression tests. It also helps testing teams to prepare the test data and execute complex test cases, involve a lot of data, or are located on different platforms.

    Test automation frameworks can bring testing teams plenty of benefits, but only as long as they know how to efficiently manage common test automation challenges. When used right, automation can boost the productivity of your team and enhance the quality of your products.

    8 challenges and tips to solve them

    1. What to automate?

    If you think 100% automation is an option, keep dreaming. You won’t be able to automate every single test case. Especially in areas like compatibility, user interface, or recovery where most testing needs to be carried out manually. Before you begin to think about automation, you need to come up with a set of criteria test cases must meet to be automatized. Which test cases are important to you?

    Usually, it’s the areas that are prone to risk, of interest to clients, or characterized by complex business logic. Identify these application areas and test case parameters to make sure that your automation efforts pay off.

    For example, it’s a good idea to automate test cases executed with different sets of data or in different environments. The same goes for test cases with a large amount of data. But it doesn’t make sense to automate test cases or modules that run just once.

    2. The right automation tool

    That’s another tricky area of test automation.

    Automation requires tools and teams might find that problematic because of various reasons:

    • they lack the expertise to make the most of a specific tool
    • they don’t know whether the tool they need exists
    • they want a tool that isn’t available on the market
    • their tools of choice don’t offer 100% test coverage
    • the cost of tools exceeds their test budget.

    And the problem is serious. According to the World Quality Report 2019-2020: Benelux, over 50% of IT leaders claimed that their organizations lack suitable tools for automation when developing test environments for their teams. 

    If your team doesn’t know how to use a tool, you can organize a training session with its creator, buy an online course for your testers, or hire a consultant who can help your team master it. Reach out to the testing community if you’re still looking for the right automation tool – forums like Stack Exchange are a rich source of information.

    Sometimes your tool might not do everything you need. In that case, you should start searching for multi-tool solutions that cover areas critical to your team. If the tool you found is way out of your budget, just prepare a cost vs. benefit analysis and present your case to the executive team. An analysis of expenses coming from bugs that would have been solved if you had the right tool in place is bound to work in your favor.

    3. Team communication and collaboration

    Test automation generally requires more communication and collaboration. If your team finds it difficult to get genuinely involved in setting automation targets and objectives, you’re in for trouble.

    You need the active involvement of your team here. Automation relies on past testing data and experience or even proof of concept that only your team can deliver. For the automated testing to work, the entire team needs to be on the same page.

    Automation testers should be in touch with not only the business team, but also manual testers, developers, and technical architects. They’re the ones who present the ROI of automation to the executive management.

    You need solid team management support in place before you begin automating your tests. Create a collaborative environment where team members can easily communicate to deliver test results on time and with as little risk as possible.

    4. A testing approach

    That point is easily one of the most significant challenges for automation engineers.

    Testers need to identify the right test automation approach to match their needs. To do that, they must ask themselves questions like:

    • How will we reduce the cost of implementing and maintaining test suites?
    • Will our automation tool help us develop useful test reports and metrics?
    • Will these suites have a long lifetime on our team?

    That’s especially important if you’re working in an Agile environment where the tested application changes regularly and test automation needs to be implemented in a way that recognizes these changes, keeping the team up to date with a cost-effective maintenance strategy.

    That’s why you should consider getting an automation solution that detects these issues and automatically updates or re-validates the test.

    5. High upfront expenses

    Automation costs a lot at its initial phase.

    While the approach is very useful in Agile contexts, the expenses involved in setting up a test automation suite prevent many teams from incorporating automated testing into their practice.

    Before presenting the automation suite to the management, make sure that you analyze and build a test automation framework, where you include elements like libraries of reusable functions. Add both hardware and software costs as well.

    And don’t forget about licensing costs if you plan to use a paid software. And even if you opt for an open-source solution, you’ll still need to consider the expenses related to training and maintaining them.

    Convincing the stakeholders that automation is worth all that effort is a difficult task. But the potential ROI is a good argument, so prepare a plan together with a budget, and go for it.

    6. Accurate use of testing resources

    Those who aren’t convinced about test automation often say that it can be carried out by manual testers thanks to tools that help to record test scripts and play them back.

    That’s simply not true.

    Test automation requires teams to design and maintain test automation frameworks, together with test scripts, build solutions, and other critical elements.

    For the automated testing to work, teams need a solid knowledge of the framework’s design and implementation. That’s why you should build your team on testing resources that have strong programming skills and test automation knowledge.

    7. Stakeholders and end-users expectations 

    It doesn’t matter how great your tools or resources are – you should always remember that testing can’t be carried out in a vacuum. It serves an objective: improving your product.

    While test automation is great for automatically checking the system for bugs, you still need humans to complete the non-automated testing sections. And these tests bring plenty of valuable information to make the best possible decision when improving the system – not for the sake of developers, but the end-users.

    That’s why you need to ensure that your testing team, the management, and other stakeholders all agree and understand the desired results of your automation plan.

    8. When to begin and stop testing?

    That’s a very difficult question all test managers face at some point.

    Hopefully, by that time they know how to judge the importance of testing processes that have been carried out.

    But the start is just as important.

    You don’t want to initiate automated testing right at the wrong stage of your software’s life cycle. That’s just a loss of resources.

    Start with manual testing – your engineers will be able to tell when the system is stable enough and ready for automated testing.

    Automation is a hot topic today, but implementing automated testing suites also presents some challenges everyone interested in automation should take into account.

    Most importantly, automation should fit the organization’s test management strategy. Have a look here to read the post I wrote about key areas that determine the success of test management. 

    The biggest software testing challenges

    Test Management in Jira 101: the basics

    Why using Jira as a testing tool is a good choice for your project

    [contact-form-7 404 "Not Found"]