Test automation undoubtedly comes with a lot of benefits. Yet there are some misconceptions about it. Here we debunk some of the most common test automation myths. Take a look!

These days, everybody in the software industry has heard about test automation , and there are different opinions about what it is, its importance, and its actual value added to software projects. Some people say that they have implemented test automation in their projects and it didn’t add value or work at all, resulting in a waste of time and money. On the contrary, others say that automating the testing process is great because it solves many problems related to software quality and has improved their software product significantly.

These two completely opposite opinions are not considering several factors: What was the test automation objective? How was it implemented? How was it maintained? To make sure we solve these questions, we have selected some of the most common myths about test automation that generate this confusion.

MYTH #1 – Automate everything!

“If automation runs quickly, it is reliable because it does not make mistakes; it can run as many times as I want without being tired. Then let´s automate everything .” This is a common thought that you have probably heard. If you think about it, it seems to make sense; however, this could be the first step toward failure.

Before starting a UI test automation in our projects, the first question should be, what is the objective for test automation?

Usually, test automation is a great opportunity to automate repetitive tests (UI and API level). Regression tests or smoke tests are usually good candidates for automation because they are done manually. This automation will provide fast feedback about the application after applying changes, freeing up time for manual functional testers to test those changes in-depth instead of using an entire day to ensure that something that is working isn’t broken.

This does not mean you cannot automate everything. You can, but you might spend valuable resources unnecessarily. Remember that the benefit of automation will be seen when you run the automated tests repeatedly. To make sure it’s worth it to implement automation, keep in mind the following:

• How much time would it take to automate this script?
• How much money would it cost to maintain it?
• How significant would the impact be if a defect isn’t detected on this functionality?
• How much time would it take to test manually?

In addition, automation is code written by someone. When that person writes the code, he/she is focused on the code but not on the application’s quality or business rules, which can result in missing defects or failures in the product. Test automation doesn’t replace manual testing; they both should work together and complement each other, helping to improve the software development’s quality.

You may be interested: Software testing best practices and tools to avoid bad testing

MYTH #2 – Test automation is expensive

Starting an automation project is time-consuming and involves creating an automation framework for later writing the scripts that otherwise will take much more time than writing manual test cases. In addition, those scripts need to be maintained when the application changes for some reason (and it often does).

If your project is a small website that will take two months to finish, and it won’t ever be touched or changed in the future, I suggest you not automate. Nevertheless, if your project lasts a long time and requires possible modifications, then you will need to do a cost-benefit analysis. In any medium-large sized project, the cost of test automation will quickly be amortized — when a thorough analysis on what to automate is done beforehand —, and several benefits will arise:

• Increased efficiency and productivity
• Reduction in your overall cost per test
• Freeing up your previously bogged down testing team to innovate and inject real value into your business
• Quick feedback with the automation’s results

The image below shows an example of a UI test automation cost-benefit analysis (only analyzing time). The conclusion is evident: It is much more expensive at the beginning, the benefits are seen after some execution cycles, and costs are lower when talking about API test automation.

MYTH #3 – It is hard to maintain

Considering that test automation involves writing code, we must consider it as a development project with all its variables (architecture, maintainability, and other aspects that are considered when developing). When we consider automation, we should think about the architecture and framework that we will use. In any development project, the maintainability of our code is a priority, and in an automation project, it becomes more important because we need to change or update the scripts every time the application requires it. If we don’t ensure maintainability, we might be dooming the project to fail .

When automating UI tests (also known as E2E or End-to-End Testing), it is also important to consider the possible flakiness produced by the framework communicating with the browser. Likewise, the browser takes time to render the different UI controls, so the test needs to wait for certain “UI conditions” to make the next testing step. When building the framework, we should take into consideration those possible problems upfront to avoid high maintainability costs.

Problems mentioned above are not present in API test automation as there is no browser interaction, resulting in faster and reliable tests. This is why API testing should always be the first candidate to be automated.

Once again, the question of w hat to automate comes into play. Generally, we will automate pieces of the application that are considered stable, are already manually tested, and are not going to change (at least not much) in the future. The key is to avoid the need to maintain or change our scripts often. However, the approach of only automatizing what is stable is changing as we are giving more relevance to continuous delivery and automating as soon as possible. In this scenario, we need to adapt to changes quickly and maintain our code.

MYTH #4 – It will solve all of our quality problems

Including a tool in a flawed process will be just a defective process with fancy tools. Quality is not only about testing. Quality must be built by the entire team and from the beginning of the project. If you are already at the point where your application quality is not good, and your team is also overwhelmed by the deadlines, then adding automation won’t be the solution.

We previously stated some of the benefits of automation. It could help free up time for your team if you add new specialized professionals to do this task. Having your current team do it and stop working on other tasks to automate may not be the wisest decision. By doing that, automation will likely fail, and the project’s quality won’t be increased.

Take a look at this: Testing in software: much more than finding bugs

MYTH #5 – Important or difficult issues won’t be caught

This is a common misconception within software teams. When we add test automation, we expect automation to detect several issues, both the important and the difficult ones.

The reality is that we usually automate functionalities that are already working to make sure they don’t break throughout the project lifecycle. If none of those existing functions are broken, then no issues will be found by automation . Be aware that we need to write our scripts so that if the application is broken, the issue can be found.

Does it mean that automation is not giving us benefits because nothing is broken? Here comes the mistake. A well-written test automation suite that passes correctly consistently tells us in a matter of minutes, without any margin of error, that the overall quality of the application is good . It also shows that recent changes didn’t break anything and that we can continue and focus on testing other things. The same confirmation could have taken several hours to do manually and without the reliance that automation provides.

To Sum Up

Automation does help the testing process. It provides quick feedback, it runs tests in a short time while it would take days if done manually, and it can run tests as many times as you want in a consistent way and without missing any steps. Automation is a very nice complementary tool for testing that will give us confidence about the status of our software quality, allowing the testing team to focus on other crucial matters, thereby increasing the team’s productivity and the possibility to find more interesting and hard to find issues.

The key is to understand what we really need to automate, what goal we need to achieve, and how we are going to make it happen.

Comments?   Contact us  for more information. We’ll quickly get back to you with the information you need.