Reducing testing costs is often the first thing organizations think about when they consider adopting test automation. After all, software testing is shockingly expensive; it accounts for an average of 23% to 35% of the overall IT spend, per the latest World Quality Report. Automating testing frees you to focus on more interesting and value-added tasks, while performing routine checks faster, more frequently, and with increased precision.
Clearly, the value is there. However, even open-source testing tools require a resource investment, and that requires the buy-in of someone higher up. How do you convince the business that test automation is worth the time, effort, and cost required?
The typical approach is to perform a simple ROI calculation: Estimate how many manual testing hours you can free up, and then multiply that by the testers' hourly rates. That's certainly a good starting point. But the elements of a strong business case for test automation consider time-to-market, quality/risk, and a much broader assessment of ROI.
1. Time to market
Time to market is all about how fast you can get a new feature or update in front of the end user. The longer it takes you to release new features, the less agile you are and the more difficult it becomes to respond to dynamic customer needs and emerging market trends. Depending on the nature of the application, delays can negatively affect revenue, operational efficiency, and/or your competitive advantage.
Given that testing is routinely cited as the top source of delivery bottlenecks, it seems fair to say that time to market is highly dependent on testing efficiency (how fast you can execute your tests).
Testing efficiency, in turn, is based on the time it takes to execute an automated test run (including preparation, execution, and analysis) compared to a regular test run (again, including preparation, execution, and analysis).
The greater the difference, the greater your potential to achieve time-to-market gains. And, the more frequently you run tests, the greater the payoff.
Other considerations that affect testing efficiency include:
- How many tests you should be running: This might be more or fewer than you're actually performing now—test case design methodologies and risk assessments can help with that.
- The availability of test data and test environments: Just defining automated tests is not enough; you need to have continuous access to all the elements required to execute them if you want to reach the finish line faster.
- The point at which testing occurs within the software delivery lifecycle: "Shift left" strategies such as adopting automated API testing and using new technologies to start testing UIs and APIs before they are implemented can have a positive effect on testing efficiency, and therefore accelerate time to market.
2. Risk reduction
Risk reduction is all about how many defects you can prevent from slipping into production. Escaped defects bring a range of woes. You've probably seen the curve showing how fixing defects after release is exponentially more costly than catching them early. But there's also downtime, support costs, and, in many industries, penalties to worry about.
Knowing your business risk coverage is key for assessing risk reduction. Determining this involves ranking the risk of your requirements or use cases against one another, mapping tests to those risk-weighted requirements, and tracking how well your testing is covering your top risks.
It's entirely feasible to have 10 strategically created tests achieve much more risk coverage than 100 tests created without risk in mind. The greater your risk coverage, the lower the defect density.
3. ROI
The third component is the one that people usually think of first: How much time and cost can you save with more efficient testing? With the help of automation, you can run more tests in the same time—or you can run the same number of tests in less time.
You can find a lot of quick-and-dirty formulas online, but figuring this out accurately depends on several factors. Be sure to consider things such as:
- Number of test cases
- Number of releases
- The time and effort required to manually create reports
- The number and types of applications you're testing
- The duration of manual test execution
- The time required to acquire/create and prepare test data
- The time required to acquire/create and prepare test environments
- The test suite's business risk coverage
- The test suite's level of redundancy
- Who is involved in testing (including business experts) and their daily rates
- The total cost of labor
- The cost of fixing a defect that slips into production
- The stages of your tests (smoke vs. daily vs. full regression)
Putting the pieces together
To give you a concrete idea of what a business case accounting for these various elements might look like, here's a real-life, anonymized example from a large European food and beverage manufacturer.
Figure 1: This organization collected quite a lot of detail about its testing efforts. Source: Jori Ramakers
All the data points play a role, but let's focus on the elements with the most impact:
- The company's current risk coverage is 45%. Most organizations just starting a testing transformation have around 45% to 55% risk coverage, so this is pretty standard. Many organizations don't calculate risk coverage. This organization has not yet optimized its test suite or adopted a methodical approach to test case design (pairwise, orthogonal, linear expansion, etc.).
- There's only one application under test, but it's a big one: SAP ERP Central Component (ECC). Test automation for complex business apps that require specialized domain expertise tends to have a higher payoff than, say, test automation for simple web or mobile apps.
- Defects found in test came to 650.
- In terms of release frequency, the company has one major release and 11 minor releases per year, plus 48 patches. They all require extensive testing to ensure that core business processes aren't disrupted.
- The cost of a production defect is €3,500 (about $4,200). This includes the cost to fix it as well as how much it costs the business in downtime.
- The total cost of testing, for this team testing a single SAP instance, was €1.4 million (about $1.7 million).
- This organization is currently undergoing a digital transformation, but testing is 100% manual. Not surprisingly, testing is a bottleneck to the company's innovation speed. It has adopted agile and is working in Scrums, but testing lags behind each sprint. It wants to increase its release frequency, but testing is holding it back.
If you look at this business case from a purely cost-savings perspective, the cost of testing actually increases a little bit. But at the same time, the risk reduction and cost avoidance gains are significant: from €2.3M to €19.8M.
Delivery is expected to be nine times faster once the company shifts a significant portion of manual testing to test automation (see figure below). The increase in testing frequency will result in almost 10 times more defects being found in test (from 650 to 5,649).
Figure 2. After moving to test automation the cost of testing actually increases slightly—but the number of defects found increases by nearly 10 times, and delivery is nine times faster. Source: Jori Ramakers
With this case for how test automation aligns to the organization's goal of accelerating innovation speed without compromising quality, the organization gained approval and executive support for its test automation initiative.
Take it to the next level
Once you have a clear view of how test automation will pay off with respect to speed, cost, and risk, you can take it to the next level. What is your business really trying to accomplish right now? Is it delivering innovation faster? Driving business efficiency? Improving digital customer engagement and experience?
Explain how your test automation initiative affects these types of high-level company goals, and you'll likely find that it's much easier to get management approval. Then it's on to the next challenge: aligning your testing strategy with these goals and reporting on how test automation is contributing to top business initiatives.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
