How To Decide The Entry and Exit Criteria For Regression Testing

Regression testing is an essential part of any software development process. It helps ensure that changes to the codebase don’t introduce new bugs or cause existing functionality to break. To effectively perform regression testing, it’s important to define the test entry and exit criteria carefully. In this blog post, we’ll go over how to decide on these criteria so that you can confidently and effectively perform regression testing.

What is Regression Testing? 

Regression testing verifies that if a change is made in one component of the product, the other components of the product continue to behave like before. It is a means to ensure that a change in one functionality of the product doesn’t result in unintentional changes in other functionalities.

This change could be anything: a defect/bug fix, code refactoring, a feature enhancement, a change in system configuration or run environment, or even upgrading a dependent library. Any change that can impact the behavior of the product calls for regression testing.

Need for Regression Testing 

Different modules of a product work together to achieve the desired behavior. A change in any module can often impact its interaction with other product modules, which may or may not be foreseen. This is the primary reason that regression testing is needed.

Advantages of Regression Testing

• Regression testing helps identify unforeseen changes in a product’s behavior after a planned change in its source code, configurations, system environment, or dependency libraries.

• Regression testing is crucial in maintaining a product’s quality.

• A feature enhancement that breaks existing features of the product or a bug fix that introduces new bugs as side-effects do no good to the product. 

• Regression testing ensures that seemingly good things are good for the product and working. 

Selecting Test Plan/Test Cases for Regression Testing 

A regression test plan could be a set of functional tests and/or a set of unit tests, whose success would indicate everything is good to go, and a failure would indicate a new problem.

What all to cover as a part of these functional and/or unit tests is a tricky problem, and solution varies from situation to situation.

A common approach is to test all the modules that don’t have the new change but interact with the module that has the recent change. However, this decision depends on the product design and your experience with the product quality.

For example, in the past, you have seen that changes in one module have had unforeseen impacts on other modules. In that case, the Regression test plan needs to be more extensive to cover sanity test cases for every module and acceptance test cases for the modules that directly interact with the changed module.

On the other hand, if you know that the product modules are very well designed and have stood the test of time on quality metrics, you can keep the regression test plan short and limited to only the modules that directly interact with the changed module.

Seeking inputs from product experts who know the internal details of the product helps a lot in creating the Test Plan for Regression Testing.

A well-versed quality engineer in the product internals or a product manager’s inputs could range from a technical assessment of the impacted modules to a business assessment of the affected user scenarios. These can serve as valuable information for the regression test plan.

As much as it depends on the product you are working with, the regression test plan also depends a lot on the resources you have at your disposal in terms of time and infrastructure.

If you are working against a strict timeline or a strict budget of resources or both, you know you cannot afford to have a pervasive Regression Test Plan, and your need of the hour is short.

Choosing a Regression Test Plan is striking the right balance between your available time and resources, guided by your experience with product quality.

Automation in Regression Testing 

A pre-existing automation setup not only saves your time from a manual run of the test cases, but it’s also less error-prone and has a baseline for test results against which the new developments can be compared and analyzed.

Suppose test automation can be scheduled to run at regular intervals; the time it takes for the product source code to build and the time it takes for the automation test suite can configure the frequency of the automated test runs. 

But automating test runs at regular intervals helps catch the problems early.  For example, suppose test automation is configured to run with every new build as soon as any new change in source code breaks an existing behavior. In that case, it can be noticed and rectified immediately.

If the frequency of tests runs daily, you need to analyze changes made only in a single day to identify the root cause of the problem.

If the frequency of test runs is weekly, the changes to identify a problem span a week.

As this frequency interval gets longer, the slower it gets to determine the cause of the problem.

Effectively for regression testing, a team’s manual testing effort becomes inversely proportional to its effort investment in automation test runs. 

The following are the possible tradeoffs you could come across while automating regression testing:

• shortage of human resources in the team to work on test automation
• lack of machine infrastructure to keep automation up and running
• extended-time of builds and test automation runs which make it practically impossible to run the test automation with every new build
• A high number of check-ins in the product source code daily which can put extensive pressure on the build and test automation infrastructure. 

The ideal frequency of your test automation runs is usually a balanced combination of your available resources and the urgency of your need for regression testing.

What is Entry and Exit Criteria?

In software testing, entry criteria and exit criteria refer to the specific conditions or requirements that must be satisfied for a test to be executed and the conditions that must be met for a test to be considered complete.

Entry criteria are the conditions that must be satisfied before a test can be executed. This might include the availability of specific resources or the completion of other tests. Entry criteria for testing helps to ensure that tests are run under the appropriate conditions and that they have the best chance of providing valuable information.

Exit criteria, on the other hand, are the conditions that must be satisfied for a test to be considered complete. This might include the number of test cases that must be run or the minimum number of bugs that must be found. Exit criteria help ensure that tests are run to completion and that they provide a comprehensive assessment of the software’s functionality.

In summary, entry and exit criteria in testing help ensure that tests are run under the appropriate conditions and provide valuable information about the software’s behavior.

How to Decide the Entry Criteria for Regression Testing?

If test automation is not integrated with the build system or automated test runs are not scheduled at regular intervals, Regression Testing needs to be performed manually. In that case, does every code change need to be regressed? The answer is No.

To qualify for Regression Testing, a code change needs to have an impact on other components of the product. To identify this, one needs to analyze the interactions of the changed module with other product modules.

Usually, developers making the change are aware of its impact on other modules and the overall product behavior and should responsibly demand Regression Testing if needed.

The following considerations can be made when deciding the entry criteria for Regression Testing:

• Change made in code is impactful. There is no need to regress modules if the change does not impact them.
• Presence of adequate test data and results. Previously executed test cases or plans should be present for Regression Testing so their results can be used as a benchmark for current behavior.
• Without available resources, time, team members, and infrastructure, Regression Testing cannot be performed.
• Intention to fix issues. If, for any reason, there is a lack of intention to fix the issues found during Regression Testing, there is no benefit to investing in Regression Testing.

How to Decide the Exit Criteria for Regression Testing? 

Exit criteria for Regression Testing are the criteria that help you choose when your regression testing should be considered complete.

Determining the exit criteria for Regression Testing is a delicate balance between resources available to you in terms of time and money and your risk appetite.

If you have the luxury of time and resources, having an extensive regression test plan that can cover as much as possible and be better safe than sorry is good.

But if you don’t have the luxury of time or resources, you need to strike a fine balance between what you can cover as part of the Regression Testing and what you are ready to live with some calculated risk.

The risk is calculated using technical and business knowledge of the product. 

Following are some suggestions that can be considered when calculating risk:

• Features of the popular product among users should be prioritized in Regression Testing.
  
• Technical areas of the product that have stood the test of time and have been known to be good on the quality reports are the ones you can choose to live with some risk.
   
• Analysis of previously reported bugs by customers as well as internal teams can help in identifying the popular features of the product as well as the weak areas of the product.
  
• The probability of a feature bug being reported by a user only arises if it is part of a user’s workflow. This implies that product areas with the most reported user bugs are also popular among users.
  
• Bugs that have been long open and still haven’t been fixed reflect the product areas that have the most negligible impact on the user’s workflows; probably, users have found their workaround solutions.

There is no thumb’s rule for deciding the exit criteria, it’s just a balance between what you need to validate for survival and what risk you are ready to live with. 

Regression Testing for Legacy Codebase vs New Codebase

Regression testing can vary significantly for legacy and new codebases. When working with a new codebase for a product, you can build the testing framework from an early stage of the product.

This is the time when you can incorporate unit tests along with functional tests in your testing framework. When these tests are run as part of regular automation, they can improve the product quality and the effort needed for regression testing later.

The sooner you start investing in your test automation, the sooner you start reaping its benefits. The earlier your test automation starts running, the better off you are in maintaining the quality of the product and minimizing the manual effort required for regression testing of every new build.

The situation becomes different when you work with a legacy code base that may or may not have test automation. Products that have legacy codebases are usually the ones that have managed to strike a chord with the users for several years, sometimes even several decades.

While developing most of these products, huge manual and dedicated testing efforts in terms of time and resources are used in the software development life cycle.

Regression testing for these products becomes challenging with evolving software development methodologies.

The legacy test frameworks were sometimes developed to reduce the manual effort rather than quickly perform the regression testing at regular intervals. And the list of challenges here can go on and on.

Despite all the challenges, legacy products are enhanced for new features, adapted to new run environments, and maintained for quality. Because working with legacy products has its benefits. 

Legacy products have massive data associated with them, which you can use to your advantage. With this data, you can quickly get information such as:

• what features of the product are popular with the users
• how do various product areas stand on quality metrics 
• what are the weak areas of the product in terms of quality 

With all this information at your hand, the trick lies in finding the right balance in your regression test plan – the balance in what you need for survival and what risk you are ready to live with, the balance in what your users care about the most and what your users care about the least.

If you can achieve this balance, you know your legacy product already has a loyal customer base that is not going anywhere as long as their expectations with the product are respectfully met.

Conclusion

Regression testing is essential in maintaining a product’s quality. To ensure good quality for the product, you need a good regression test plan with well-defined exit criteria.

To determine the exit criteria, you must identify what matters most and least for the product. Therefore, to achieve the right regression test plan, take inputs from all product stakeholders and find the right balance.

Frequently Asked Questions

What does Homoscedasticity in Regression means?

Homoscedasticity in regression means the variance of the residual or error component in a regression model is constant across the dependent variable’s values. It is also known as “Homoskedasticity.

What are the Entry and Exit Criteria for Unit Testing?

Unit Testing Entry Criteria:

• Ensure no item is registered in the query issue relating to the need. 
• Completed, authorized, and released documents for Technical Design.

Exit Criteria for Unit Testing:

• Remove unneeded label files, including all information logs, warnings, and so on. 
• Remove unused/unreachable/redundant code.
• Passing the Unit Test and the code is complete regarding the test requirements.

Is Regression Testing Before or after UAT?

Yes, regression testing is performed before UAT ( User acceptance testing). UAT has no involvement with the actual software functionality; it is the last part of the software testing process. Regression testing is performed on product/application features and upgrades.

Suggested Reading

Entry and Exit Criteria in Software Testing Life Cycle

Regression Testing vs Retesting-Differences and Examples

Automated Regression Testing Challenges in Agile

Automated Regression Testing by QA Testers

Regression Testing: Challenges, Strategies, and Best Practices