Automated Regression Testing: Challenges, Process And Steps

This tutorial explains the Challenges of Automated Regression Testing. We will also learn about the Process and Steps to Automate Regression Testing:

We will explore how to automate the regression test cases starting from identifying them, selecting a tool to do the automation, doing cost, time & effort analysis, writing scripts and finally delivering it to the manual test team so that they can execute the test cases from anywhere anytime.

If you are wondering why only the regression test suite, its just that the regression test suite is the prime candidate for automation as it is a set of test cases which are repetitive and time taking. Thus, automating them would indeed save you lots of resources and it would be less time-consuming too.

You will get quick reports on the regression test cases and you can use these steps to automate any other test suites too.

=> Click Here For The Complete Regression Testing Series

Automated Regression Testing

Recently, when I wanted to start my new Automated Testing Project with four resources, I thought of applying any one of the Agile methodologies. But I was not able to continue because a series of questions were raised in my mind.

The questions were like: 

• Is it possible to use Agile methodologies in Automated Testing?
• Can I use traditional tools?
• Should I go for open-source tools?
• What are the challenges I have to face if I am implementing automation in an Agile Environment?

In this tutorial we will analyze some of the challenges that we face while implementing Automation with Agile methodologies. Automated Regression Testing in an Agile environment stands as a risk of becoming chaotic, unstructured and uncontrolled.

Agile Projects present their own challenges to the Automation team. Agile methodology emphasizes team collaboration and frequent delivery of a product.

Factors like unclear project scope, Multiple iterations, Minimal documentation, early and frequent Automation needs and active stakeholder involvement, etc., demand a lot of challenges from the Automation Team.

Automated Testing: Challenges in an Agile Environment

There are several Agile challenges for the Automation team.

A few of them are briefed below.

Challenge 1: Requirement Phase

Test Automation developer captures the requirements in the form of “user stories”, which are brief descriptions of the customer-relevant functionality.

Each requirement has to be prioritized as follows:

High: These are mission-critical requirements that absolutely need to be done in the first release.
Medium: These are the requirements that are important, but can be worked around until implemented.
Low: These are the requirements that are nice-to-have but not critical to the operation of the software.

Once priories are established, the release “iterations” are planned. Normally, each Agile release iteration takes between 1 to 3 months to deliver. Customer/software folks take the liberty to make too many changes to the requirements.

Sometimes, these changes are so volatile, that the iterations are bumped off. These changes are the greater challenges in implementing the Agile Automation testing process.

Challenge 2: Selecting the Right Tools

Traditional, test-last tools with record-and-playback-features force teams to wait until after the software is done. Moreover, traditional test automation tools don’t work for an Agile context because they solve traditional problems that are different from the challenges facing Agile Automation teams.

Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle and it becomes appropriate to deploy automation. So the choice of testing tools becomes critical for reaping the efficiency and quality benefits of agile.

Challenge 3: Script Development Phase

Automation testers, developers, business analysts, and project stakeholders altogether contribute to kick-off meetings where “User-Stories” are selected for the next sprint. Once the “User-Stories” are selected for the sprint, they are used as the basis for a set of tests.

As the functionality grows with each iteration, Regression Testing must be performed to ensure that existing functionality has not been impacted by the introduction of new functionality in each iteration cycle.

The scale of the Regression Testing grows with each sprint and ensures that this remains a manageable task and the test team uses the test automation for the regression suite.

Challenge 4: Resource Management

The Agile approach requires a mixture of testing skills, that is, test resources will be required to define unclear scenarios and test cases, conduct Manual Testing alongside developers, write automated regression tests and execute automated regression packages.

As the project progresses, specialist skills will also be required to cover further test areas that might include integration and Performance Testing.

There should be an appropriate mix of domain specialists who plan and gather the requirements. The challenging part in Resource management is to find out test resources with multiple skills and allocate them.

Challenge 5: Communication

Good communication must exist among the Automation Testing team, developers, business analysts, and stakeholders. There must be a highly collaborative interaction between the client and the delivery teams.

More client involvement implies more suggestions or changes from the client. It implies more bandwidth for communication. The key challenge is that the process should be able to capture and effectively implement all the changes and the data integrity needs to be retained.

In traditional testing, developers and testers are like oil and water, but in an agile environment, the challenging task is that they both must work together to achieve the target.

Challenge 6: Daily Scrum Meeting

Daily Scrum Meeting is one of the key activities in the Agile Process. Teams do meet for 15 minute stand up sessions. What is the effectiveness of these meetings? How long have these meetings helped Automation Practice Developers? etc., are discussed at this meeting.

Challenge 7: Release Phase

The aim of the Agile project is to deliver a basic working product as quickly as possible and then to go through a process of continual improvement. This means that there is no single release phase for a product. The challenging part lies in integration testing and acceptance testing of the product.

Steps to Automate Regression Testing

The process to be followed to automate regression can be precisely divided into the following steps:

These 7 steps are explained below in detail in simple terms for your easy understanding.

1. Identification

#1) Identify the test cases which should be a part of the regression test suite.

• To start automating regression test cases, the very first thing that you need to do is to get the regression test cases identified and properly defined with all the steps, data and prerequisites.
• To make sure you have an effective regression test suite, don’t forget to include:
  • Test cases with recurring defects.
  • Test cases that cover end to end scenarios.
  • Test cases that are more visible to the end-users.
  • Test cases on boundary values.
  • Good mix of positive and negative test cases.
  • Complex test cases.

#2) Identify the automation tools that are best for your requirements and application behavior. Once the Regression test cases are identified and ready for automation, identify the tools that fit your test cases the best.

The best way to identify the tools is to make a matrix with the tools and your requirements and then keep a track of which tool satisfies which requirements.

Suggested Reading => List of the Top Automation Testing Tools

#3) Identify the Programming language that you want to use. With so many tools available on the market, multiple languages are supported by these tools. Thus, it’s important to identify the programming language in which you want to write your automation test cases.

Example: Let’s assume that we have a project where we want to automate a regression test suite for a browser-based application.

As Explained above, we will identify the Test cases.

• Let’s assume that our test case is “ Verify that a user can log in successfully using a valid username and password”.

Next, we will identify the Automation Tools.

• A browser-based test case can be automated with “Selenium”, “Ranorex”, “TestComplete”. Let’s decide on the Selenium tool as it best fits the requirements.

Now, let’s identify a programming language.

• We want to use “Java” as the programming language as it’s a highly supported language.

2. Analysis

#1) Do Cost analysis. It is very important to work within the budget limits. Thus, after the identification phase, you’ll have an idea of how many test cases need to be automated and which are the possible tools that can be used.

All the findings from the identification phase will help you to come up with an approximate budget and thus, you can get any approval, etc., if required.

#2) Do resource and effort analysis to see if you have the resources to work on this. Along with the cost analysis, it is very important to do a resource and effort analysis for better allocation of resources and using their time on the project efficiently.

While estimating resources and efforts, make sure to account for risks like, if someone falls sick, or if some test cases need more resources for execution, etc.

#3) Do Time Analysis. Time analysis is required to make sure that you can finish the automation project within the budget and timelines. While working on time analysis it will be helpful to prepare a timeline chart to monitor the progress during the development.

For a better timeline analysis of the project:

• Identify the tasks and subtasks for your projects.
• Prioritize tasks and subtasks.
• Draw a Gantt chart or network diagram for better picturization of the timeline.

Example: Going forward with our example in consideration from the identification phase, the explanation for this phase is given below:

Cost Analysis:

Let’s assume that the approximate cost for this project is $X amount.

Resources and Efforts:

For this, one test case has to be automated from end to end and we need one resource full time for approx 24 hours and we also need one more resource to review the work. Thus, we need 2 resources and the effort estimation is around 40 hours.

Time Analysis:

For this, let’s draw a small Gantt chart to see the timeline.

3. Training/Hiring

#1) If some resources require training, then plan their training. Sometimes you may have some manual testing resources who are interested in writing automation test cases or some people have worked on automation but different tools and are willing to learn the tool which you have selected for your automation.

Identify these resources and plan for their training so that they can start working on the automation of regression test cases.

#2) If we need more resources then work on the hiring plan. Once you have done the resource analysis for the efforts and if you are not able to satisfy the needs with the already available resources, then, plan on hiring some new resources with proper skills that are required for the project within the budget.

Example:

Let’s say we already have a resource who is familiar with Java concepts and wants to learn Selenium. Then we’ll arrange for selenium training for that person.

If we don’t have any resources available for automation. Then we’ll hire a person who has some background in automating such test cases using selenium and java.

4. Framework and Guidelines

#1) Once the tool and resources are ready, work on coming up with a framework or deciding on which one to use from the existing frameworks. Already built multiple frameworks can be used or you can build your framework from scratch.

When selecting or building a framework, make sure that you involve the components about, test cases, logs, reports, input, database connection, etc.

#2) Decide on the other supporting tools that you want to use. As developing automation scripts is a development task that involves writing code, it would be much better to figure out the other development tools which would be needed to support writing your scripts.

For Example, some tools that can be of help include Git, GitHub, Jenkins, etc.

#3) Outline the guidelines for writing automation scripts. We need to outline the guidelines so that all the resources working on the project are in sync and use the same naming conventions, the same procedures for code check-in/check-out, and the same programming language.

Example:

Framework: Let’s decide on a BDD (behavior-driven development) framework for automation testing.

Supporting tools: The tools that we need to fully support automation will be: GitHub, Jenkins, Log4J, Cucumber, and JUnit.

5. Automation Scripts

Start writing automation scripts. Once we have everything i.e., the tool, programming language, required skills and test cases that need to be automated, we can start writing automation scripts.

While writing the scripts, we have to make sure that:

• Guidelines are followed.
• We are using the tools.
• Test cases are modularized.
• We should be able to reuse the components if they are required in multiple test cases.

We also have to make sure that the code is properly maintained in the version control tool so that all the team members can collaborate easily.

Example:

Let’s write actual scripts to make this test case run. Samples from the scripts can be as shown below.

First, the cucumber scenario for this test case would look as below:

Feature: Verify Login functionality
As a user I want to Login into the application
Scenario Outline: Login into application
Given I open application
When I Enter username “<username>”
And I Enter password “<password>”
And I click on Login button
Then I go to Home page

Examples:
| username | password |
| testuser1 | password1|
| testuser2 | password2 |

After the feature file, we will implement the step definition for the steps mentioned in the feature file.

public class Login {
LoginImpl loginImpl = new LoginImpl();
@Given("^I open application$")
public void i_open_application() {
loginImpl.openURL("URL");
}
@When("^I Enter username \"([^\"]*)\"$")
public void i_Enter_username(String arg1) {
loginImpl.enterUserName(arg1);
}
@When("^I Enter password \"([^\"]*)\"$")
public void i_Enter_password(String arg1) {
loginImpl.enterPassword(arg1);
}
@When("^I click on Login button$")
public void i_click_on_Login_button() {
loginImpl.clickLoginButton();
}
@Then("^I go to Home page$")
public void i_go_to_Home_page() {
loginImpl.verifyHomePage();
}
}

Finally, the actual implementation of the login functionality class would look as below:

public class LoginImpl {
WebDriver driver;
public LoginImpl(){
System.setProperty("webdriver.chrome.driver", "webdriver/chromedriver.exe");
driver = new ChromeDriver();
}
public void openURL(String string) {
driver.get(string);
}
public void enterUserName(String arg1) {
driver.findElement(By.id("UserName")).sendKeys(arg1);
}
public void enterPassword(String arg1) {
driver.findElement(By.id("Password")).sendKeys(arg1);
}
public void clickLoginButton() {
driver.findElement(By.id("LoginButton")).click();
}
public void verifyHomePage() {
String currUrl = driver.getCurrentUrl();
if(currUrl.equals("homePageURL")) {
System.out.println("Home page verified");
}
}
}

6. Review

#1) Code review: Once an automation developer is done with writing automation scripts, it is very important to go through the different levels of code review.

Code reviews help in

1. Identifying wrong verifications.
2. Finding code optimization points.
3. Finding better ways to implement the functionality for efficient use of resources.

Code reviews also expose a developer’s work to other developers and give it a different perspective and room for improvement too.

#2) Test cases review: Along with the code review, the test case review is also very important. We need to make sure that the automation test scripts when executed perform the same set of actions and verifications as we expect from the manual test cases.

Thus, reviewing automation test cases with business analysts or test experts helps in boosting the confidence in automation testing of those test cases.

Example:

For our example in consideration, let’s say for code review we got comments like “search element by id and not name”. Here, we’ll take this into account and modify our script accordingly.

Also, a test review can be done to add steps for testing if you are on the home page after a successful login. We will add this to our scripts too.

7. Deliver

Deliver test cases so that anyone can run them at any time. Once the automation scripts are ready for use, it is very important to come up with a delivery plan for the automation scripts.

This plan is required as we want to make sure that automating the test cases does not limit its execution to a specific set of people or skills. Everyone in the team or the project should be allowed to execute the test cases.

One possible delivery plan is to provide a Jenkins job that can be triggered to execute automated test cases.

Example:

In our case, let’s assume that we have delivered the test case using a Jenkins job. This Jenkins job, takes the code from GitHub, builds it and runs the test cases on a different machine.

Once the job is successful, it will show you the test report generated. Anyone who has access to Jenkins can run this job. This job can be scheduled to run at a specific time.

Conclusion

If we can meet these challenges in a well-optimized manner, then Automated Regression Testing in the Agile environment is an excellent opportunity for the QA to take up leadership of the agile processes.

It is better placed to bridge the gap between the users and developers, understand what is required, how it can be achieved and how it can be assured prior to deployment.

Automation practices should have a vested interest in both, the results, as well as continue to ensure that the whole evolving system meets business objectives and is fit for purpose.

Automating a regression test case is always helpful as it’s the best candidate to start automation testing. You can follow the steps mentioned above to automate any test suite and not just the regression.

Automation testing is very helpful and cost-effective too and the only time investment in automation testing is only in writing scripts and maintaining them. Thus automation testing needs to be properly planned and scheduled for a successful project.

About the Author: J.B.Rajkumar has more than 15 years of experience in both Academics and Software Testing. He has worked as a Corporate Trainer, Test Lead, QA Manager and QC Manager. 

Let us know your comments/suggestions about this article.

=> Visit Here For The Complete Regression Testing Series