Should mobile applications always be tested for performance as part of our Mobile Application Testing Strategy? How to do it? What are the benefits of Apptim in this regard? These questions and more will be addressed in this article, with interviews with Sofia Palamarchuk, Matias Reina, Fabián Baptista, and Andréi Guchín. 

First of all, before diving into valuable details about performance testing on mobile apps, it is important to understand why it is so relevant to focus on creating a Mobile Application Testing Strategy. The answer is straightforward: mobile apps are getting more and more prominence on a global scale every day.

Technology impacts every aspect of people’s lives: it allows and improves countless processes. At this point, the growth in the use of cell phones and their apps is already unstoppable.

According to the “Digital 2022 Global Overview Report”, published by Data Reportal in partnership with We Are Social and Hootsuite Global, the world’s population stood at 7.91 billion in January 2022, and more than two-thirds (67.1 percent) of the world’s population now uses a mobile phone, with unique users reaching 5.31 billion by the start of 2022. This meant 95 million new mobile users over only one year.

In this context, the “State of Mobile 2022” report, made by Data.ai, revealed that mobile apps are used more than ever, reaching 4.8 hours per day in mobile-first markets. So, as you can see, having a solid and consistent Mobile Application Testing Strategy that includes performance tests is crucial in the digital age, with so many people online at the same time.

“Consumers are migrating their attention and wallets to mobile as over $320,000 flowed through the app stores every minute of 2021, an increase of nearly 20% from previous records in 2020. Mobile services from both early adopters and mobile-forced players remain in high demand — with global consumers downloading over 435,000 apps per minute”, outlined the research.

As a result of the pandemic, mobile usage skyrocketed, and user expectations have reached an all-time high, with 61% of users saying they will no longer tolerate poor app performance. In such circumstances, users are becoming increasingly demanding regarding application quality. Due to the many options available, it is easy to change apps, so a company can lose business due to the poor quality of its apps.

Now that we have the whole picture, let’s get started! How can mobile apps be tested for performance? Why is our Mobile Application Testing Strategy so important? 

Find out below, through an interview with Matías Reina, CEO at Abstracta; Sofia Palamarchuk, CEO at Apptim; Fabián Baptista, CTO at Apptim, and Andréi Guchin, Performance Hub Lead at Abstracta.

– Why is performance testing a key element in the app testing pipeline?

Matías: If I have a terrible performance in my app, the experience of my users is bad, they get frustrated and go to another app or simply stop using it. Being able to include both client-side (on mobile phones) and server-side performance checks in each release is essential to be able to release quickly/automatically and with quality, especially with a good performance of my app. 

Sofía: Developers are pressured to release new versions of an app faster with high quality, and need to mitigate performance risks before they reach their users. With continuous mobile performance testing, developers can ship faster while keeping mobile app performance and user experience under check.

Andréi: As technologies improve, users become increasingly demanding in terms of speed and performance. Today, for example, with the advent of 5G networks and the promise of full connectivity and much higher speeds, users expect their application response times to be lower and lower. 

However, these technologies do not have a direct impact on processing times within the system components that contain an application. In other words, even if my network is fast, my application will still be slow if the performance of my system is low. Hence the importance of running performance tests on this type of system. 

– Does it always make sense to run performance tests on mobile applications as part of our Mobile Application Testing Strategy? 

Matías: Having tests on the most relevant aspects of performance (start-up time, app size, etc) has a very low cost, is very simple, and can avoid rollbacks on the code in production and bad experiences for my users. Since the cost is low and the benefit is clear, I think it is always important to have these kinds of tests in my app development cycle.

Sofía: There are different types of performance tests that one can run on a mobile application, and each has different goals and costs. For every type of app today that is the front door of a business, it is important that certain aspects of performance are evaluated as this will have a huge impact on user experience and at the end of the day, revenue and brand. I always recommend starting small, with some automated checks for startup time and UX response times of main critical user flows, and then expanding to other areas of performance depending on the business needs and target audience.

Andréi: Ideally, yes. Even so, if you have limited time and/or resources, it is advisable to evaluate priorities. For example, if the application is functionally very unstable, my suggestion would be to focus on functional testing rather than on performance, since it is useless for it to respond faster or support more concurrent users if it does not do correctly what it has to do.

Another example could also be that you have a system that can be used both by the mobile app and the web from a browser on a PC and most users access the system via the web. In that case, having limited resources, I would focus the effort on the performance of the web system instead of the mobile performance.

– Which oblivious bugs could be ignored if performance tests are not run on mobile apps?

Sofía: Bugs related to hardware device usage, like excessive memory usage of your app or battery drain that will affect the end user. Slow loading times (more than 3 seconds) or low rendering times (less than 15 FPS) will create a bad user experience and if it continues, most users will abandon the app. 

Fabián: A typical problem that is difficult to find without focusing on these tests is when the app is installed for the first time, or after emptying the local data, there is often a synchronization or cache process that is not detected by the teams evaluating the quality of a new release.

– What are the main challenges testers face when they run performance tests on mobile, as part of their Mobile Application Testing Strategy?

Matías: When you need to extend client-side performance testing coverage, you must decide whether to use emulators or real devices. And you need to develop scripts that automate some of the steps that are performed on the app. The second point takes a lot of effort.

Fabián: The main challenges to evaluate performance are to accurately measure the time taken of an end-to-end user action, and generate robust automated tests and similar backend conditions to avoid false positives. 

Andréi: Something that is key when automating test scripts is to be able to record the flows at the request level between the device and the servers where the application is hosted. The latest versions of mobile operating systems (iOS, Android, etc.) have a security update that prevents capturing such requests in a simple way, which causes having to use other strategies to automate that sometimes take more effort or make the tests not so realistic.

– What is the reason for testers claiming that mobile applications are harder to test than PC applications?

Matías: The tools that are available for mobile applications are more “complicated” in general than PC tools. As a tester, I often need Android Studio or Xcode, and not all testers have the hardware (a powerful PC) and the knowledge to be able to use them. On the other hand, the variety of devices, screens, operating systems, etc. makes compatibility a serious issue.

Sofía: Device fragmentation (dozens of different OS and screen sizes) and network conditions (mobile devices can be connected from different environments – 3G, 4G, 5G, wifi, etc) make it harder to have good test coverage while keeping costs down.

Fabián: Mostly the great fragmentation of devices, and the lack of agility to put a new build in the hands of the tester immediately. 

Andréi: In addition to what has already been mentioned, mobile testing is relatively newer than the other and therefore there is less accumulated experience. About 10 years ago, when I started in this world there was very little information about how to test specifically for mobile apps and that implied a greater effort to solve certain problems.

– Why is it important to test app server-side performance?  

Matías: If the servers do not respond in a timely manner, the experience will be bad even if the client-side code is 100% optimized.

Sofía: Load performance tests help identify bottlenecks on the server side that will lead to longer response times or even downtimes (app crashes) when thousands or millions of users are concurrently using the app. It is crucial to take it in mind for our Mobile Application Testing Strategy.

Andréi: When we talk about performance, many client-side variables affect it directly and we cannot control it. Such as the type of device, the operating system version, and everything related to the network: connectivity, network type, bandwidth, and number of devices connected to the same antenna. Servers are part of the system in which we can have more control and more impact when looking for performance optimizations.

– Why is it relevant to test app performance on a physical or emulated device?

Matías: When we want to have an idea if our performance worsened with respect to the previous version, we can use either an emulator, which is cheaper or real devices. What is important is to have a configuration that is always the same, to use as a benchmark, and then be able to compare each new version of the app under the same conditions. 

If we want to have a lower level of risk, also analyze the real compatibility, and know the exact experience that our apps will have, we need to go to a real device. It will depend on the requirements of each app which will be the best option. I think that for most apps using emulators is enough.

– What are the benefits of Apptim in this regard? How can it help us with our Mobile Application Testing Strategy?

Matias: In manual testing, Apptim desktop helps to understand what happens in the application “under the hood” without the need to use Android Studio or Xcode. It also helps to report incidents much more easily, building bridges between developers and testers, and giving them more information while speaking the same language.

On the other hand, Apptim CLI allows integrating into the development pipeline any kind of test to evaluate client-side performance, both on emulators and on real devices in a surprisingly easy way.

Sofía: Apptim is a mobile performance platform to proactively measure app performance and UX. While most tools focus on monitoring app performance using SDKs, Apptim provides developers and testers with a lightweight profiling solution to set up, maintain, and scale performance testing on real devices, before deploying a new version of the app. With the use of test automation, Apptim can improve productivity and mitigate performance risks before they reach production.

For finishing, Fabiián Baptista shared 5 tips for creating a good performance testing strategy as part of a robust Mobile Application Testing Strategy:

1- Start by adding simple KPIs that don’t need automated tests, such as app size and startup time tests.

2- Automate simple test cases.

3- Choose low-end devices as the first benchmark.

4- Add pass-fail criteria for UX time taken on critical transactions.

5- Define your top KPI for your business and add it to your pipeline.

Would you like to contact Apptim? Over 10,000 teams have used Apptim for mobile performance testing! Get in touch here.

Are you looking for a partner to run performance tests in your mobile app? Abstracta is one of the most reliable companies in software quality engineering. Get to know our solutions, and contact us to talk about how we can help you grow your business.

Follow us on Linkedin & Twitter to be part of our community!