Android Apps Quality Assurance

Introduction

This article has been written for customers and other mobile developers to show the scope of QA services required for performing a full check of a mobile application (for Android-based apps).

The importance of app testing (a.k.a QA or QC) is typically underestimated, misunderstood, and treated as a completely useless part of the app development budget.

Customers are used to performing quick explorative QA via clicking on every screen of the app. Development budget appears to be low. Developers are also happy as in most of the cases, quality feedback is quite positive.

However, one of subsequent application releases is sure to prove, that the application build is far from stable and there's quite a few issues to be addressed. The customer can’t quite figure out at which point this happened, how on earth could this ever happen, and whether this is a common issue, or specific OS- or device-related issue. The app needs to be updated, but the budget for this update is uncertain, and may as well reach about a half of the total initial development budget. This is when panic kicks in, and finger pointing starts.

And just think that none of this might have happened if it weren’t for:

• very “superficial” testing of the first release (or none whatsoever)
• no information of real quality level of the app
• no information on what was exactly tested? Guys only can say something like “I remember that it 100% worked, but it’s not working any more. But within what conditions did it work then?
• no information for quality comparison (no test reports for first app release and all subsequent releases. No possibility to track when the issue appeared, what devices it relates to, what OS versions it relates to, what other condition may have impact on it)

With proper QA process, it is much more possible to reveal serious issues on initial development stages, and easily fix them. Having no QA, customer has a risk of having serious issues that may appear after warranty period and then it’s way more difficult to resolve.

In this article, we’ll cover a scope of work for QA team on a brand new (not updated) mobile application project, that runs on Android, and is accessible for non-tablet devices only.

SEE ALSO: Main Quality Assurance Tools for Mobile Apps Testing

QA Scope

QA activities involve several steps that help make complete application review:

• Test Plan
• Functional Testing
• UI Testing
• Stability Testing
• App analysis via development tools

Nowadays, there are about 4000 models of Android-powered smartphones and tablets, and they differ quite a lot. It will be quite expensive, difficult and time consuming to test application on all these models. That's why customer should analyse his market, identify top 10 popular devices and popular OS, and limit the scope of devices. In this case development company can provide a guaranty that application will work perfectly on identified phones.

High fragmentation of those devices, and differences in their operating systems, usually cause the following issues and sometimes the risks that cannot be easily prevented

• custom OS build that come from the device manufacturer
• custom OS build from mobile network providers
• custom OS build from mobile shops networks
• custom OS builds that don’t support Google Services
• custom OS updates for all cases mentioned above
• if the user has a root access to the system and can grant it to some side applications (as in such case some of 3rd party services can have any effect)

That is why our QA team focuses only on adjacent OS versions (for real devices), and on exactly defined OS versions (for official emulators).

Usually, we make sure the decision on 4 emulators and at least 2 real devices support is discussed and approved with the client. Testing on these devices will cover a complete QA process (a full test plan). Emulators however, are never used for final QA testing.

As for the emulators, they can be used only for cases where device specific functionality (camera, bluetooth, GPS, flash etc) is not of highest priority, or is not considered.

Target cases where mobile emulators can be used:

• check whether application works properly on the OS version that was declared as a primary;
• UI issues testing
• app analysis via development tools (performance, memory leaks, assets management like files operations)
• emulation of calls / text messages
• emulation of network environment

One of the benefits of using emulators, is that it’s very easy to obtain the cache of the app, it’s database and all possible private data of the application. There’s no way you can have this done with the usual devices. Only the rooted ones can provide these features.

Below are the examples of Android emulators, and their properties:

Also, real Android Devices that are used for quality assurance for mobile app:

As a rule, mobile app testing includes one iteration of three top scenarios / cases of primary importance.
This is an observable testing, aimed to check the app for possible issues on a set of additional devices (additional devices can be suggested by the customer).

Test Plan

Test plan is a document, that explains the testing strategy and methodology, resources and schedule of the testing process. It describes testing environment: requirements, conditions and cases that will be verified by a QA engineer, and an approach of mobile app quality validation.

Test Plan can be created on a basis of:

• former company knowledge
• UI/UX design guidelines related to selected platform
• development guidelines
• common failure cases
• application specs
• customer notes (regarding key features and priorities)

Functional Testing

The main purpose of functional testing is to assess the quality of system performance (within the selected scenarios).

See the below user stories, as examples:

• Order Taxi
• Add Credit Card to your pocket
• Search for the nearest fuel station
• Buy some goods via received discounts

Mostly, our company performs functional testing, using limited number of devices and emulators (by default, there will be 4 emulators and 2 devices (as mentioned above).

If agreed, customer can add any additional mobile devices to the set.

UI Testing

Testing a mobile app for UI issues, requires the following:

• UI designs specs
• reference to some existing application with similar UI elements
• just general UX design concept (that refers to standard Android UI) with minor customization
• other cases

Even the simple (from the first sight) screen can have several different states, which must be implemented correctly. Let’s imagine a screen with a list of messages. By default it will have 3 different states: empty list, couple messages, a lot of messages. And we’re not talking about any interactive events, like what to do when a new message comes in.

In UI testing validation the scope is huge. Some of the main UI issues are mentioned below:

• setting proper UI layout (paddings, margins, fonts, texts, text alignments)
• editing graphic resources (images, icons, and their quality)
• correct animations, depending on usage scenarios (view scrolls, view transitions, button clicks etc)
• correct popups, notifications, messages etc
• localization is a different story. All the items listed above should be checked for every supported language.

With a large number of screen densities and supported languages QA time will increase respectively.

QA time for UI testing usually increases because of:

• screen densities
• large number of supported languages

Lets say, we have an application that consists of 15 different screens and let’s say 15 more states of those screens. From a QA engineer point of view, it is reasonable to consider those screens as different (as UI is different on each of them). As a result, the application testing will total in 30 screens.

Taking into account that we are planning to check the app on 4 emulators and 2 devices, total summary of screens to check will be (4x30 + 2x30) = 180 screens.

Stability Testing

This type of functionality validation will include the following:

• testing for app state management cases
• testing for online/offline usage (and poor internet connection)
• cases of receiving call/sms during server communication
• via throwing of artificial application crashes

More details on the stability part of the QA are provided below.

General Stability should be checked

• When going to background, an app should prevent accidental data loss due to Back navigation
• When app returns to the foreground
• When the app is resumed from the Recent Apps switcher
• When the app is resumed after the device wakes up from sleep (locked) state, the app directs the user to the exact state in which it was last used
• When the app is restarted from Home or All Apps, the app restores the app state as closely as possible to the previous state
• When the app resumes from the sleep mode
• When the app resumes from the lock screen
• When the app is running in landscape/portrait mode (draw/hide hardware keyboard if exists)
• inApp purchase workflows
• Application or Service were terminated by device operating system. QA should check how these terminated entities are restored

Note: testing of service stability is completely separate and a quite complicated task. If service crash occurred, the OS will automatically restore it, so even if some critical issue exists, QA engineer may think that the service is still stable. To avoid this, QA Engineer has to enable notifications about service crashes in the device settings menu.

Check out : ​Quality Assurance Tools for Stability/Stress Testing for Android

Network Stability

• Device is connected to the Internet via Wi-Fi
• Device is switching between Wi-Fi / 3G / 2G
• Device is out of network reach
• Resume connection when the device gets back into network reach
• Data sync operations are processed correctly, after re-establishing connection
• Cases of poor Internet connection (intermediate online, offline states)
• Cases when receiving call / text messages during data sync

App Analysis Via Tools

Benefit of QA and Development Team working at the same company is that they can easily share their experience with each other, thus performing more solid audit of developers efforts. That way, QA Engineers will also have access to the source code and a possibility to:

• analyse code via different tools (e.g., memory leaks)
• analyse logs produced by the application (for crashes, for unallowed data output like credentials, for other unexpected messages that prove fact of violation of agreed workflows)
• analyse artifacts produced by the application (preferences, database, files, cache)
• have alternative points of view on working with the application

These activities are mostly performed simultaneously with functional testing.

Summary

Following services are included in the QA scope:

• creating a test plan
• functional testing (based on a test plan)
• UI Testing
• Stability testing
• App analysis via development tools

Usually, there are 2 options of QA services for customer to consider:

• Minimum Option (QA on single device )

• Recommended Option (QA on device/emulator set)

The difference of these options is only in a device set (devices + emulators) that are used for application audit.

Minimum Option

The option considers QA only on single device. List of proposed devices are provided in table below. Customer should select one of these devices and decision usually depends on particular project.

Recommended Option

Most solid QA will consider application audit on the following set:

• QA on emulators
• QA on real devices
• top scenarios QA on other devices (device set is limited)

Primary Devices

Emulators

Secondary Devices

SEE ALSO:

• Porting iPhone App to Android: is Testing on Real Devices Absolutely Essential?
• QA Tools to Check Apps` Networking Activities
• Useful Tips for Developing a Successful Educational App

About Lemberg

Lemberg is a UK mobile and web development company with strong client base in the UK, Europe, and the USA.

Starting from 2007, Lemberg has been helping leading design and marketing agencies, start-ups, innovative businesses deliver brilliant digital solutions for a number of the world’s biggest brands.

Our goal is to go beyond clients’ expectations: as a technology partner, we take the responsibility for implementing the most ambitious, creative and innovative ideas.