Testing Processes within App Development Agencies

In this article, I’d like to share some of the lessons I learned working with these companies, and more specifically about the testing aspects of such projects. Before diving into these lessons, I’d like to highlight some of the key concerns that agencies deal with day to day:

- Projects that are often short-lived.

- Ever changing client requirements and reprioritization

- Difficulties in coordinating client handovers

- “Sign-off” of projects often dragged out by clients

- Staffing and resource management challenges based on projects and priorities.

- High priority on successful launch

- Limited or lack of QA (understaffed)

- Client complaints due to declining app ratings

- Lack of real device testing (or not enough devices for testing)

 

Some of these highlighted challenges are why some of our clients got in touch with us, giving reasons such as “We want to test more quickly” or “We want to test on more devices”. You might think that the adoption of test automation would be similar for both ‘app agencies’ and in-house teams, but ‘app agencies’ have some unique challenges and often a different setup.

Form any agencies, in our experience, manual testing has been the go-to method. Sometimes this testing is performed in joint collaboration with the client.

We were curious to see whether mobile test automation could be phased in, and how best to do so. Implementing test automation can be particularly daunting for such agencies, notably because in depth knowledge of mobile test automation is very often not present, or not distributed widely enough in such companies.

Agencies very often consist of loosely knit project teams, meaning that people who work together on project A may not work together on the next project. This leads to a unique set of challenges in terms of mobile test automation, even if there is a knowledgeable resource: the resource in question may simply not be available for a given project. 

If mobile automation is new to the company or limited to key individuals, integrating it into daily processes, or even simple planning for integration can be a challenge. Even basic questions such as what should be automated and when often go unanswered. How does one implement effective tests that can be scaled to hundreds or even thousands of tests? 

Without dedicated and experienced testing resources, familiar with multiple frameworks and technologies, it is very difficult to build automated testing into a very fluid team structure and process.

Despite these challenges, I have seen successful automation implementation strategies in some agencies. In most cases, this success can be attributed to a senior automation engineer made available to the organization long enough to work with key individuals, or to help project teams to establish the first 10 to 20 tests. Once an environment is properly set up for automated testing, and teams see the benefits, adoption is almost assured.

Based on the above-mentioned discussions, and our experiences working with said clients and others, the following five lessons and best practices have helped us greatly. We hope they will be of help to you:

 

Challenge: Clients are often not precise enough on requirements and user stories.

Often, establishing an in-depth test-case test plan is a solution unto itself, as it can provide an overview of which use cases should be tested overall, and by extension weed out user stories or requirements that are unclear or poorly conceived.  Not only does this overview serve to plan and scope the testing necessary, but it forces the client to think critically about the definition of a successful test, which is very useful for obtaining sign-off.

Usually the initial project requirements can be used to create such a plan. Even if created at a later project phase setting up such a test-case plan can be done with a minimum of required input, depending on the scope. A simple Google spreadsheet similar to this template can be enough to get an overview on the scope.

For this process, it also helps to ask the client which devices and OS versions the app should support. In particular, you want to be sure that the developer is not using a library that may not be supported by lower OS versions , and later find out that said library needs to be stripped out.

Challenge: Unexpected, unplanned or changing requirements halfway through the implementation can cause code complexity and bring regressions.

The nature of app agency projects are usually “green field” projects meaning no prior code is written and the developer starts from scratch. Since the client counterpart often does not have any prior mobile experience we have seen that very often new ideas and change requests pop up as the app is being created. Change requests from the client often result in redefined priorities which can require stripping out, refactoring, or implementing new code.

Changes to any given part of the app can affect untold others. Being able to quickly execute regression tests becomes crucial, in order to ensure that when other functions fail, they are caught quickly. Skipping tests and communicating to the client that changes have been adopted should always be the last resort. The client will inevitably find the low-hanging bugs and will lose confidence in the team, and the company. 

In order to avoid this scenario, companies that haven’t adopted automation often compensate with large manual testing teams. These teams are often very costly to maintain for every day testing, yet inadequate when a crisis occurs. Which brings us to the next common problem: Not enough resources in place to conduct adequate testing.

Challenge: App development agencies often have to deal with multiple projects, tough deadlines and limited developer resources.

In the best of circumstances, it is often a challenge to ensure that you have the best available resources on an ‘as needed’ basis. This is especially true when working in partnership with a new client, as you have yet to discover their experience and organizational values. 

A common solution used by such companies is to leverage external help on an ad hoc basis (hiring outside agencies, or leveraging online communities). In order to address this need, here at Testmunk, a key part of our offering is the access provided to our automation engineers. This capability is especially popular with agencies that also take on the maintenance and ongoing development of their client’s app projects, since for every new release and/or upcoming new OS version, the agency has to go through a rigorous testing process.

Challenge: Clients often have media budgets and strict timelines set for their app launch, and thus app development agencies experience a great deal of pressure to ensure that everything goes smoothly for the launch.

By laying out a brief plan, and sketching out the activities required prior to launch,  we have found that many organizations have improved their launch success. This plan should include individual responsibilities for clear accountability, such as who is responsible for beta testing, or who is responsible for verifying UI elements. 

It should also list the processes to be followed, such as the process for reporting bugs. Lastly, this plan needs to provide key definitions, and establish clear standards, such as determining criteria for identifying a high priority bug. Other considerations that may be included are staged rollouts or a targeted Slack/Facebook group for your early adopters. We discussed these procedures here.

One common issue that is often overlooked is the UX issues that are caused by OS or device fragmentation. This can be particularly problematic for an app agency, due to the many differences between clients, and the constant influx of new people involved in your projects.

You can learn more about typical UX issues due to different device factors here.

Another problem commonly encountered in an app agency, is that you might have ironed out a definitive testing (or development) process that has worked in previous efforts, but be unable to use it due to imposed client processes you may not agree with. 

All of these factors have to be considered and adjusted for in your initial plan. A plan may not survive the initial sprint, but it is still important to create one, in order to have a frame of reference for the amount of change.

Challenge: A new iOS or Android version has been released, and suddenly an old client calls you in a panic. Users are complaining and your agency is being pressured to update the app as quickly as possible. The first challenge is to figure out what has failed/broken. The second challenge is to ensure that the broken feature (or features) not only function on the new OS, but remain functional on the older versions.

Besides testing and carrying a large set of devices, we recommend you look into real device cloud testing. Experience has shown us that it is vital to perform as much of your testing on real devices as possible, since emulators do not provide you a true representation of the possible problems that can be encountered. Here at Testmunk we carry a large set of different devices and OS versions at all times, and update our cloud device lab constantly.

App development agencies often deal with a high amount of client pressure and are consequently under a great deal of stress to deliver on time. Agencies are also frequently facing scope changes and limited resources to its project, yet currently still rely on repetitive and time-consuming manual testing. 

At Testmunk we have been working with several agencies that have had great success phasing in test automation in order to boost efficiency and productivity within their development process. With automated testing on a constantly updated real device cloud, you will also never be caught unprepared for client emergencies arising from OS or device releases. 

Nevertheless, it is important to either have a senior automation engineer as a part of your project, even if only at the beginning to set ground rules and ideally to track the quality of your test automation implementation. In other words, it is our belief that automation done right is vital to the growth of any app agency, and particularly helpful to app agencies that focus on client projects on an ongoing basis.