iOS 13 testing challenges

The arrival of Apple's new operating systems, iOS 13 and iPadOS 13, represent big releases with multiple updates and features that represent good news for consumers, such as the increased security and privacy features, enhanced user experience, and consolidated single sign-on. For app developers, it presents some exciting opportunities for innovation.

For anyone involved in ensuring the quality of those apps, however, it takes testing to a whole new level. Plus, if history is any indication, adoption of the new iOS will be fast, so there is no time to lose.  

These iOS changes will heavily impact test lab coverage to include the new operating systems. Teams will need to have a focused test matrix to ensure testing of existing iOS 12 devices, which currently represent the lion's share of the market. Plus, Apple will discontinue support for many legacy devices, but of course, apps will need to be maintained.

All this makes for a potentially very complex, labor-intensive test environment, with much wider test lab coverage. Test data will grow significantly with the rise of the new platforms and OS versions. Having full control and visibility into the test reports will be critical to providing fast feedback to developers and test engineers.

Fortunately, there are some steps that can be put in place — and relatively quickly —  to manage the ever-growing mountain of tests. First up is to make sure that a robust test matrix is in place, covering iOS 13, iOS 12, and legacy platforms such as iOS 11 and potentially even older ones, depending on the devices that need future support.   

Next, carry out a comprehensive regression (functional) test audit across all platforms, legacy and new, and then once that is done, create a new lab configuration that supports them all. Keep in mind the differences that are required based on the audit of test code using either Appium or XCUITest, the two de facto frameworks for iOS testing.

iOS 13 testing challenges

Test automation scripts that were developed for iOS 11 and iOS 12 will need to be properly adjusted to also work on iOS and iPadOS 13 to accommodate the changes to the UI and popup notifications. As the platforms mature, the gaps between iOS and iPadOS will grow and will require a change in the process of ongoing maintenance to the test code for the two platforms, which will have two different release trains going forward.

While testing all new features is important, some are more challenging and deserve priority. For instance, Dark Mode and Apple sign-in not only impact UI across the platforms but these features will not be supported for some legacy devices and OS versions, making the iOS/iPadOS platform more fragmented and hard to maintain. So, it is advisable to continually test the native, hybrid, and responsive web design (RWD) apps with Apple's new user interface, Safari browser, and Dark Mode feature. Automating tests as much as possible will help address the sheer scale of the test landscape. 

Native and other iOS apps must function and behave the same across both the new legacy devices and OS versions. Covering the growing test matrix in an Agile/DevOps reality can only be sustainable with a proper test automation strategy. The UI across the different platforms, the popup notifications, and the performance are different. Covering these across the different permutations is necessary and the more you can automate, the faster teams will be aware of any issues or regressions.

The new iOS also introduces pop-up and security features, so review and consider updating the permissions policy within the app. Similarly, adding the new single sign-on option in iOS 13 will consolidate user log-ins across all access points.

Finally, build an updated performance benchmark for assessing the app and/or website's performance and responsiveness to the new features, such as the faster app launch. Test tools that include smart analytics and machine learning can help drill down into the real issues and business impact of the results.  

It is no news to anyone that apps are only going to continue to become more sophisticated with an increasingly diverse array of brands and models to support. The introduction of iOS 13 represents a challenge and step-change in how apps are tested, but it is arguably a useful catalyst for app teams to review their current test processes and tools, and put in place a strong foundation not just for now, but for the future too.