What you need to know about customizing Software as a Service

It’s a given for web and mobile application development that performance testing is critical to application success. Yet, for organizations wishing to customize third-party (3P) software as a service (SaaS) rather than running it out of the box as written, performance testing can be problematic. For some types of performance testing, it is nearly impossible due to limitations imposed by the SaaS platform developer. (Author’s note: To avoid confusion, for the remainder of the article I will refer to SaaS platform developers as “parent companies.”)

There are ways to get around some of the impediments. Where that is not possible, teams can run calculations and develop simulations that give them a reasonable amount of confidence in their work. The alternative - not performance testing - is unacceptable, as it almost always leads to problems in production.

As a performance testing consultant, I have worked with numerous Orasi customers, helping them ensure reliable performance for their customized SaaS implementations. One platform where we are frequently called in to assist is Salesforce, which has strict rules relating to customizations and the testing that surrounds them. 

For this article, I will use Salesforce as an example, describing some of the key impediments to performance testing and offering best practices for overcoming (or at least minimizing) them. Although this information is platform-specific, it will be largely applicable to many 3P SaaS products.

It’s understandable that 3P SaaS parent companies would want to restrict performance testing. After all, their reputations are built on - or destroyed due to lack of—application performance. In achieving application stability and reliability, they must weigh many factors and overcome myriad challenges, especially:

- Customers in an array of industries that have wildly varying user counts and usage patterns.

- Enormous, unpredictable load swings.

- User expectations for consistency of service on virtually every possible device.

To promote stability, when SaaS parent companies allow customizations, they frequently restrict access to the platform. Many develop specialized testing tools for customers to use - and create protocols customers must follow. Salesforce, for example, requires companies customizing the Salesforce platform to submit a performance test plan for advance approval.

Additionally, the Salesforce corporate wisdom on customizations is essentially, “Write clean, bug-free code, and your customizations should work just fine.” Performance testing is surest way to bring such theory into practice. It ensures your customizations provide value to your end users and helps you avoid bumping against Force.com governor limits in production.

With 3P SaaS, some software teams persuade themselves that testing is less important, because they are working within an environment that has been tested by others. This may be true with the vanilla versions of a SaaS system, but it is completely inaccurate when customizations come into play. For 3P SaaS products, the importance of testing customizations cannot be overstated.

With a well-built product like Salesforce, configurations, such as adding objects/fields, building workflows and reports, and creating validation rules, should be reasonably safe. That is fortunate, because some companies let users perform these configuration themselves. (This is not a best practice - users should be encouraged to use out-of-the-box features when possible. Administrator-level approval should be required for specialized configurations.)

Customizations bring a much greater degree of risk. Precisely because Salesforce is so easy to customize, it illustrates perfectly how much trouble enterprises can get into when their teams start tinkering. Following is a list of the primary customizations allowed in Salesforce:

- Apex classes and/or triggers

- Visualforce pages/components

- Visualforce emails

- Integrations with 3P Systems

- Creating Lightning components

- Building Sites.com/Force.com sites

- Using Cascading Style Sheets to alter page appearance 

- JavaScripting to add capabilities

For any of these customizations, individual snippets of code can be bug free, and they may pass unit testing, but that doesn’t mean they will perform as designed within the system as a whole. (Another Orasi author discussed this issue last month. You can read that article, here.) Even minor changes can cause a ripple effect across a system and break other components that are not obviously related.

Furthermore, customizations are often done at the department level without consideration for their overall impact. Optimally, each customization should be considered as part of a whole, with regression and performance testing conducted after each customization. Otherwise, as organizations layer customization after customization, the environment will become disorganized and support will become next to impossible.

With Salesforce and many other SaaS platforms, I see four core issues that can significantly impact the activities of software teams

- Multi-tenant environment: Multi-tenancy is one of the biggest challenges of testing 3P SaaS customizations. Without robust performance testing, teams have no way of knowing how or if performance will be diminished by heavy app server loads from other users outside the testing company.

- Scheduling challenges: Developers and testers customizing 3P SaaS products frequently face scheduling challenges, because the parent organization controls when they can have access to an approved sandbox environment.

- Governors: Salesforce and other SaaS parent companies use governors to restrict the amount of load any team can place on the live system. Salesforce does not allow stress testing, at all.

- Device and browser testing: A key selling point for 3P SaaS parent companies is often very broad devices and browser compatibility. As a result, teams must test their customizations on as many devices and browsers as possible. In the case of Salesforce, the debut of the new Lightning GUI has complicated matters even more.

Ideally, I would hand you an easy solution for mitigating the impacts of 3P SaaS testing, but there isn’t one. The only way to ensure good performance with Salesforce and 3P SaaS customizations is to have a mature plan for approving, managing and testing customizations, individually and as part of the entire system. Teams must be effectively managing test data, and the organization must be making use of both automation and virtualization. Teams should also be using correctly developed scripts and performing proactive script management.

Ideally, teams should have access to a dedicated, in-house testing environment, or they should license robust, cloud-based load generators where they can perform performance testing. Teams should also take advantage of advanced tools such as HPE LoadRunner or HPE StormRunner Load. These solutions make it feasible for teams to run tests—under every possible scenario—for a multi-tenant, globally adopted platform. Testing scenarios should reflect not only the user base for the current release but also the expected growth over the next year.

Finally, your organization should have a quality assurance program that enables you to make productive use of raw test data and turn it into actionable KPIs. Then, that information should be used to refine your development and testing efforts. (If you aren’t familiar with such programs, read about the methodology here.)

If your company isn’t already engaging in best-practices customization management, development and testing, I urge you to become an evangelist for proper testing. Advocate for an evaluation of your company’s current development and testing environments.

Ask management to devise a plan for improvement, calling in outside assistance if needed. Companies no longer must build their own best-practices testing labs. Hosted, cloud-based testing services offer an affordable alternative for performance testing at the levels required by 3P SaaS.

If you are not following the recommendations outlined here, you are operating on borrowed time. A comprehensive performance testing plan will help you avoid the post-release performance failures that can occur with layers of inadequately tested customizations in a multi-tenant SaaS deployment.