What you Might be Missing When You Try To Test for Everything

As you approach a load test, you may be tempted to focus on specific elements, such as the number of threads you’re using, response times or hits. However, it’s important to look at the process itself from a high level as well. By doing so, you’ll set yourself up for a more effective and successful testing sequence. 

Hopefully you’re in the habit of running frequent load/stress tests. You likely have a mental checklist of questions that run through your mind, including:

- How many threads per JMeter engine should I use?

- Can my engine handle 10% more threads?

- Should I start with the maximum load, or add as I go?

All of these questions are important and should be carefully considered - but what about the load test itself? Have you considered how the actual procedure will be managed?

It’s easy to get so wrapped up in the technical details that you lose sight of the overall picture. Even when using BlazeMeter’s self-service load testing solution - which is designed to simplify performance and load testing for developers and QA testers- it’s still vital to take control of the process management to ensure each test runs smoothly. 

Load testing not only tests your servers but also the strength of your team’s ability to react. Everyone needs to be in sync in order to quickly analyze the data, identify the bottlenecks and even solve them (if possible) during the test.

For example: the developers/QA testers should create and share the script with all departments involved in the test, to make sure that everyone understands the scenario and how it can affect their particular system. An n-tier application is made up of several disciplines, each one requiring the same level of attention - be it database, front-end, back-end, or monitoring services. 

It is imperative that the staff understand the upcoming load test scenario, regardless of their knowledge of JMeter and/or load testing, so as to create the most agile environment for the duration of the test. If the staff clearly understand what is happening when a bottleneck is hit, it’s much easier for them to analyze the data and quickly tend to the issue. If they aren’t on par, there’s a much greater chance that they won’t be able to overcome the bottlenecks and delays will occur.

 

Once the ins and outs of the infrastructure are understood, further managerial tasks are needed. For example: sending out updates and involving your operations team and third parties like Akamai or Amazon to avoid complications such as blocking, shaping, or even inadvertently breaking a legal agreement during the test. While they may seem excessive, it’s really worth taking these supplementary steps before starting the load test to ensure the best results. 

When it comes to the technical aspects of load testing, as detailed in How to Run a Load Test of 50k+Concurrent Users, it’s important to supply everyone involved with a test scenario outlining the exact processes of the test. By taking this step, even staff members who are not JMeter experts will understand that during the test, they may experience issues like an excessive number of log-ins, 'add to cart' requests, image usages, database queries, and so on. The scenario should be simplified and easy to read for everyone involved. It should include a defined scaling process and explain that the ultimate goal for the number of users may only be met within a number of load tests.

The technical approach usually includes building the script (and its data) to address the specific scenario, testing it locally and verifying that it actually works in JMeter. Once the script performs as expected using the SandBox feature, you should evaluate how many threads can be applied to a single load engine without crashing (ensuring the load engines themselves won’t be the bottleneck). Then, the script will be scaled to a cluster of engines, and more clusters can be added as you go. Learn More.

Important points to consider:

1. Does your data need to be unique per engine? If so:

- Set up a different CSV for each engine

- Use the “Split CSV file” feature to unique files and distribute among the load servers

- Use JMeter functions like __InstanceID,__RandomString(), __threadNum() and __UUID() to add unique data to your test (e.g: email registration)

2. Does your scenario have data manipulation in real time? If so:

- Use BlazeMeter’s real time feature

- Control RPS/s with multiple engines using with the Limit RPS feature

- Use the “dynamic load” feature to add more horsepower during the test

3. Do you use timers? If not, you should (even if it’s just 300ms) because every user has some think time between pages.

- You can use the Throughput Shaping Timer or the Constant Throughput Timer to control your test hits/s

From a management perspective, of course, every company is unique. But most share the same key criteria, these include:

1. Get everyone on the same page before you schedule the test -  Call a meeting for your department, giving about two weeks lead time. It allows everyone involved to see the expected script scenario and everyone can discuss issues like who to alert about the test and possible interferences.

2.Including a representative from each team during the initial load tests - This will ensure smoother sailing once the test is up and running. In later tests, you might want to schedule the actual environment you’re going to load test. For example: in a staging environment, the VP of R&D needs to know that the environment is going to be used, which may affect updates to production. Whereas, if load tests are run on production environments, a landing page should be created to notify users that maintenance procedures will take place during that particular time.

3.Creating a drawing or flow chart of the actual script - This is an additional measure but a useful one -  after all a single picture says a thousand words. It helps people who aren’t familiar with JMeter or aren’t aware of what’s happening with particular clients or in the front-end to relate and understand what’s expected to happen.

 Depending on the test’s complexity, you can make a flowchart of:

- Your script flow

- Cache warm up (after doing a reset cache)

- Test scenario

- The flow of the load (e.g: 2-hour test, ramping up to 50K within 30 minutes, after 60 minutes adding 25K for another 30 minutes)

- The event success criteria (e.g: Reaching 50K users / 80K RPS)

 This method effectively divides the load test into two parts: the pre-load test, which creates all of the data, and the actual load test itself. Appointing a load test supervisor to announce the various stages of the test will help everyone involved to stay on the same page.

4. Consolidate and share information in real time. Utilizing an organized platform, such as Campfire or HipChat will allow everyone to communicate quickly during the load test, as well as enable the supervisor to control and deliver critical assignments when needed. Additionally, these platforms provide a space for each department to present their conclusions from the test. You can also record the test - which will be a great help when it comes to running future tests and reports. 

5. Schedule the right people for your load test. Make sure you have every discipline you need during the load test including your 3rd parties! You don’t want to start the load test and discover that suddenly your 3rd CDN provider is throttling your requests (or even blocks them due to DDoS policies).  

Some key people you should consider to have with you:

1. DevOp engineer (who will be responsible for the APM data that will help you analyze your application during the load)

2. Security engineer (to make sure the load is not being blocked)

3. A product manager / senior manager – to take management actions if needed

4. Any of your 3rd parties which will be affected by the load test

Taking all of the above into consideration while you plan your main event will allow you to:

1. Get everyone on board on the process and its importance

2.  Probably discover some critical steps you missed and will make to load event more efficient

3. It will allow you to react fast once you saturated your application and perform real time analysis to pinpoint that bottleneck

4. And eventually - have a better chance for an effective and successful load test event