What major organizations want to avoid?

“Home page is currently unavailable”; “Your transaction cannot be processed now. Please try again later”; these are some of the error messages usually end-users encounter during peak business hours. The most annoying message is “HTTP Error 503 – Service unavailable”. Enterprise business organizations, predominantly verticals like Banking, Retail, Insurance, Telecommunications, and Healthcare applications have a critical role in day-to-day activities in terms of transactions. Even small amount of timely loss impacts their business growth and the stock market value.

Major US bank introduced a new $5 debit rule which charges monthly fee from their customers. A day after the rule implemented, users were unable to access their bank accounts and they can see only temporarily unavailable error. Major US Retail chain’s website crashes again and the outage extended for 2.5 hours which impacts on their stock market price.

During above instances, enterprises won’t explain the root cause of the issues nor their technical problems. But it is a crystal clear that any organizations want to avoid the outage/downtime during the business hours.

Performance Testing – An Overview

Before deploying the code into production environment, it is important to validate the functional and non-functional aspects of the application. Performance testing is one of the important types of testing under non-functional testing group.

Performance testing is the super set term. According to IEEE definition, performance testing is conducted to evaluate the compliance of a system or component with specified performance requirements.

Load testing is the subset of performance testing. Load testing validates the applications’ capacity. E.g. how many users can do transactions concurrently, how many orders application can process per minute etc. the process of inject the demand/load on the application and measuring its response.

Introduction to Load Testing and Beyond

How much pounds of weight you can lift in gym? 200 pounds, 300 pounds, 1000 pounds? Every person has a breaking point, mentally, physically and intellectually. The same principal applies in load testing as well.

How much load can server handle at a specific point of time? 100 users, 200 users, 10K users? Every server has a breaking point.

The main objective of load testing is to study the behaviors of the application under test (AUT) under specific load circumstances.

Load testing helps engineers to understand the applications’ behavior and availability.

• Load testing addresses following queries:
• Does the latest codes shows desired performance?
• Does the new change in the hardware configuration perform well?
• Does the new business rule or feature that implemented makes significant effect?
• Does the upgrade in the environment works under limits?

Load Testing Process

Now we will see about load testing process and how to study the behaviors of the application during load test. Following are the key phases of load testing any application.

Identifying the Critical Scenarios and Metrics

Identifying critical scenarios is an inception and critical phase in load test process. Following are the various factors play a vital role in this phase: expertise in the application domain, business rules and requirements, technology etc.

Following the important metrics that needs to be collected and monitored:

1. Response time – how fast your application is?
2. Throughput – how many orders your application can process i.e. transactions per second?
3. CPU Utilization – What is the percentage utilization of your CPU?
4. Memory Utilization – What is the percentage utilization of your memory?
5. Network Utilization – What is the percentage utilization of your network bandwidth?
6. Disk Utilization – What is the percentage utilization of your disk?
7. User load – What is the maximum number of concurrent users your application can withstand?

Also there are different kinds of metrics which needs to be identified such as Client-side metrics, server-side metrics, business metrics, network metrics etc.

Following are the examples of critical scenarios for banking applications

S. No	Business Scenario	Metrics
1	Login	Response time should be < 2 seconds for 500 Vusers
2	Fund Transfer	Response time should be 100 Funds Transfers should happen per minutes
3	Apply for Credit Card	Response time should be < 2 seconds for 300 VUsers200 Credit Card application should get processed in 2 minutes

Designing the Work Load Model

Workload model helps test engineers to design the test execution as per real time scenario. In production environment how the application is getting utilized. In order to inject production like pattern in the test, effective work load model design is required.

Below is the one of the workload model type which will be used for load testing. Graph will be plotted Users vs. Time. There are three phases: Ramp-up, Steady State and Ramp-Down.

 

• Ramp-up:            In this stage, users will be injected gradually to the server for the specific duration e.g. 15 minutes and e.g. scenario Login

• Steady State:     In this stage, all the users will be gathered at one point called rendezvous point and starts their business transactions for 60 minutes and for the scenario Search

• Ramp-down:      In this stage, all the users again gathered at one point and start logging off from the application for the duration of 15 minutes

 

Designing the Test Scenarios

When effective work load design is available, next step is to start generating the scripts. First step in generating scripts is to record your business transactions. Load testing tools supports recording feature which enables test engineers to record their actions.

Once the business transaction is recorded, immediate is to enhance the scripts. Following points should be considered in script enhancement phase:

• Think time
• Correlations
• Runtime settings
• Iterations
• Comments
• Actions
• Data Parameterization
• Checkpoints/Assertions
• Error handling
• Naming conventions
• Run logic
• Speed/Browser simulation

 

Once the script is enhanced, it is mandate to perform unit testing to baseline the test script.

Test Execution by Simulating Load

Script execution is a critical phase in load testing process. In this phase the defined load will be injected to the server and its behavior will be recorded and analyzed. Poor load injection will not help in analyzing the applications’ performance.

Before injecting load to the server, following points should be consider:

1. Test Environment configuration – this should be a mirror of production environment
2. Small is BIG – warm up your server by injecting small load for shorter span
3. Slow and Steady – inject the load slowly and steadily
4. Load Injections Server – check the availability and the ability of the load generation server
5. IP Spoofing – validate the IP Spoofing settings
6. Reports – enable report generation settings such as various formats and options

Test Execution Phases

The test execution consists of pre-initiation, execution and post testing activities.

The pre-initiation activities include

• Restore of database instance
• Restart all load generator machines
• Warm-up runs of scripts to confirm the functionality availability

Following activities performed during test execution

• Execution of scripts
• Real time monitoring

The Post Test activities include

• Saving the test results
• Collate and analyze the test results
• Save the application and database server logs
• Collate all performance counters monitored
• Test Result reporting

Report Analysis

Identifying performance bottlenecks is a blend of art and mathematics. Basic knowledge about statistical terms and correlation of various charts is required for an effective analysis. Following are the key steps gets involve in report analysis:

1. Analyzing the results by comparing with previous results/iteration
2. Analyzing the results by comparing with defined metrics

Provide the following details in your final report: Average, Minimum, Maximum, Mean, Median, Mode, Standard Deviation, 90^th / 95^th percentile, Error percentage, Throughput, Utilization of CPU, Memory, Disk and Network and Histogram analysis (if required). Apart from above metrics, you can add additional performance counters to your results.

Benchmarking and Recommendation

This is the final phase of load testing process. Results will be compared with industry standard benchmarks. If there is a conflict, root cause analysis should be performed with the help of developers/architects and the necessary recommendations should be provided.

Also it is important to document all the learning and issues encountered during load testing for future reference.

Load Testing Entry Criteria

It is important to define and validate the load testing entry criteria before start testing. Following are the points which needs to considered:

• Regression testing should be completed and the environment should be stable without any defects
• Test Strategy & Plan should  be reviewed and signed off by the architects and clients
• The database set up should have been completed
• The testing, monitoring tools required should have been installed before script development and test execution phases
• The testing team should have access to testing tools and server machines

All the project teams such as development team, environment team etc. should be aware of performance testing cycles to keep the servers in quiet mode during testing

All user processes that consume more memory and CPU need to be turned off from all load generator consoles

Load Testing Exit Criteria

After successful testing gets completed, following points should be considered:

• All the performance test cases defined by the non-functional requirements document should be validated
• All the test scenarios defined in the test strategy document should have been executed successfully
• The test scenarios should simulate real-business mix of transactions
• Response time should be captured for all transactions from each test run
• The test results should include all necessary graphs for system resources and application metrics specified in the strategy document
• All the test types defined in the test strategy document should have been executed successfully
• Transaction failure rate of 2-5% is accepted at peak load

Load Test Data

Load test data plays a significant role in load testing. The databases in the performance test environment should have data volumes similar to production. For applications which are in production already, a copy of the latest data should be taken from production to the performance testing.

For applications that are yet to be in production, effort should be spent for test data set up. Privileges and access permissions should be provided.

Risks and Mitigations

Risks and Mitigations should be in place to avoid chaos during load testing. Following are the points should be considered:

Conclusion

Load Testing helps to study the behaviors of application under the specified load. Performance bottlenecks which causes performance degrade can be identified during load testing. Following are the basic methodology for load testing: identifying critical scenarios, effective work load model, test script design with enhancements, load simulation, collecting the performance metrics and benchmarking & recommendations.

This post is also available in: Anglais