Introduction and Background

Performance is the utmost factor in testing a web application as it directly impacts the end user experience. Performance testing measures system throughput and latency with a variety of concurrent users over long periods of time.

As the technological advancements are leaping bound, more and more businesses are turning to cloud computing to cut down infrastructure costs and making their business more agile. This increasing adoption of cloud in mobile and website applications is demanding on businesses to implement a thorough performance testing before the application rolls out in the market. The cloud-oriented performance testing helps to overcome the challenges of a test strategy that relies entirely on traditional performance testing methods.

An application needs to be tested thoroughly before it hits the market. It has to be evaluated for speed, stability, and scalability under varying load conditions. It cannot go live without addressing any issues related to poor usability, system failure, functional discrepancies, and many other. Performance testing helps in mitigate risks of going into production. Let’s first understand a bit about cloud.

Cloud

Cloud is a pay-per-use kind of model which you can scale whenever you want. You can demand the additional resources required and it will be allocated to you. With a high degree of location independence, virtual resources are dynamically assigned and reassigned according to the customer’s need.

Some of the Cloud metrics which are important for a successful Cloud implementation are storage, processing, bandwidth and number of users accessing it at any given point. Elasticity is one of the major factors for the cloud infrastructure as it defines the cloud’s ability to deal with varying load variations by adding more resources during high load or consolidating the users to fewer points when the load decreases, all in a live cloud setup. scalability, Availability, Fault tolerance and Reliability are the other factors which defines a good cloud infrastructure.

Cloud Scope and Service Models

A public cloud is managed by a third-party service provider and its access is by subscription. A public cloud provides a bunch of business processes and infrastructure services on adaptable price-per-use basis.

Private clouds are used by a single organization. They offer most of the benefits as public clouds, and they give greater flexibility to the organization. A private cloud can retain and customize the ability to standardize the best practices of an organization. Service Level Agreements (SLAs) are negotiated between the provider and consumer. Performance testing and capacity management can be customized according to the needs of the consumer.

Many enterprises cling to both public and private cloud models by integrating the two approaches into hybrid clouds. Certain business and technical factors usually drive the decision on which cloud deployment model to use and each model can have considerable implication for performance.
The service stack provided by the cloud can vary. Here are the layers, starting at the bottom:

• Infrastructure as a Service (IaaS): In this model an enterprise can outsource the equipment used to support processes and networking components. The equipment is owned by the service provider. The provider is solely responsible for its maintenance. Here the client can pay on a per-use basis.
• Platform as a Service (PaaS): This model facilitates consumers to deploy applications that are created using programming languages and libraries onto the cloud infrastructure. Here the consumer need not manage the cloud infrastructure but has complete control over the deployed applications and certain settings for the hosting environment.

• Software as a Service (SaaS) is the facility offered to consumers to use the provider’s application running on cloud infrastructure. A variety of client devices can access the applications through a thin client or program interface and the consumer need not manage the underlying cloud infrastructure.

Cloud Testing

Cloud Testing is a term which is used to define testing which is done using cloud infrastructure i.e we do not need to install hardware’s or any resources locally and we can use the on-demand cloud infrastructure for our testing. In case of performance testing, the use of cloud testing makes it easy to create the environment for test. Overall it reduces the time and cost by simulating the users across the world.

Cloud-Based Performance Testing

Cloud based performance testing allows you to test your application across several platforms without breaking the bank. Testing teams can deploy testing scripts to pre-configured cloud-based load generators in order to reduce required resources and efforts. In addition, it helps in stimulating load tests containing numerous concurrent users from different geographies. Performance testing in cloud is quite different to testing in non-cloud setup and right strategy is essential.

Why cloud in performance testing?

With massive increase in Internet traffic, in terms of number of users as well as the size of data, an application may get hits from all corners of the world. Cisco reports that total data centre traffic will grow from 39% in 2011 to 64% in 2016. In order to tackle all the users, the performance of the application needs to be measured in peak traffic. So it is essential to test immediately to higher concurrency level: 

Performance testing in cloud infrastructure can help test engineers to take the right step in evaluating more effective, large scale and realistic tests.

Steps – Cloud Performance Testing

Challenges and Limitations

The testing model involves various components such as the performance testing tools and the system under test that are deployed ‘on premise,’ within a local environment. In most cases, performance testing is done in a lab environment. This lab is normally a subset of production environment. The lab requires a copy of infrastructure, servers, hardware and software from the production to host the system under test, and the performance testing tools. Setting up such a lab requires considerable investment and effort, and the management of overheads such as maintenance costs. An organization setting up such a lab will need to hire administrators to maintain and support the lab. This may further involve additional training costs for the resources.

The other important aspect we need to consider is that the on-premise approach does not measure up to the actual replica of the geographical distribution of users and their behavior. It can answer the queries regarding performance or capacity with a good degree of accuracy and highlight the main performance bottlenecks. However, we may actually ignore certain aspects such as network latency and behavior of network infrastructure. So we might not be testing the actual user experience. Because of the versatile demands of the current market, its complexity and the dynamic nature of applications, performance testing has become a challenging task.

These are some of the issues with traditional on-premise model. Cloud enabled performance testing can address the above issues.

Strategy – Cloud Performance Testing

Moving performance and load based application testing to the cloud, brings the cloud benefits to the Software testing by lowering capital and operational costs, and also support for distributed development & testing teams. In an environment where it is absolutely necessary to build the right strategy for testing applications in the cloud, performance testing forms a key aspect of any cloud testing strategy, where the application has to be tested for scalability, accessibility, availability, and security while being hosted on the cloud. Cloud-based testing lets you get the maximum by utilizing a cost-effective means of testing applications at scale. It also allows you to simulate load tests constituting millions of concurrent users coming from multiple geographical locations.

Below are the types of performance tests you can perform on cloud:

• Stress test over Cloud
• Load & performance test over Cloud
• Browser Performance testing over Cloud
• Latency testing on Cloud
• Targeting infrastructure test on Cloud
• Failover test on Cloud
• Capacity test on Cloud
• Soak test on Cloud

In Cloud performance testing we use cloud-based infrastructure for conducting our tests. Cloud performance testing not only helps us to deploy our application over the cloud, but it also supports to generate the user load across different geographical locations without setting up any additional components.

We need to consider Storage, Processing, Bandwidth and the total number of users to successfully implement the cloud environment.

We can set up the performance environment in any of the three ways based on our project scope.

• Deploy application in a local lab and load generators on the cloud.
• Deploy application in the cloud and load generators in the local lab.
• Deploy both application and load generators on a cloud.

The cost effectiveness is one of the major factor which is pushing the cloud computing. The right test strategy should focus on the motive behind moving the cloud, including cost savings, infrastructure access ease, reduction in cycle times, choice of public, private or hybrid cloud etc. The strategy should also clearly mention the type of tests to be performed in the cloud, do you want only Soak tests, or capacity tests or all the ones mentioned above. There are certain risks associated with the Cloud and that should be kept in mind before taking the final decision. Building a test environment, infra requirements for the tests and the required testing tools, bandwidth, hardware and software, etc. are the other parameters which should be part of your test strategy.

Security, quality, reliability, latency, bandwidth etc. should be carefully looked upon if you are going with any external cloud service provider.

After all the setup is done, we finally need to execution of tests. For performance tests, it becomes typical to vary the load with varying user’s demands and many other factors, so this should be tested according to the defined test strategy. Some of performance tests explained below:

#Stress Test

Stress test is conducted to determine the application’s behaviour beyond breaking point. Simulators are used to create intense load conditions but the cost of building such on-premise testing environment is too high. Therefore, performing stress testing in the cloud offers an affordable and scalable alternative to businesses. Extreme care should be taken while performing stress testing on the public cloud as it is shared by multiple users.

#Load Test

Testing teams must gather all the information from cloud provider on load statistics of other customers sharing the cloud. This will help them gain important insights regarding the response time while considering other applications hosted on cloud. Run multiple load tests against varying load conditions and keep track of maximum, minimum and average response time in order to get exact response time for the application.

#Test for Scalability and Elasticity

Elasticity testing is performed to determine if the application performance meets the defined SLAs (Service Level Agreement) or not and how much it is further scalable. This approach includes increasing the load to cross the threshold and check the scalability of application as per the load applied. Cloud is a pay-per-use model and therefore, implementing elasticity testing in cloud can help businesses to validate the financial invoice based upon the infrastructure consumption

#Volume Test

Volume test is performed to test the application against the scalability of virtual environment. There are numerous memory monitoring tools which testing teams can use to monitor the virtual memory.

Benefits – Cloud Performance Testing

All levels of testing could be performed in cloud infrastructure, but performance testing benefits greatly from cloud environments.

Flexibility

Different levels of tests can be executed on discrete environments at the convenience of an enterprise. Performance testers no longer have to wait until the end of the testing phase in order to move to a production-like environment for their performance and stress tests. Instead such an environment can be brought into action at will.

Simplicity

The cloud model provides a new level of simplicity in the form of bug fixing environments that can be launched as quickly as the configuration can be put in place.

Comprehensive Testing

End-to-end tests for more generic processes can be performed in the cloud. All the necessary components can be published in the cloud to create the complete chain of systems. In this manner the overall business process can be tested;

Cost Reduction

Cloud environments could be enabled and disabled at will, reducing the cost of environmental management. Cost reduction is the major factor influencing companies to choose Cloud. As per IDC survey reports, economic benefits are the key drivers of cloud adoption.

Cloud Testing leverages the cloud infrastructure, minimizing the unit cost of computing and increasing the efficiency of performance testing. The report on cloud enabled testing service providers reveals that the cost savings usually range from 40% to 70%.

Cleaner and Greener Testing

It is apparently true that cloud computing capabilities make it significantly greener than traditional models and this is true for testing process. By just sharing cloud resources for their test infrastructure, enterprises can use IT resources on demand and eliminate waste. Consumers using cloud infrastructures can minimize energy use and deliver environmental savings in carbon dioxide of around 55%.

Greater Control

Cloud-based environments can provide greater control on test execution, analyse application performance and find bottlenecks while the tests are running. Cloud model allows test engineers to ascend from a few thousands to millions of concurrent users to evaluate breaking points.

Conclusion:

The benefit received from cloud testing is incomparable, but there are some concerns associated with cloud performance testing which should also be kept in mind, like the test results may not always be accurate due to the varying performance caused by the provider’s network condition.

At times there could be chances of a service outage from the provider, and you may not always get the same resources. Also, there will be some challenges associated with the migration or the move from traditional method to cloud.

References:

https://www.cigniti.com/blog/strategy-for-the-performance-testing-in-cloud/

https://www.softwaretestinghelp.com/cloud-performance-testing/

https://www.greenhousedata.com/blog/best-practices-for-load-testing-cloud-applications

http://www.agileload.com/agileload/blog/2012/11/15/cloud-performance-testing

https://azure.microsoft.com/en-in/overview/what-is-cloud-computing/