Popular Posts

Friday, July 22, 2011

PaaS testing

PaaS (Platform as a Service) is an application delivery model which provides services to design, develop, test, deploy and host applications. It plays a key role in cloud computing space similar to the other services, IaaS (infrastructure as a service) and SaaS (software as a service). Google App engine, vmForce and WSO2 Stratos are some of the popular PaaS offerings.
This post is not about detailed concepts of PaaS or cloud computing. Rather, I'm going to look at the testing and quality assurance aspects of PaaS. PaaS testing is not a widely discussed topic. There is no pre-defined model for PaaS testing. We need to explore the features of PaaS and derive an approach for testing PaaS apps. I will go through some basic components based on my experience of testing WSO2 Stratos opensource cloud platform.

Hosted services testing - Minimized deployment overhead

In simplest terms, PaaS testing is about testing a hosted platform. So, there is no deployment and configuration overhead for the test team. Testers are expected to access the hosted services remotely using web browser and carry out functional and non-functional testing.

Lesser test platform combinations

When you test a standalone product or a combination of products, you should try out the possible platform combinations such as different types of clustering setups, multiple DBMSs, Application Servers or various operating systems. However, when you test the same product suite or platform on cloud (PaaS), you are bound to one optimum configuration stack. You have one or two choices. There is no requirement to try all possible platform setups.

Multi-tenancy aspects

Multi-tenancy allows a single application to emulate multiple application instances. With multi-tenancy, a single application can be shared across many organizations. Therefore, series of tests should be carried out carefully to verify the multi-tenant aspects. For example, if company A logs in to a web application hosting service in your PaaS offering and deploy a web app, then the users of company B should not be able to locate it unless company A made it public. In other words, the configurations done by one tenant should not be exposed to other tenants.
These mission-critical aspects are some of the key requirements which should be taken into account in any PaaS testing model.

Performance and scalability

Any hosted service should conform with the SLAs. Similar to a website, services included in a PaaS can be accessed by multiple users concurrently. Therefore, sufficient amount of performance testing must be carried out. The usual usage pattern of standalone technology platform can be dramaticallty changed when it is hosted as a service in cloud. For example, the general use cases of WSO2 Carbon SOA middleware platform are different from the standalone version when the platform is exposed as a service. WSO2 SOA middleware platform consists of 12 different products and all of them includes a management UI for administration purposes. In non-cloud based deployment, the multiple, concurrent users access to the management UIs are minimum since it is very rare that hundred of administrators accessing management console at once.
However, if this platform is hosted as a service in cloud (PaaS), then the concurrent user access to the management UIs are a highly desirable scenario.

It is a requirement that the hosted platform should be able to handle load seamlessly without affecting consumers. Auto-scaling is a key feature of WSO2 Stratos cloud platform. When new resources are needed, WSO2 Stratos transparently adds services and when load goes down, WSO2 Stratos automatically brings services down. Testers should ensure, with the load, the new EC2 instances spawning up and down correctly. The tools like hybridfox are very useful in these situations.


If your PaaS provides users with the ability to store their data in cloud and various database centric operations, then the testing will be much complicated. As a tester, you must ensure the availability of data sources, accessibility and provisioning aspects. You must also take extra care that, if the data are geographically distributed in cloud, then it adheres to the legal requirements of the users of your PaaS.


Security is the utmost important aspect of any cloud offering. It is recommended to have a separate group of security testers who do penetration testing and ethical hacking in order to ensure secure infrastructure of your PaaS. WSO2 Stratos cloud platform allows users to deploy their webapps, web services and various custom code. Therefore we must ensure that no vulnerable code are deployed and a code deployed by one tenant does not affect the operations of others or whole platform.


Public PaaS offerings are usually based on virtualized infrastructure such as Amazon EC2. Therefore, a full functional test cycle should be carried out on EC2 VMs inorder to make sure that there are no regressions due to virtual servers.

In addition to the above key areas, PaaS testing model should consists of the general web application testing aspects such as cross browser testing, accessibility testing, localization testing etc..

Monday, July 18, 2011

Data driven testing with Jmeter user parameters

This is a follow up to one of my previous posts which explained data driven web service testing using CSV config element in Jmeter. There, we used CSV file to read input data for SOAP/XML-RPC sampler.
In this post, we will look in to using User Parameters pre-processor element as the data source instead of a CSV file.

Step 1

We are going to use the same web service which we used in my previous post, temperature conversion service. Please add the SOA/XML-RPC sampler, the SOAP request and the necessary thread group as described in step 1 and 2 of that post

Step 2

Lets parameterize the payload of SOAP message so that different requests will be sent to the service with each test run. Instead of reading data from a CSV file, we can add a User Parameter pre processor element in Jmeter test plan.

Right click on the thread group of your jmeter test plan and select Add --> Pre Processors ---> User Parameters
Click on Add Variable and specify celcius as the name of variable. Add few users and enter celcius values for each user as shown below.

Step 3

Now, parameterize the payload of SOAP as follows.


Step 4

Increase the thread count corresponding to the user count in your user parameters pre-processor element and run the test. You will notice that the Celcius figure will be varied in each request.

Based on your requirements, you can select either CSV config element or User Parameter pre-processor element for data driven testing. If you have large number of variables to be parameterized, CSV config is the best option.