Popular Posts

Saturday, January 5, 2013

Effective Test Automation and WSO2 System Test Framework

The automated tests are an absolute necessity in agile software development. However,  the true benefits of test automation can only be achieved only through a systematic process which should consist of;
  • Focused automation strategy
  • Dedicated teams to maintain and manage test automation infrastructure
  • Firm commitment by everyone in engineering NOT specific to a separate testing team
For the past few years, we tried to build various types of automation test frameworks as part of WSO2 Carbon middleware platform. Our first attempt was to build a Selenium based test framework back in 2009. We were able to use that for four release cycles in that year. Later on, the drastic architectural and UI changes made the framework unmanageable hence we could not continue use it.

WSO2 System Test Framework (A.K.A WSO2 Clarity) came in to picture during 2011. The new test framework development was handled by a separate test automation team headed by Krishantha. The team overcame many challenges and worked like crazy to build a test framework which would serve not only for one or two release cycles but as a long-lasting test automation solution.

As I mentioned in many of my previous blog posts, it takes time to achieve the real advantages of test automation. The one most important thing is, everyone needs to be patient. You cannot achieve greater test coverage in overnight. WSO2 Clarity test framework is primarily driven through admin web service APIs. There is a little chance to break tests in between releases when comparing to traditional UI oriented automated tests.
The future releases of WSO2 middleware products and cloud services will evaluate the success or failure of the new WSO2 Clarity test framework. But personally, I'm experiencing the true value given by the new framework in day-to-day work which are carried out at WSO2.
WSO2 Clarity framework is an integral part of WSO2 continuous integration system (CI). The tests are automatically executed as part of the build process. However, we usually have many requirements to execute tests externally instead of gluing into build system. For example, we patch the products, we do configuration changes such as change the underlying OS, JDK or DBMS etc.. In such cases, a test framework which is tightly coupled with build system does not help much. Because of that, we wanted to have a mechanism to launch tests externally from the maven build. On the other hand, we wanted to have a simple mechanism to run tests without building massive Carbon code base.
Thanks to Krishantha and test automation team,  binary deliverable of Clarity test framework has been provided as a solution for this long outstanding requirement. A preview version of it can be downloaded from https://svn.wso2.org/repos/wso2/people/krishantha/wso2clarity-1.0.5.zip

This will surely help anyone who uses WSO2 products/services not specific to internal testers and/or developers. How?
Suppose, you download the latest version of WSO2 ESB and want to customize default settings (e.g:- you simply want to deploy the product in IBM JDK and change the default H2 registry database to DB2). Now, you want to make sure there are no adverse effects due to those changes. You can download the Clarity binary and launch all ESB integration/system tests. The test runner is completely ANT based and you just need to have Apache ant. Update the clarity configuration file by modifying the host, IP etc (as mentioned in INSTALL.txt inside Clarity binary distribution) and run the tests you need.
Note that, we still have a preview version of Clarity binary. This will eventually be part of product downloads.

No comments: