Popular Posts

Wednesday, November 21, 2012

TIP: Error handling in WSO2 ESB APIs when the requested API resource does not match with the given URL template or pattern

When you send a HTTP request to an API in WSO2 ESB, if the requested API resource URL does not comply with the API resource definition, ESB returns HTTP 202 accepted response to the user. Due to  this behavior, the client is unable to distinguish if they have made an error. We can use the following procedure to return a fault to the client in case of non-matching API resource is requested by client.

Step 1:

Suppose, we have an API similar to the following. Note that, the target web service is a JAXRS web application (jaxrs_basic) which is included in WSO2 Application Server by default.

<api xmlns="http://ws.apache.org/ns/synapse" name="jaxrs" context="/jaxrs">
   <resource methods="GET" uri-template="/customers/{id}">
               <address uri="http://localhost:9764/jaxrs_basic/services/customers/customerservice"/>

Step 2:

Send the following HTTP GET request to the above API. (You can use a soapUI web testcase as instructed in one of my previous blog posts).

GET http://localhost:8280/jaxrs/customers-wrong/123

Step 3:

You will get HTTP 202 accepted response because the request URL does not match with the resource url-template defined in the API.

Step 4:

Now, add the following fault handling sequence in ESB. Once the API receives a request which does not match with any of the resource url patterns/templates, this sequence will be invoked. Make sure to name the sequence as _resource_mismatch_handler_

 <sequence xmlns="http://ws.apache.org/ns/synapse" name="_resource_mismatch_handler_">
         <tp:fault xmlns:tp="http://test.com">
            <tp:type>Status report</tp:type>
            <tp:message>Not Found</tp:message>
            <tp:description>The requested resource (/$1) is not available.</tp:description>
         <arg xmlns:ns="http://org.apache.synapse/xsd" expression="$axis2:REST_URL_POSTFIX"/>
   <property name="RESPONSE" value="true" scope="default"/>
   <property name="NO_ENTITY_BODY" action="remove" scope="axis2"/>
   <property name="HTTP_SC" value="404" scope="axis2"/>
   <header name="To" action="remove"/>

Step 5:

Send the above HTTP GET request again. This time, you will get a proper error as shown below.

HTTP/1.1 404 Not Found
Content-Type: application/xml; charset=UTF-8
Accept-Encoding: gzip,deflate
Date: Fri, 16 Nov 2012 06:06:51 GMT
Server: Synapse-HttpComponents-NIO
Transfer-Encoding: chunked 

Monday, November 19, 2012

How to enable mail transport for WSO2 data services

We can use mail transport to send and receive messages to and from web services using MIME complaint mail messages. We can configure mail transport for Axis2 services which are deployed in WSO2 Application Server by following the instructions given in the official documentation.
In there, you will notice that, a set of mail transport specific parameters need to be configured in the services.xml of Axis2 service in order to enable mail transport in Axis2 services.
How can we do the same for data services which are deployed in WSO2 Application Server?
This post summarizes the procedure of configuring mail transport for a data service.

Step 1:

Download and install WSO2 Application Server-5.0.0 or a later version. Configure mail transport sender and receiver in axis2.xml configuration file which can be found at CARBON_HOME/repositiry/conf/axis2/axis2.xml (see WSO2 Application Server official documentation on how to configure mail transport sender and receiver in axis2.xml)
After configuring axis2.xml, make sure to restart the server.

Step 2:

Once the server is started, you will see the existing services become faulty with the warnings similar to the following. 

WARN {org.apache.axis2.transport.mail.MailTransportListener} -  Unable to configure the service TestService for the MAILTO transport: Service doesn't have configuration information for transport mailto. This service is being marked as faulty and will not be available over the MAILTO transport.

The reason for this is, though we have configured mail transport globally in axis2.xml configuration file, we have not configured mail transport at individual service level. Thus, if you have Axis2 Services or data services or any other service type, the mail transport will not be enabled for them. You can configure mail transport for specific Axis2 services as instructed in here.

However, data services do not have a services.xml associated with it by default. Thus, we need to follow a different procedure for them.

Step 3:

Locate the *.dbs file in CARBON_HOME/repository/deployment/server/dataservices/ directory. Let's assume the data service is GUEST_T_DataService.dbs. Create a file GUEST_T_DataService_services.xml at the same directory. (Make a note of the name of the file which is in the format; _services.xml)

GUEST_T_DataService_services.xml can be similar to the following. Note the mail related parameters section.

<service name="GUEST_T_DataService">
<Description>This sample demonstrates some of the main functionalities of a dataservice using an RDBMS data source
 <!--mail related parameters-->
        <parameter name="transport.mail.Address">synapse.demo.1@gmail.com</parameter>
        <parameter name="transport.mail.Protocol">pop3</parameter>
        <parameter name="transport.PollInterval">5</parameter>
        <parameter name="mail.pop3.host">pop.gmail.com</parameter>
        <parameter name="mail.pop3.port">995</parameter>
        <parameter name="mail.pop3.user">synapse.demo.1</parameter>
        <parameter name="mail.pop3.password">mailpassword</parameter>

        <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>
        <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
        <parameter name="mail.pop3.socketFactory.port">995</parameter>
        <parameter name="transport.mail.ContentType">text/xml</parameter>


Step 4:

Now, check the dashboard of your data service. You will see mail transport is enabled as shown below. (Restart server if the mail transport is still not enabled for data service).


Monday, October 29, 2012

"Web Services Testing with soapUI" book is published!

It is with extreme pleasure that I announce, Web Services Testing with soapUI, the first book authored by me, is now published! This is a remarkable achievement of my life and I'm really happy to make the first announcement about the new book through my blog.

On 9th of September 2011, Packt publishers suggested me to write a book on soapUI. By that time, I had published many blog posts about soapUI and SOA/Web services testing. I hesitated to take the challenge at first, because I had been extremely busy with my work at WSO2 and I doubted whether I would find any spare time to work on authoring a book. After taking various facts into consideration, I agreed to start the authoring process in November 2011.
Writing a book has been a great experience. I read a lot and gathered a great deal of information. Finally, I was able to complete the book with 332 pages, covering almost all important features of the free version of soapUI.

I always loved to read and write since my school days. My father and mother guided me to read news papers from my early childhood. I still remember, I published my own news paper when I was studying at grade 6 and represented it in many exhibitions at my school. Therefore, I have no others but my beloved family to whom the book was dedicated for!

Excerpt from page 11:

"This book is dedicated to my parents, who have raised me to be the person I am today 
and my beloved wife Thushari and my loving kids, Risith and Nethul."

I acknowledged all who helped me in many ways to make the book successful, as follows.

Excerpt from page 5:

Making a book reality takes many dedicated people, and it is my great pleasure to
acknowledge their contributions.
First, I'd like to thank Packt Publishers, in particular, Kartikey Pandey – Senior
Acquisition Editor, who proposed me to write this book. I'm grateful for all the
help I got from the editorial staff at Packt Publishers in reviewing this book,
specially Hithesh Uchil – Lead Technical Editor and Sai Gamare who coordinated
the progress of writing, by ensuring that I stayed on track.
This book has benefited from a great set of technical reviewers. I'd like to thank
each of them for volunteering their time reviewing drafts of this book and providing
valuable feedback. Specially, my colleague at WSO2 QA team, Evanthika Amarasiri
who carried out in-depth quality assurance process in all chapters by executing
each sample.
I sincerely thank my wife, Thushari for her patience, support, and understanding
throughout the writing process. Many thanks to my beloved parents who raised me,
made me the person who I am today by providing their insightful guidance in all
aspects of my life.
Though I'm unable to name individually, I would like to extend my heartfelt
gratitude to many colleagues at WSO2, who never hesitated to give their support to
the fullest extent, whenever I requested help on various subject matters. I must thank
Dr. Sanjiva Weerawarana, Founder, Chairman and CEO of WSO2, Inc. whose vision
inspires me and guides me to accomplish my career aspirations.
Finally, a big thank goes to the developers and contributors of Smartbear software
for making soapUI the world's best open source web services testing tool.

If you are a part of a team that builds service-oriented solutions or makes use of web
services in your project, and your primary involvement is testing such a solution,
then this book is the ideal reference for you. This book will help you to understand
the common challenges of SOA testing and how soapUI can be utilized effective
manner in testing your applications.
This book would also be a good reference for developers and QA engineers who do
researches and evaluations on various commercial and open source web services
testing tools. Though you are an experienced software professional or a novice tester, you
will quickly be able to learn the most important features of soapUI by following the
simple step-by-step instructions given in this book.

The contents of the book has been classified into 13 chapters as follows.

Chapter 1, Web Services Testing and soapUI, introduces soapUI by giving an overview
of its history, features, and installation of soapUI in your computer. We will begin
our journey towards learning soapUI by discussing some key characteristics of SOA,
Web services and Web services testing in general.

Chapter 2, The Sample Project, introduces the sample web services project which will be
used as the target application for functional and performance testing in the remaining
chapters of the book. In this chapter, we will build a simple web services based
application using Apache Axis2 open source web services framework. The primary
objective of building this sample application is to use it in all demonstrations of soapUI
features. As we will not discuss any topics related to soapUI or web services testing
in general in this chapter, you may skip the details and download the sample web
services project from http://www.PacktPub.com/support.

Chapter 3, First Steps with soapUI and Projects, serves as a guide for getting started with
soapUI projects. Based on one of the web services that we built as part of the sample
web services project in Chapter 2, The Sample Project, we will discuss the schema and
WSDL of the web service in detail. We will use soapUI to invoke the operations of
sample web service and discuss the SOAP requests, responses, and faults.

Chapter 4, Working with Your First TestSuite, demonstrates the basic constructs of a
soapUI project—TestSuites, TestCases, and TestSteps—which prepares you for the
next chapters of the book. We will also look into the validation of responses using
assertions and soapUI properties.

Chapter 5, Load and Performance Testing with soapUI, covers the steps that you
would have to follow when using soapUI as a load and performance testing
tool. We will demonstrate the load test strategies provided by soapUI and the
load test specific assertions.

Chapter 6,Web Services Simulation with soapUI, briefly describes how web services can
be simulated using soapUI. We will demonstrate the usage of soapUI mock services
model and static as well as dynamic mock responses.

Chapter 7, Advanced Functional Testing with soapUI, introduces the testing aspects of
web services extensions such as WS-Security and WS-Addressing. We will use an
improved version of the sample web services project which we built in Chapter 2, The
Sample Project for the demonstrations in this chapter.

Chapter 8, Getting Started with REST Testing, introduces the concepts related to
RESTful web services and how soapUI can be utilized in RESTful services testing.
We will demonstrate the use of soapUI in RESTful services testing by using a
publicly hosted sample web application.

Chapter 9, Testing Databases with soapUI, briefly describes the direct database query
invocations of soapUI. In this chapter, we will discuss the database testing features
provided by soapUI such as JDBC requests and assertions.

Chapter 10, JMS Testing with soapUI, demonstrates the use of JMS in soapUI. By
exposing one of the sample web services over JMS transport, we will explore
the JMS testing capabilities provided by soapUI.

Chapter 11, Extending soapUI with Scripting, introduces the scripting facilities given
by soapUI in order to extend the default behavior of soapUI tests. We will look into
the use of soapUI API methods through Groovy scripts inside our tests.

Chapter 12, Automated Testing with soapUI, demonstrates various automated testing
approaches with soapUI. In this chapter, we will discuss the integration of soapUI
tests with build tools such as Apache Maven.

Chapter 13, Miscellaneous Topics, introduces some useful tools integrated with soapUI
such as WS-I validation tool and the utilities provided by external web services
framework such as Apache Axis2. This chapter also demonstrates the use of soapUI
when testing services by sending attachments.

Download the book from http://www.packtpub.com/web-services-testing-with-soapui/book today and let me know your thoughts.

Thursday, September 20, 2012

How to secure a SOAP web service in WSO2 Application Server and invoke it using HTTP basic authentication with soapUI

In a previous blog post, I explained the steps to deploy a web service in Apache Axis2, secure it with HTTP basic authentication and invoke it using Apache Jmeter.  In this post, we will repeat the same using different set of tools and frameworks which are comparatively easy-to-use and user-friendly.
We will secure a simple SOAP based web service which is hosted in WSO2 Application Server, configure an authentication security policy and invoke the service using HTTP basic auth through soapUI.

Download and install WSO2 Application Server-5.0.0 
Download and install soapUI 4.0.1 or later version

Step 1:

Start WSO2 Application Server by running wso2server.sh startup script. Access the management console url (https://localhost:9443/carbon) and log in with the default administrator credentials (username=admin, password=admin)

Step 2:
Go to the Deployed Services page and you will find "HelloService" is deployed by default. Click on the service and go the the service dashboard. Click on Security to configure a security policy for the service. You will find the default set of security policies in the Security for the service page. In order to protect the service with HTTP basic authentication, we can configure Usernametoken security scenario. Username token security policy secures the service using username and password. Select UsernameToken security scenario and click on Next. Select admin as the user group which need to be authenticated. Click Finish to complete the security policy configuration.

Step 3:
Now, if you check the WSDL of the web service, you will see a WS-Security policy is added to the WSDL. Therefore, if you send a SOAP request, it must include the necessary WS-Security headers.
You can read one of my previous blog posts to see how you can send a secure SOAP request to a service using soapUI.
The beauty of WSO2 Application Server is, it allows you to send a HTTP GET request with basic authentication headers to the same web service. Let's see how we can send a HTTP GET request to invoke the above HelloService which has been secured with Username Token security policy.

Step 4:
Open soapUI.   Click on File --> New soapUI Project. Enter a name for the project. Select Create Web TestCase option (We are not going to deal with any SOAP messages, therefore we do not want to create usual soapUI project using a WSDL). This will open Add Web TestCase dialog. Enter the following URL as the web address.
Web address = https://localhost:9443/services/HelloService/greet?name=soapUI

Note that, this is the url of the GET request which we are going to submit.

Step 5:

Once you click on OK in the above dialog, soapUI will open an HTTP request editor with the above request URL. Click on Request tab to move into the request editor if you ended up in the response tab.
Click on Aut tab where you can specify user credentials. Enter the following credentials of admin user which we used to secure HelloService at the step 2.


Step 6:

Before invoking the service, make sure Authenticate Preemptively option is selected at the HTTP Settings of soapUI Preferences dialog (File --> Preferences --> HTTP Settings --> Authenticate Preemptively).
Now, submit the request. You will see the following in the raw view of request.

GET https://localhost:9443/services/HelloService/greet?name=soapUI HTTP/1.1
Accept-Encoding: gzip,deflate
Authorization: Basic YWRtaW46YWRtaW4=
User-Agent: Jakarta Commons-HttpClient/3.1
Host: localhost:9443

And, the response will be similar to the following.

HTTP/1.1 200 OK
Content-Type: application/xml;charset=UTF-8
Transfer-Encoding: chunked
Content-Encoding: gzip
Vary: Accept-Encoding
Date: Thu, 20 Sep 2012 08:38:31 GMT
Server: WSO2 Carbon Server

Hello World, soapUI !!!

Thursday, September 13, 2012

Setting up minimum deployment of WSO2 product cluster with management/worker separation

The latest version of WSO2 Carbon (Carbon-4.0.0) platform (WSO2 ESB-4.5.0, WSO2 Application Server-5.0.0, WSO2 Governance Registry - 4.5.0 etc) supports a new deployment model which allows you to setup product clusters with separated worker and management nodes. In a worker/manager separated cluster setup, the management node(s) is used to deploy and configure the deployment artifacts where as the worker nodes are used to serve the requests received by clients.
I'm not going to spend much time on a detailed explanation about this new deployment approach since my objective is to take you through the minimum worker/manager deployment setup of WSO2 Application Server cluster as quickly as possible. You can find more about the theoretical aspects of this new deployment model in Afkham Azeez's blog.

Deployment diagram

We are going to setup the minimum cluster setup in our local machines. Thus, we will use 3 product instances as follows.

Load Balancer - used to load balance the requests between Application Server cluster nodes.
WSO2 Application Server management node - all deployments and configurations are performed through this node.
WSO2 Application Server worker node - Application server requests are served by this node.


Download the latest version (5.0.0) of WSO2 Application Server from here.
Download the latest version(2.0.0) of WSO2 Elastic Load Balancer from here.

Setting up WSO2 Elastic Load Balancer

Step 1:
Extract wso2elb-2.0.0.zip into a directory in your local file system. Let the extracted directory be WSO2_LB_HOME

Step 2:
Go to WSO2_LB_HOME/repository/conf and edit the loadbalancer.conf as follows.

appserver {
        domains   {
            CharithaASdomain {
                hosts mgt.charitha.appserver.wso2.com;
                sub_domain mgt;
                tenant_range    *;

            CharithaASdomain {
                hosts charitha.appserver.wso2.com;
                sub_domain worker;
                tenant_range    *;

As you can see above, we have defined two cluster sub domains, mgt and worker under a single clustering domain which is named as "CharithaASdomain".
Both these sub domains consists of one WSO2 Application server instance in each. mgt sub domain includes the management node of our cluster and the worker sub domain consists of a worker node. (See the deployment diagram above).
The two WSO2 Application Server instances (in our case cluster sub domains, because each sub domain has only one instance) can be setup in two physical servers, 2 VM instances or a single machine. In our example, we will set everything up in our local machine. In order to identify the two nodes uniquely, we will define hostnames for both. Therefore, mgt subdomain will be identified using mgt.charitha.appserver.wso2.com and the worker subdomain will be identified by  charitha.appserver.wso2.com
Since we are running everything in local machine, let's update the /etc/hosts file to match up with the hostnames we use. mgt.charitha.appserver.wso2.com charitha.appserver.wso2.com

Step 3: 
Uncomment localMemberHost element in WSO2_LB_HOME/repository/conf/axis2/axis2.xml and specify the IP address (or host name) which you are going to advertise to the members of the cluster.

<parameter name="localMemberHost"></parameter>

Save loadbalancer.conf, axis2.xml and /etc/hosts files.

The above are the only configurations which need to be done in WSO2 LB. You do not have to configure anything in other configuration files in the minimum deployment setup.
Now, start WSO2 Load Balancer by running wso2server.sh script which can be found at WSO2_LB_HOME/bin. You will see the following logs at server startup.

[2012-09-13 20:04:08,407]  INFO - TribesClusteringAgent Managing group application domain:CharithaASdomain, sub-domain:mgt using agent class org.wso2.carbon.lb.endpoint.SubDomainAwareGroupManagementAgent
[2012-09-13 20:04:08,408]  INFO - TribesClusteringAgent Managing group application domain:CharithaASdomain, sub-domain:worker using agent class org.wso2.carbon.lb.endpoint.SubDomainAwareGroupManagementAgent
[2012-09-13 20:04:08,410]  INFO - ServerManager Server ready for processing...
[2012-09-13 20:04:08,483]  INFO - AutoscalerTaskServiceComponent Autoscaling is disabled.
[2012-09-13 20:04:08,537]  INFO - PassThroughHttpSSLListener Starting Pass-through HTTPS Listener...
[2012-09-13 20:04:08,550]  INFO - PassThroughHttpSSLListener Pass-through HTTPS Listener started on port : 8243
[2012-09-13 20:04:08,550]  INFO - PassThroughHttpListener Starting Pass-through HTTP Listener...
[2012-09-13 20:04:08,555]  INFO - PassThroughHttpListener Pass-through HTTP Listener started on port : 8280
[2012-09-13 20:04:08,557]  INFO - TribesClusteringAgent Initializing cluster...
[2012-09-13 20:04:08,583]  INFO - TribesClusteringAgent Cluster domain: wso2.as.lb.domain
[2012-09-13 20:04:08,587]  INFO - TribesClusteringAgent Using wka based membership management scheme
[2012-09-13 20:04:08,599]  INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/
[2012-09-13 20:04:08,699]  INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/
[2012-09-13 20:04:08,846]  INFO - TribesClusteringAgent Local Member
[2012-09-13 20:04:08,847]  INFO - TribesUtil No members in current cluster
[2012-09-13 20:04:08,849]  INFO - TribesClusteringAgent Cluster initialization completed.
[2012-09-13 20:04:09,412]  INFO - StartupFinalizerServiceComponent WSO2 Carbon started in 14 sec

Let's proceed with configuring WSO2 Application Server management node.

Setting up WSO2 Application Server Management Node

Step 1:

Extract wso2as-5.0.0.zip into a directory in your local file system. Let the extracted directory be WSO2_AS_MGR_HOME.

Step 2 (axis2.xml configuration):

First, we need to enable clustering at axis2 level in order for management node to communicate with load balancer and the worker nodes. Open  WSO2_AS_MGR_HOME/repository/conf/axis2/axis2.xml and update the clustering configuration as shown below.

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">

<parameter name="membershipScheme">wka</parameter>

Specify the cluster domain. Note that, this must be the same which we defined in loadbalancer.conf

<parameter name="domain">CharithaASdomain</parameter>

<parameter name="localMemberHost">mgt.charitha.appserver.wso2.com</parameter>

<parameter name="localMemberPort">4250</parameter>

We will add a new property "subDomain" and set it to "mgt" to denote that this node belongs to mgt subdomain of the cluster as we defined in loadbalancer.conf.
  <parameter name="properties">
            <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
            <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
            <property name="subDomain" value="mgt"/>


Now, add load balancer IP or host name (in our case, we will refer to the load balancer using and the local member port (4000) as defined in the axis2.xml of WSO2 LB) as a well-known member.

Step 3 - catalina-server.xml configuration:
WSO2 Application Server management node is fronted by Load Balancer. Therefore, we need to configure the proxy ports which are associated with HTTP and HTTPS connectors. These proxy ports are the corresponding transport receiver ports opened by WSO2 LB (configured in transport listeners section in axis2.xml).
Open WSO2_AS_MGR_HOME/repository/conf/tomcat/catalina-server.xml and add the proxyPort attribute for both HTTP and HTTPS connectors as shown below.

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"

 <Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"

Step 4 - carbon.xml configuration:

Since we run multiple WSO2 Carbon based products in same host, we must avoid the possible port conflicts. Therefore, edit the following element in WSO2_AS_MGR_HOME/repository/conf/carbon.xml by assigning port offset 1 to the Application Server management node.


Update mgtHostName and HostName elements in carbon.xml as shown below.



We must do one more configuration change in carbon.xml before moving to the next step. As we discussed at the beginning, Application Server management node is used for deploying artifacts such as web applications and web services. The deployed artifacts in management node must be synchronized automatically to the worker nodes in cluster. This deployment synchronization mechanism is pretty straight-forward in WSO2 Carbon based products.  The default SVN based deployment synchronizer can be used to auto-commit the deployment artifacts to a pre-configured SVN repository. Then, the worker nodes can be configured to automatically check-out the artifacts from the same SVN location.

Let's configure SVN based deployment synchronizer in carbon.xml of management node.

Make sure to replace the SvnUrl, SvnUser and SvnPassword according to your SVN repository.

Step 5: Starting the management node of WSO2 Application Server cluster

Go to WSO2_AS_MGR_HOME/bin and run wso2server.sh to start the server. During the server startup, you should see cluster initialization messages similar to the following.

[2012-09-13 23:04:25,820]  INFO {org.apache.axis2.clustering.tribes.TribesClusteringAgent} -  Initializing cluster...
[2012-09-13 23:04:25,840]  INFO {org.apache.axis2.clustering.tribes.TribesClusteringAgent} -  Cluster domain: CharithaASdomain
[2012-09-13 23:04:25,843]  INFO {org.apache.axis2.clustering.tribes.TribesClusteringAgent} -  Using wka based membership management scheme
[2012-09-13 23:04:25,852]  INFO {org.apache.axis2.clustering.tribes.WkaBasedMembershipScheme} -  Receiver Server Socket bound to:/
[2012-09-13 23:04:25,960]  INFO {org.apache.axis2.clustering.tribes.WkaBasedMembershipScheme} -  Added static member

At the same time, look at the logs of Load Balancer.

[2012-09-13 23:04:41,012]  INFO - RpcMembershipRequestHandler Received JOIN message from
[2012-09-13 23:04:41,012]  INFO - MembershipManager Application member joined group CharithaASdomain
[2012-09-13 23:04:41,013]  INFO - ClusterManagementMode Member joined cluster
[2012-09-13 23:04:52,016]  INFO - DefaultGroupManagementAgent Application member Host:, Port: 4250, HTTP:9764, HTTPS:9444, Domain: CharithaASdomain, Sub-domain:mgt, Active:true joined application cluster

By looking at the above logs, we can conclude that Application Server management node has successfully joined cluster and ready to receive requests through the load balancer.
Now, we can log in to the management console of the Application Server management node. Access https://mgt.charitha.appserver.wso2.com:8243/carbon/ and log in to the management console with the default administrator credentials (admin/admin). Go to the "Deployed Services" page and click on WSDL1.1 link of the HelloService. You will get an HTTP 500 error since the server looks for the URL, http://charitha.appserver.wso2.com:8280/services/HelloService?wsdl, which is not yet available. As you can see from this URL, though the service is deployed in the management node, all requests are served by the worker nodes. charitha.appserver.wso2.com is the hostname which we have given for the worker node and we have not started the worker node yet.

Setting up WSO2 Application Server Worker node

Since we have completed setting up the management node, worker node configuration is quite straight-forward. Let's configure the worker node in our local machines.

Step 1:
We are not going to extract another fresh version of wso2as-5.0.0.zip. Instead, make a copy of the management node into a different directory, so that we can reuse most of the settings which we used above. Rename the directory as wso2as-5.0.0-worker. Let this directory be WSO2_AS_WORKER_HOME.

Step 2 - axis2.xml configuration:
In addition to the changes which we did in axis2.xml of the management node, there are few modifications need to be done for worker. First, we update localMemberPort element since we launch two instances of cluster nodes in the same machine. Open WSO2_AS_WORKER_HOME/repository/conf/axis2/axis2.xml and edit the following element.

<parameter name="localMemberPort">4251</parameter>

Application Server worker node belongs to the "worker" sub domain of the cluster domain as we have configured in loadbalancer.conf of WSO2 Load Balancer. As we did with the management node, we can add a new property, "subDomain" as follows to represent this.

<parameter name="properties">
            <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
            <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
            <property name="subDomain" value="worker"/>

Step 3: carbon.xml configuration:

First, we should specify a new port offset for the worker node. Edit WSO2_AS_WORKER_HOME/repository/conf/carbon.xml and update the portOffset value as follows.


Also, update the HostName element as shown below. We do not need to specify MgtHostName element since this node is designated as the worker node in Application Server cluster.


Now, we need to configure the SVN based deployment synchronizer to automatically check-out deployment artifacts from a common SVN repository. The worker nodes of a cluster SHOULD NOT commit (write) artifacts hence we must disable AutoCommit property in the deployment synchronizer configuration as shown below.


Step 4: Start WSO2 Application Server Worker node

We are ready to start the worker node of our cluster. Go to WSO2_AS_WORKER_HOME/bin and start the server as follows. Note that, the workerNode system property must be set to true when starting the workers in a cluster.

sh wso2server.sh -DworkerNode=true

Once the worker node joins the cluster, you will see the following messages in WSO2 Load Balancer logs.
[2012-09-14 06:32:41,050]  INFO - RpcMembershipRequestHandler Received JOIN message from
[2012-09-14 06:32:41,051]  INFO - MembershipManager Application member joined group CharithaASdomain
[2012-09-14 06:32:44,387]  INFO - ClusterManagementMode Member joined cluster
[2012-09-14 06:32:52,052]  INFO - DefaultGroupManagementAgent Application member Host:, Port: 4251, HTTP:9765, HTTPS:9445, Domain: CharithaASdomain, Sub-domain:worker, Active:true joined application cluster

Testing the cluster

There are many ways to test our cluster deployment. Let's follow the simplest path. 

- Log in to the management console of Application Server management node 
- Deploy a new Axis2 Web service (Go to Manage --> Axis2 Services --> Add --> AAR service)
- Once the service is deployed in the management node go to the services list and click on Tryit
- Invoke the service

You will notice that the requests are served by the worker node of the cluster. If you click on WSDL1.1 link of any service, the WSDL will be served by the worker (http://charitha.appserver.wso2.com:8280/services/HelloService?wsdl)

Let me know if you come across any issues when setting up the cluster. 

Sunday, July 29, 2012

Software testing misunderstood

I recently read a nice blog post by James Bach where he discussed common misinterpretations of software testing specially in agile teams. While agreeing with most of his ideas' the following statement made a lot of sense to me

Avoid labeling an activity as a “test” unless it represents a sincere effort to discover a problem in a product 

In my software testing career, large number of people I have worked with, regardless of  their QA or development background, misinterpreted this basic concept. I do not have to explain how dangerous it is to claim that a feature of your product "works" just by running it. "Testing" is NOT merely running the product. A good test should uncover some sort of problem of your product/feature. Objective of testing should be to finding bugs. Just wearing QA hat and running the product does not do any good. It just claims that your product runs in one way but it will not be the only execution path. Interpreting "running product features" as "testing", destroys the quality of software in many ways.

- False claims contribute to draw a wrong picture about the status of product. The release plans, schedules everything is decided based on those untruthful information
- Large number of bugs uncovered in production use
You, during development cycles, update many stake holders of your project that the feature is running properly. You may also do multiple demonstrations of your feature to various parties. You must understand that the demonstrations of the feature or training sessions of the new feature just executes one or two of the hundreds of execution possibilities. Hence, it should not be interpreted as an indirect test execution and comes into conclusions that the feature works perfectly.
- Building a deadly trend among fellow developers/testers that quality is not important factor
When people start to believe that "running" the feature as "testing", then it like a virus spread across whole devs/testers of the organization. People just focus on happy path (positive testing) which eventually ruins the quality of the product.

Monday, July 2, 2012

Consuming a SOAP service using WSO2 API Manager

SO2 API Manager is the new kid in the block, the first fully open source API management platform which can be used to create, manage, consume and monitor APIs. WSO2 API manager is based on the award winning SOA middleware platform, WSO2 Carbon thus it inherits the features provided by the modular Carbon platform.
API authors and publishers can publish both REST and SOAP services to external consumers or partners through WSO2 API manager. In this post, we will look in to the steps of publishing and consuming a SOAP based web service using WSO2 API Manager.

Download and install alpha version of WSO2 API Manager from http://dist.wso2.org/products/api-manager/1.0.0-alpha/wso2am-1.0.0-ALPHA.zip. Similar to the other WSO2 Carbon products, you just need to have JDK-1.6.0 or later to run WSO2 API Manager. Then you can just extract the binary and start the server.

Step 1:

We will use a simple SOAP based web service hosted in WSO2 StratosLive Application Server. You can find the WSDL of the service at http://appserver.stratoslive.wso2.com/services/t/superqa.com/Axis2Service?wsdl

Step 2:

WSO2 API Manager consists of three different applications which serve for distinct user requirements. The API Gateway which is accessible through https://localhost:9443/carbon, is used as a gateway for API calls. The API Provider web app is used to publish, manage and monitor statistics of APIs. API provider can be accessed through https://localhost:9443/apiprovider. API Store application acts as the API consumer front end where users can subscribe for the published API. API Store is accessible through https://localhost:9443/apistore Ideally, three different user categories access these three applications. The API gateway is usually hosted and managed by API management provider. Therefore, we can assume that the default admin user of WSO2 API Manager as the API management provider thus he/she initiates creating rest of the user roles in the system. The API provider webapp is usually accessed by people who want to publish and manage new APIs. Therefore, we can think of another distinct role related to creating and publishing APIs. Finally, the API consumer user access API store webapp to subscribe for APIs.

Though we can use the default admin user to access all of the above applications, to be much realistic, lets start to create the roles and users associated to the API provider and API store applications.

  • Log in to API gateway app as admin user (user name=admin, password=admin).
  • Click on Configurein left menu and select Users and Roles
  • Click on Add New Role, enter "publisher" (or whatever you want) as the role name and click on next
  • Select the following permissions from the permission tree

Admin Permissions --> Configure --> Governance
Admin Permissions --> Login
Admin Permissions --> Manage --> API --> publish
Admin Permissions --> Manage --> API --> create
Admin Permissions --> Manage --> Resources --> Govern

  • Click on Finish to complete the creation of new role
  • Now access the Users page and click on Add New User. (say 'charitha' as the user name of new user, password is 'charitha')
  • Assign the above role (publisher) to the new user
We have just created a new user who is granted permissions to access API provider webapp.

Step 3:

Access https://localhost:9443/apiprovider. The log in page of the API publisher application will be launched. Log in to the application using the credentials of the user which we have created in the previous step.
Click on Add menu item in the left navigation pane to add a new API.

Enter the following values for the mandatory fields in the above form.

Name: Axis2ServiceAPI
Context: /SOAPService
Version: 1.0.0
Endpoint URL: http://appserver.stratoslive.wso2.com/services/t/superqa.com/Axis2Service
Tier Availability: Bronze

Leaving the other default values as they are, click on Create to add the new API. The published API will be shown in the All APIs page and it will be in the CREATED state.

Step 4:

The APIs which are at the CREATED state cannot be consumed by any user. Therefore, we need to publish the API to API store. Click on the Axis2ServiceAPI-V1.0.0 in All APIs page which will direct you to the detailed information page of the selected API. Click on Lifecycles tab. Select the state as PUBLISHED. Select Push To Gateway option so that the API will be registered in the API gateway.

Clicking on Update will publish the API. If you see the log of API gateway, you will notice the following message. INFO - API Initializing API: charitha--Axis2ServiceAPI 

Step 5: 

Now, the API which we have created from the SOAP based web service is in PUBLISHED state so that a consumer can subscribe to the API and invoke it. API store can be used for the consumers to subscribe to APIs. Access https://localhost:9443/apistore/. The landing page of WSO2 API Store application will be shown. You will notice the API which just has been published in available in that page.
Click on Signup at the top right corner to register a new consumer user. Enter 'apiconsumer' as the username and 'apiconsumer123' as the password.
Log in to API store using the new consumer's credentials.

Step 6:

Now, we are inside the API Store web application. We should create an application to make API subscriptions and consumption. Click on My Applications tab.
Add a new application with the name, AxisApp as shown below.

We need to generate an OAuth key to consume any API. Click on the Axis2ServiceAPI - 1.0.0 in the API list or left pane of the API Store web application.  Select Axis2App from the Applications dropdown list at the right of the page and click on subscribe.

The "Subscription Successful" confirmation dialog will be popped up. Click on Go to My Subscriptions button to check our new API subscription.

Click on Generate in the above page to create a new API key. Once the key is generated, click on Show Key and copy the API key.

Step 5: Now, we have everything required to consume the API. API has been published to the store, subscribed to the API and possessed an API key. Lets invoke the API using any client. In this example, I will use soapUI, the world's best and most user-friendly web service invocation utility to invoke the API.

  • Create a new soapUI project or use an existing project. 
  • Add the following SOAP request to your soapUI project.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://service.carbon.wso2.org">

  • Add http://localhost:8280/SOAPService/1.0.0/ as a new endpoint url in soapUI request editor. (This url can be found under the overview tab of Axis2ServiceAPI - 1.0.0 in API store)
  • Add the following HTTP header 
Authorization: Bearer  <Your API key which has been generated when subscribing to the API >
  • Now, your soapUI request editor will be similar to the following.

  • Submit the request and check the response. Since we have subscribed under bronze tier, you will be allowed to send 1 request per minute

Friday, May 18, 2012

Managing SOA artifacts in different environments using WSO2 Governance Registry


Please note that the instructions given in this blog post are applicable ONLY for ESB-4.0.X and G-reg-4.1.X, G-reg-4.5.0 versions. The deployment approach of the latest ESB versions has been changed and the recommended deployment synchronization approach has been switched to SVN based synchronizer. Because of that the following set of instructions cannot be applied to the latest WSO2 ESB and G-reg product versions.

Managing artifacts of a service oriented solution is one of the most important features expected from any SOA middleware platform. In a typical service oriented project, SOA artifacts are subjected to move through multiple phases. Usually, separate environments are maintained for the activities associated with those phases.
  • Development environment
Service development and system integration tasks are carried out in a separate physical environment. Depending on the requirements, there can be multiple SOA middleware solutions used in development environment to facilitate service development, integration and deployment processes.Once the development tasks are completed,  SOA artifacts are transferred into QA environment for QA verification.
  • QA environment
Solution testing is done in an independent environment which is usually identical to the production settings. Various functional and non-functional tests are performed in QA environment. Upon successful QA verification, the artifacts are moved to production or staging environment

When the service oriented solution becomes complex and there are large number of SOA artifacts, automated artifact governance mechanism is required to ensure smooth transition between various environments. In this post, I will take you through a simplified SOA artifact governance process using WSO2 SOA middleware stack. We will explore a use case similar to the following.

Separate development and QA environments are used to maintain the SOA artifacts. For example, solution developers implement various configuration artifacts in ESB development node. After completing those development tasks, solution developers move those artifacts in to QA environment which allows the testers to carryout QA activities in an independent environment.  The moving of artifacts will be done automatically using the features provided by central SOA governance solution.
In this example, we will use WSO2 ESB as the enterprise service bus middleware, WSO2 Governance Registry as the SOA governance solution.

Lets go through each of the steps in detail. We need to have 2 WSO2 ESB instances and 1 WSO2 G-reg instance which runs on mySQL. All these instances will be deployed in single host.

Setting up G-reg and ESB products
Step 1
Download the latest versions of WSO2 ESB and WSO2 G-reg from here. First we will configure WSO2 G-reg instance. Extract wso2greg-4.x.x.zip into your file system. We will refer to the extracted location as GREG_HOME.
By default, WSO2 products run on the file based embedded H2 database. Since we need two ESB instances connect to the database used by G-reg instance, we will configure G-reg to run on a mySQL database. Enter the following commands to create a database and grant permission to a user.

mysql> create database greg_db;
Query OK, 1 row affected (0.00 sec)

mysql> use greg_db;
Database changed
mysql> grant all on greg_db.* to regadmin@localhost identified by "regadmin";
Query OK, 0 rows affected (0.05 sec)

Step 2
Now, we need to change the default database configuration in G-reg instance through registry.xml configuration file. Open GREG_HOME/repository/conf/registry.xml and update dbConfig element as shown below.
<dbConfig name="wso2registry">
Copy mysql jdbc driver to GREG_HOME/repository/components/lib and start G-reg server by running wso2server.sh

Step 3
We have completed the configurations of WSO2 Governance Registry instance. Lets move forward with setting up ESB instance in development environment.

Extract the downloaded wso2esb-4.x.x.zip into a directory in your file system. We will refer to this directory as ESBDEV_HOME. Since we are running all product instances in a single machine, we need to start them up in different ports. Therefore, change port offset parameter in ESBDEV_HOME/repository/conf/carbon.xml as follows.

Step 4
As shown in the above diagram, we will mount /_system/config collection, where the ESB artifacts are stored in to /_system/dev collection of G-reg instance. ESBDEV_HOME/repository/conf/registry.xml is used to configure that.
<dbConfig name="configgovregistry">
<remoteInstance url="https://localhost:9443/registry">

<mount path="/_system/config" overwrite="true">
<mount path="/_system/governance" overwrite="true">
Start ESB dev instance. If you log in to management console of the ESB instance (https://localhost:9444/carbon) and access registry browser, you will notice the config and governance collections are mounted (See the arrow icons) to the remote G-reg database.

Step 5

Repeat step 3 and 4 with another copy of wso2esb-4.x.x.zip. Lets call it ESBQA instance. (The root directory of ESBQA instance will be referred to as ESBQA_HOME). Make sure to specify a different offset value in carbon.xml. In this instance, we will mount /_system/config collection to /_system/qa collection of G-reg instance. Update registry.xml of ESBQA node as follows.
<dbConfig name="configgovregistry">
<remoteInstance url="https://localhost:9443/registry">

<mount path="/_system/config" overwrite="true">
<mount path="/_system/governance" overwrite="true">

Start the server. We are done with the product configurations. However, we have not done any artifact governance configuration yet. In the next steps, we will look into moving artifacts between the two ESB instances.

Using WSO2 Governance Registry to move artifacts between environments
As I explained at the beginning, we use central governance registry instance to manage the artifacts produced by each of DEV and QA environments. The ESB artifacts produced by development environment are stored under /_system/dev collection of governance registry where as the artifacts used in QA environment are stored in /_system/qa collection. When the development tasks are compelte and ready for QA, we can manually copy the artifacts into the relevant locations and configure the QA environment. Obviously it will be a painful task to copy large number of artifacts into various locations of ESB QA environment by hand. Therefore, we need some kind of automated artifact copying mechanism.
WSO2 Governance Registry provides us with registry extension features to extend the core functionality of G-reg to use in these types of situations. The lifecycle management feature is such a useful extension provided by WSO2 Governance Registry which can be used to manage life cycle of a resource stored in registry.
The standard WSO2 G-reg distribution is shipped with a default ServiceLifeCycle which can be used to move service artifacts among different environments. We will modify the default ServiceLifeCycle to copy artifacts stored under /_system/dev collection to /_system/qa when promoting from development stage to testing.

Log in to G-reg management console and navigate to Extensions --> Configure --> Lifecycles. Click on Add New LifeCycle and add the following life cycle.
<aspect name="ESBLifeCycle" class="org.wso2.carbon.governance.registry.extensions.aspects.DefaultLifeCycle">
    <configuration type="literal">
            <scxml xmlns="http://www.w3.org/2005/07/scxml"
                <state id="Development">
                        <data name="checkItems">
                            <item name="Configurations Completed" forEvent="">
                            <item name="Transform Rules Done" forEvent="">
                            <item name="Routing Rules Completed" forEvent="">
            <data name="transitionExecution">
                            <execution forEvent="Promote" class="org.wso2.carbon.governance.registry.extensions.executors.CopyExecutor">
                           <parameter name="currentEnvironment" value="/_system/dev"/>
                           <parameter name="targetEnvironment" value="/_system/qa"/>                                
            <data name="transitionUI">
                            <ui forEvent="Promote" href="../lifecycles/pre_invoke_aspect_ajaxprocessor.jsp?currentEnvironment=/_system/dev/"/>
                    <transition event="Promote" target="Testing"/>  
                    <state id="Testing">
                   <transition event="Demote" target="Development"/>            
You can see that we have used org.wso2.carbon.governance.registry.extensions.executors.CopyExecutor class to copy one or more resources from one environment to another. Executors are custom extensions to G-reg which used to trigger a custom execution logic at the time of state transition (e.g:- Dev to QA, QA to Production). When artifacts are promoted to Testing state from Development, the CopyExecutor is triggered and the events defined under transitionExecution element are carried out. In this lifecycle, CopyExecutor copies any resource which is at currentEnvironment to targetEnvironment

Note that, CopyExecutor has not been shipped by default in WSO2 G-reg-4.1.X versions. Hence, you need to implement org.wso2.carbon.governance.registry.extensions.interfaces.Execution interface. However, in current trunk based versions (which will be released within few months), this executor is included by default and you can use it out-of-the-box.

Once the life cycle is created and saved, we can apply it to the relevant collection in registry. In this example, we need to move the artifacts stored at /_system/dev to /_system/qa. Hence, we must apply  ESBLifecycle to /_system/dev collection.

Navigate to /_system/dev collection in registry browser. Click on Lifecycle in the left pane. Select Add Lifecycle and choose ESBLifeCycle from the dropdown list. Click on Add to add ESBLifeCycle to /_system/dev collection.

We do not have any artifacts in /_system/dev collection yet. Therefore, lets add some ESB artifacts through ESB management console. Log in to the management console of ESB DEV instance and add a sequence. (say it is "devsequence). This will create a new sequence in ESB and it will be stored under /_system/dev in G-reg
Now, go back to G-reg management console and navigate to /_system/dev collection. Select the check-list items such as Configurations Completed, Transform Rules Defined etc.. and click on Promote.
After promotion, click on Search --> metadata in left menu in G-reg management console and search for the resource which has just been added ("devsequence"). The resource will be copied to /_system/qa collection.

We have just completed automatic promotion of resources which have been created from ESB DEV environment to ESB QA environment. However, you will notice that the ESB sequence artifact which has just been added, is not "deployed" in ESB QA instance. Because of that, you will not see "devsequence" in sequence list of ESB QA instance. The artifact is added to the relevant registry location but in order for deployment of the resource, ESB needs to check-out the resource from /_system/qa/repository/deployment/server/synapse-configs/default/sequences location in registry to the corresponding deployment directory of file system of ESB QA server. 
DeploymentSynchronizer comes into action in this situation. As I have explained in a previous blog post, DeploymentSynchronizer can be used to synchronize deployment artifacts among cluster nodes.
Lets enable registry based deployment synchronizer for ESB DEV and ESB QA instances.

- Shutdown both ESB DEV and ESB QA servers
- From WSO2 Carbon-4.0.0 release onwards, we need to enable Tribes based clustering in order for deployment synchronizer to communicate with cluster nodes. Open ESBDEV_HOME/repository/conf/axis2/axis2.xml and enable clustering and specify a unique domain name.
<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
<parameter name="domain">wso2.charitha.domain</parameter>

Repeat the same with ESBQA_HOME/repository/conf/axis2/axis2.xml as well.

- Enable deployment synchronizer configuration in ESBDEV_HOME/repository/conf/carbon.xml
- Similarly, Enable deployment synchronizer configuration in ESBQA_HOME/repository/conf/carbon.xml. ESB QA instance automatically checks out the resources from /_system/qa collection of G-reg. Therefore, we just enable AutoCheckout in that node.

We have completed enabling deployment synchronizer for both ESB development and QA servers. Now, try out adding another sequence from ESB DEV instance (say it is "devsequence2"). Once the sequence is added, access G-reg management console and browse /_system/dev collection. Promote the lifecycle again. This time, you will notice that devsequence2 is deployed in ESB QA instance upon successful promotion
In this post, I took you through the steps of moving artifacts between different environments using WSO2 ESB and WSO2 G-reg. We created artifacts in ESB using management console. However, the recommended best practice is to build SOA artifacts using a tool like WSO2 Carbon Studio and upload the CAR artifacts to ESB. We will look into CAR based SOA governance mechanism in a future post.