Popular Posts

Monday, November 25, 2013

Simulating RESTful services with soapUI

I have discussed about the mock services which simulate SOAP back-ends in chapter 6 of Web Services Testing with soapUI book. For the past few years, service orientation has been shifting towards RESTful web services from SOAP which demands the necessity of mocking various types of RESTful services. Being the one-stop tool kit of SOA developers/testers, soapUI provides users with different options to simulate RESTful services. In this post, I will discuss one of such a simple mechanisms to create RESTful mock services in soapUI.


soapUI-4.5.2 or later

Simulating POX (Plain-Old-XML) with soapUI

Suppose you have a web service which returns XML response. Usually, the HTTP Content-Type of such a message is application/xml or text/xml.
  1. Create a new MOCK Service in soapUI. You can either create a Mock service from a new WSDL (ouch! we are dealing with RESTful services. Why do we need WSDL?? Does not matter. We just want to have a mock service regardless of where it originated from. We will tweak the response to be RESTy. Trust me!) or add a new mock service by right-clicking on an interface of an existing SOAP based service.
  2. Either way, you will have a mock service similar to the following.

    Note that, our mock service will be exposed at http://localhost:8088/restMockService
  3. We are going to simulate a response with text/xml content. Create an XML file in your file system. i.e:-


    Save the file as xmlresponse.xml.
  4. Once a HTTP request (regardless of the content of incoming message) hits the mock service, it will return XML response. We can execute a groovy script at this time and respond back with XML output. This can be achieved by having a OnRequest Script in soapUI mock service interface. Click on OnRequest Script tab at the bottom pane of mock service editor.
  5. In the OnRequest Script editor, add the following script.

    //To output text/xml
    mockRunner.returnFile(mockRequest.httpResponse, new File("/home/charitha/Desktop/xmlresponse.xml"))
    return new com.eviware.soapui.impl.wsdl.mock.WsdlMockResult(mockRequest)
  6. Start the mock service by clicking on green arrow icon at the top left corner of mock service editor. 
  7. Send any HTTP request to http://localhost:8088/restMockService. You will observe the following output.

    HTTP/1.1 200 OK
    Content-Length: 35
    Content-Type: text/xml
    Connection: close
    Server: Jetty(6.1.26)

    Simulating JSON output with soapUI

    This is quite similar to the above procedure but a few tweaks are required to format the JSON response to be a valid JSON with application/json content-type.

    1. In the same mock service which we created above (or a different one), we will add a new OnRequest script. First, create a file in your local file system with the following content.

        "getQuote": {
          "request": { "symbol": "WSO2" }

      Save the file as jsonresponse.json.
    2. Now, add the following script as OnRequest Script
      import javax.servlet.http.HttpServletResponse
      import com.eviware.soapui.support.Tools
      def response = mockRequest.httpResponse
      File file = new File("/home/charitha/Desktop/jsonresponse.json")
      FileInputStream fin = new FileInputStream(file)
      response.setStatus( HttpServletResponse.SC_OK )
      long length = file.length();
      response.setContentLength( ( int )length );
      response.setContentType("application/json;charset=utf-8" );
      Tools.readAndWrite( fin, length, response.getOutputStream() );
      return new com.eviware.soapui.impl.wsdl.mock.WsdlMockResult(mockRequest)

      You will question why we cannot just return the output of jsonresponse.json straight-away as we did with text/xml content. As explained by Ole Lensmar (The creator of soapUI) in http://forum.loadui.org/viewtopic.php?f=2&t=15020&p=36012 , this is a hack to respond back with application/json content-type. If you just return the file, the content-type will be delivered as application/javascript.
    3. Now, restart the mock service and send a HTTP request to above mock service endpoint. You will get a response similar to the following.

      HTTP/1.1 200 OK
      Content-Length: 60
      Content-Type: application/json;charset=utf-8
      Connection: close
      Server: Jetty(6.1.26)
        "getQuote": {
          "request": { "symbol": "WSO2" }

      Similarly, you can simulate what ever RESTful back-end you want with soapUI. For example, if you want to return text/html, create a simple HTML in your file system and add the following script.

      //To output text/html
      mockRunner.returnFile(mockRequest.httpResponse, new File("/home/charitha/Desktop/htmlresponse.html"))
      return new com.eviware.soapui.impl.wsdl.mock.WsdlMockResult(mockRequest)

      To return text/plain response, create a text file in file system and add the following script.

      //to output text/plain
      mockRunner.returnFile(mockRequest.httpResponse, new File("/home/charitha/Desktop/plaintextresponse.txt"))
      return new com.eviware.soapui.impl.wsdl.mock.WsdlMockResult(mockRequest)

Friday, October 25, 2013

The difference between JSON streaming builder and the default JSON builder in WSO2 ESB

When a request/response is received by ESB, the message will initially be converted into XML infoset. This process is known as message building and the builders are responsible to do this transformation based on the HTTP Content-Type of the incoming message. The default JSONBuilder (org.apache.axis2.json.JSONBuilder) converts the JSON into corresponding XML representation. If I explain this through an example, suppose your incoming JSON response/request is similar to the following.

  "getQuote": {
    "request": { "symbol": "charitha" }

If you log this message inside inSequence of your message flow (using log mediator), you will observe an output similar to the following.

LogMediator To: http://www.w3.org/2005/08/addressing/anonymous, 
WSAction: , SOAPAction: , MessageID: urn:uuid:f37b8466-7cb3-41d6-9187-b5a0a7648f16, 
Direction: response, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><getQuote><request><symbol>charitha</symbol></request></getQuote></soapenv:Body></soapenv:Envelope> 

Look at the Body of the SOAP envelope. You may notice the JSON to XML mapping clearly in there.

The message formatters do the complete opposite of builders. They are responsible for formatting the message into the relevant wire format. So, the default JSONMessageFormatter converts XML back into JSON.
Obviously this JSON <--> XML conversion may lead to a certain degree of information loss.

In order to avoid that, we have introduced the JSON streaming builder (org.apache.axis2.json.JSONStreamBuilder) which does not covert JSON message to intermediate XML format. If you conduct the same test which I explained above with streaming JSON builder, you will see a message similar to the following when you use log mediator within the message flow.

LogMediator To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:33d9ad3f-83bc-46ee-adb9-1cafd7e5a9cc, Direction: response, Envelope:

In the above message, there is no XML representation of the JSON message. The Body element is empty.

Therefore, if you use streaming JSON builder/formatter, you will not be able to access the message payload. If you want to access/modify the payload, that can only be achieved using script mediator. To access/modify JSON paylaod, you can use mc.getPayloadJSON() or mc.setPayloadJSON() methods as explained in http://docs.wso2.org/display/ESB470/Sample+441:+Exposing+a+SOAP+Service+Over+JSON

Sunday, September 22, 2013

QA mind-set

In the agile world of software development, quality assurance function is embedded into the main stream development process and it is not considered as an activity managed by a separate QA team.
Even with following many agile testing principles, why do some teams still fail to deliver products/projects with acceptable quality?
According to my experience, I believe the issues with adopting the QA mindset can be considered as the primary reason for many software quality concerns.
So, what is QA mindset and why is it a critical factor in software testing?

If you are a software developer, your primary objective is to complete the implementation of the feature/module which you have been assigned to do without schedule slippages. But, you are obliged to deliver it with best quality. So, you follow good agile testing principles, may be follow TDD (Test driven development),  code reviews, write automation tests, do manual tests for the scenarios which you cannot automate, run performance  tests etc..
However, when your "well-tested" feature is in UAT at your client's environment  or verified by another person, the bug tracking system is filled up with many bugs.
What went wrong in your testing? Why did you miss all these bugs?

Software development has become more and more complex with all kinds of heterogeneous platform integrations which we use in today's applications. You are no longer expected to have the luxury of testing the traditional 3-tier applications. Instead, you need to think about large array of integration scenarios. The complexity multiplies by many factors when you are developing middleware. For example, what are the implications when your app runs on a cloud provider? what can be the behavior when the application is integrated into various DBMSs? what kind of changes can be expected in feature X of your app when it integrates with a third party enterprise application?
How will the application behave when different message types are processed? Will the feature X provide the same functionality with each message type under various platforms?
There can be endless questions!

You should not even think about a process without comprehensive test automation methodology to address such a complex matrix of test combinations.  However, you can never replace human brain by a test automation tool. The test scenarios, which are the inputs for your test automation tool, have to be derived by you. Thus, it is really important to adopt QA mindset in any testing activity regardless of its nature (automated or manual).

QA mindset is all about the approach of looking at the problem at hand. When you develop a feature, you are mostly optimistic about its functionality since your mind forces you to believe that your creation is correct. You are biased towards your own work.
If you seriously want to avoid a third eye complaining about your creation, always use QA mindset in testing. There is a tester inside all of us. Get him out when you start testing!

  • Avoid pre-mature feature completion announcements. Do not judge the functionality by just observing the positive work flow.
  • Think about all possible integration scenarios. Write them down and execute each very carefully
  • If your component consists of feature X, Y and Z, test every attribute of each feature. Do not leave out anything obvious for a third eye to try and complain
  • Conduct more and more exploratory tests. Research on similar features implemented by others and look for missing use-cases.
  • If you are enhancing an existing feature, collect the issues reported by your customers on the old implementation and assess each of them against the new implementation.
  • Maintain a record of what you have tested so that a separate tester will not be repeating the same test scenarios which you have already tried out
  • Do not be fooled by code coverage figures. Try to break your implementation.
A lot of work.. huh? yes.. but at the end of the day, your work has to be part of a production system. So, quality is not something to compromise. Start to love software testing. That is the only way to deliver quality! 

Sunday, July 21, 2013

Working with HTTP multipart requests in soapUI

You can use HTTP request test step in soapUI to submit messages with various Content-Types.  In this post, we will have a quick look into the multipart/form-data requests in soapUI.

multipart/form-data requests usually come into action when you do HTML form submissions with file attachments. For example, have a look at the following HTML form post.

<form action="http://localhost:8090/CKFileUploadApp/UploadServlet" method="post" enctype="multipart/form-data">
    <input type="file" name="file">
    <input type="text" name="paramName">
    <input type="submit" name="Submit" value="Upload File">

In the forms like above, we have file upload option as a form input where we can upload a file along with the form submission. In that case, the request's HTTP content type will be multipart/form-data. How can we simulate such a request using soapUI?

You can have your own file upload servlet as the backend. But in order to demonstrate the scenario, I have hosted a sample webapp, CKFileUploadApp.war from here. Make sure to change the following context parameter value in WEB-INF/web.xml directory before deploying the web application in Apache tomcat.


Step 1

Start to create a new soapUI project. Specify a name and select Create Web TestCase option.

Add Web TestCase dialog will be displayed. Enter http://localhost:8080/CKFileUploadApp/UploadServlet as the Web Address. Clear Start Recoding Immediately option and click on OK.

Step 2

Once the new HTTP request is added as explained in the first step, change the HTTP method to POST. The request editor should look like the following.

Step 3

Choose multipart/form-data from the Media Type drop down. Now, click on Attachments tab at the bottom of the request editor (see below).


Click on + icon at the top left corner of the attachment window to browse and attach a file to request. Browse for a file in your local file system and add it as an attachment. Now, the attachment editor will be similar to the following.

Step 4

We have everything ready to send the request with a file attachment. Click on submit button to send the request. You will get a HTTP 200 response with the message "Uploaded Filename: <filename>" message. Click on the raw view of the request. You will see the Content-Type of the request is set to multipart/form-data. You will also notice the relevant MIME boundaries of the request.

Since I have attached an xml file to the request, you can observe the Content-Type of the attachment part of the message as text/xml.

Content-Type: text/xml; charset=UTF-8; name=registry.xml

Tuesday, July 16, 2013

OAuth 2.0 grant types with WSO2 API Manager - II - Implicit

This is the second of a series of posts related to OAuth-2.0 grant types in WSO2 API Manager (WSO2 Carbon platform). Therfore, I strongly suggest you to read and work on the examples described in the first post before proceeding with this.

In this post, we will go through Implicit grant type which is the recommended practice if your application (client) is a mobile application or a browser based app such as a JavaScript client.  The key difference of implicit grant when comparing to the Authorization Code is, the client receives access token as the result of the authorization request. In our previous post, which was about Authorization Code grant, client had to make separate requests for authorization and access token. Also note that, the implicit grant does not include client authentication because it does not make use of client secret.

Before attempting to work on the sample, let's have a look at the steps involved in implicit grant type.

1.   Application (client) does a token request from the authorization server by sending a HTTP GET request with the following query parameters.

response_type = token

The first two are mandatory parameters where as the last two can be optional. 

2.  Upon receiving the request, the authorization server must return a 302 redirection back to the client with an Location header pointing to the URL of user consent page. (e.g:- Location: https://localhost:9443/carbon/oauth/oauth2_authn_ajaxprocessor.jsp)

3.  User (resource owner) confirms the authorization requested by client (application) by specifying his credentials.

4. Authorization server redirects user back to the application (to the callback url which has been specified at the first step) with the access token.

Let's explore more on the above steps, using our sample web application (acts as the client/application) and WSO2 API Manager (acts as the authorization server).

Step 1

Access the OAuth playground application as instructed in "Setting up client" section in the previous post. Once you click on "Import Photos" icon, you will be landed in a page where you will find a form with various options such as Authorization Grant Type, Client Id etc..

Step 2

Select Implicit as the Authorization Grant Type. 

Copy the consumer key value from the application you have subscribed in WSO2 API Manager (see above) and enter it in Client Id text box.

Specify any string value as scope. We do not really worry about scope attribute in this example. 

Enter callback URL which must be identical to the value you have specified at the time of creating the new application in WSO2 API Manager. 
e.g:- http://localhost:8090/playground2.0/oauth2client

Enter Authorize endpoint. This should be the endpoint of authorization server where it accepts the authorization requests. In WSO2 API Manager, there is an API to handle all authorization requests and it can be accessed through http://localhost:8280/authorize.

Once you completed adding all values in the form in the playground app, click on Authorize.

This will generate HTTP GET request similar to the following. You can see it contains all mandatory URL parameters which we have discussed in point 1 under the general introduction of "Implicit grant type".

GET /authorize?scope=api_scope&response_type=token&redirect_uri=http%3A%2F%2Flocalhost%3A8090%2Fplayground2.0%2Foauth2client&client_id=ePCzEHajPOZRKus4XS3pva_Ec5Ua HTTP/1.1


User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,es;q=0.7,en;q=0.3

Accept-Encoding: gzip, deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Connection: keep-alive


Cookie: i18next=en-US; region1_configure_menu=none; region3_registry_menu=none; region4_monitor_menu=none; region5_tools_menu=none

Step 3

When you click on "authorize" with all required parameters, the application generates the above HTTP GET call and you will be redirected to the user consent screen as shown below. 

Click on Authorize. You will be provided with options to enter user name and password (username and password of the resource owner/end user).

Type admin/admin as user name and passeword respectively and click on login.

Step 4

You will receive the Access Token as shown below.

Now, we can use this access token to do the actual API call.

We will explore Client Credentials OAuth-2.0 grant type in our next post.

Monday, July 15, 2013

OAuth 2.0 grant types with WSO2 API Manager - I - Authorization Code

WSO2 API Manager is a complete open source solution to manage APIs. It provides authorization and authentication for APIs using OAuth 2.0 standard. According to the OAuth specification, the client needs to get authorization from the resource owner when requesting an access token. The authorization is expressed in the form of an authorization grant, which the client uses to request the access token. There are 4 grant types defined in the OAuth spec.

  • Authorization code
  • Implicit
  • Resource owner password credentials
  • Client credentials

Almost all scenarios explained in the online documentations written on WSO2 API Manager, make use of the Resource owner password credentials grant type. In there, we exchanges access token for user credentials and client_id and client_secret.

I'm going to explain the use of all 4 OAuth grant types using WSO2 API Manager through a serious of blog posts. In this post, I will explain how we can use Authorization Code grant type with WSO2 API Manager.

1. Download and install WSO2 API Manager-1.4.0
2. Install Apache Tomcat

When calling an API hosted in WSO2 API Manager, we need to associate an OAuth access token as part of the requests HTTP header to authenticate the caller.

POST http://localhost:8280/myapi/1.0.0 HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: "urn:echoInt"
Authorization: Bearer 21737b20a287fb4e207bf08bf3b17924

WSO2 API manager consists of multiple components to provide unique functionalities required to manage APIs hosted in it. Authorization server, which is one of such components, responsible for issuing tokens which are needed to authenticate API consumers.  Therefore, before making a API call, the application user should possess the relevant access token with him.

Obtaining access token can be done in multiple ways. In other words, the four of the above OAuth grant types can be used to obtain access token from the authorization server.
Let's see how we can obtain access tokens from each of the OAuth 2 grant types.

Setting up Client (Third party application)

As shown in the above picture, we will use a web application hosted in Apache Tomcat as the client.
Download playground2.0.war from here and copy it to TOMCAT_HOME/webapps directory. Make sure to update the following parameters in WEB-INF/web.xml


localhost is the server where we host WSO2 API Manager and 9443 is the default SSL port of it.
Also, make sure to update param-value of setup to AM as shown below.


Restart tomcat and access http://localhost:8080/playground2.0/ (assuming tomcat is running on HTTP port 8080). You will be directed to the landing page of our sample application. Click on Import Photos and you will be shown the following page.

We can use this application to request access tokens using the four OAuth2 grant types.

Create and subscribe to a new API in WSO2 API Manager

Before exploring OAuth grant types supported by WSO2 API Manager, we need to add an API using the API publisher web app in WSO2 API Manager, publish it to the API gateway component and finally subscribe to the API through API store web app. I'm not going to repeat those steps in this post since I have already explained them in one of my previous posts. You will also find the WSO2 API Manager official documentation useful. However, I will explain one modification which will be important to continue with our example.

In WSO2 API Manager, an application user subscribes to an API or multiple APIs through an application. The OAuth keys are generated per application. Once you have access token in a particular application, you can call any APIs associated with it using the same key.
When you add a new application, you will notice an option to include a Callback URL with the application (see below).


Callback URL is a redirection URI in the client application which is used by the authorization server to send the client's user-agent (usually web browser) back after granting access. In our example, we will use http://localhost:8090/playground2.0/oauth2client as the Callback URL.

Once you create the application, you can subscribe to one or more APIs using the new application. 

Make a note of access token, consumer key and consumer secret specific to your application. 

Authorization code grant type with API Manager


Authorization code grant type is the recommended approach if the client is a web server based application such as a servlet, JSP etc. The authorization code grant type can simply be explained using the following steps.

1.   Application (client) requests an authorization code from the authorization server by sending a HTTP GET request with the following query parameters.

response_type = code

The first two are mandatory parameters where as the last two can be optional. 

2.  The authorization server must return a 302 redirection back to the client with an Location header pointing to the URL of user consent page. (e.g:- Location: https://localhost:9443/carbon/oauth/oauth2_authn_ajaxprocessor.jsp)

3.  User (resource owner) confirms the authorization requested by client (application) by specifying his credentials.

4. Authorization server responds back to the callback URL specified at the first step with an Authorization Code.

5.  Application makes a HTTP POST request to get access token with the following POST parameters

grant_type =  authorization_code
code = authorization_code_value_obtained_from_step_4
redirect_url = redirect_url_of_the_application
client_id = value_of_consumer_id
client_secret =  value_of_consumer_secret

6. Finally, the authorization server returns an access token optionally with a refresh token

Let's explore more on the above steps, using our sample web application (act as the client/application) and WSO2 API Manager (act as the authorization server).

Step 1

Access the OAuth playground application as instructed in "Setting up client" section. Once you click on "Import Photos" icon, you will be landed in a page where you will find a form with various options such as Authorization Grant Type, Client Id etc..

Step 2

Select Authorization Code as the Authorization Grant Type. 

Copy the consumer key value from the application you have subscribed in WSO2 API Manager (see above) and enter it in Client Id text box.

Specify any string value as scope. We do not really worry about scope attribute in this example. 

Enter callback URL which must be identical to the value you have specified at the time of creating the new application in WSO2 API Manager. 
e.g:- http://localhost:8090/playground2.0/oauth2client

Enter Authorize endpoint. This should be the endpoint of authorization server where it accepts the authorization requests. In WSO2 API Manager, there is an API to handle all authorization requests and it can be accessed through http://localhost:8280/authorize.

These APIs can be found in "Deployed APIs" page in API Manager management console (gateway) as shown below.

Once you completed adding all values in the form in the playground app, click on Authorize.

This will generate HTTP GET request similar to the following. You can see it contains all mandatory URL parameters which we have discussed in point 1 under "Authorization Code grant type".

GET /authorize?response_type=code&redirect_uri=http%3A%2F%2Flocalhost%3A8090%2Fplayground2.0%2Foauth2client&client_id=ePCzEHajPOZRKus4XS3pva_Ec5Ua%20&scope=my_scope HTTP/1.1

Connection: keep-alive

User-Agent: Jakarta Commons-HttpClient/3.1


Step 3

When you click on "authorize" with all required parameters, the application generates the above HTTP GET call and you will be redirected to the user consent screen as shown below. 

Click on Authorize. You will be provided with options to enter user name and password (username and password of the resource owner/end user).

Type admin/admin as user name and passeword respectively and click on login.

Step 4

You will be directed to a new form with an Authorization Code. (see below)

Specify the same call back url which you have entered in step 2. (http://localhost:8090/playground2.0/oauth2client)

We are going to obtain an access token from the authorization server. Thus, we need to specify the endpoint URL of the API which returns access tokens. You can find it in the same way as we got the authorize endpoint from  "Deployed APIs" page in API Manager management console (gateway). It will be similar to http://localhost:8280/token

Specify client secret (consumer secret) value which can be found in the relevant application in the API store of  WSO2 API manager (see above).

 Click on Get Access Token. This will generate a HTTP POST request similar to the following.

POST /token HTTP/1.1

Content-Type: application/x-www-form-urlencoded

Cache-Control: no-cache

Pragma: no-cache

User-Agent: Java/1.6.0_29


Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

Connection: keep-alive

Content-Length: 223

Compare this with the mandatory parameters which we have explained under point number 5 in authorization code grant type.

Step 5

You will get an access token (typically a JSON response) which can be used to do the actual API call.  

We will discuss the usage of implicit OAuth grant type in the next post. Stay tuned!!!

Friday, July 5, 2013

Broker trust relationships with WSO2 Identity Server

WS-Trust can be considered as an extension to WS-Security specification which primarily provides methods for managing security tokens and ways to broker the trust relationships. The web services trust model explained under the WS-trust specification defines three key participants.

  • Security token service
  • Service Consumer
  • Service provider (Relying party)

Security token service (STS) is a web service that issues security tokens based on requesters needs. The consumer sends token requests to the STS as well as append tokens into the actual service request and submits them to the provider. The service provider makes the authentication decison on the service based on the token provided by STS. The service provider may also request token validations from the STS.

This post is not yet another detailed explanation about WS-trust and the associated frameworks. You can find many references just by googling the term, ws-trust. Instead, I'm going to take you through a set of step-by-step instructions on how WS-trust concepts can be used with WSO2 SOA middleware. Hope this post will be a starting point for your journey towards learning WS-trust.

Use Case

We are going to work on the basic trust establishment use case in the context of WS-trust specifiction. A SOAP web service consumer requests security token from a STS. We will make use of WSO2 Identity Server as the token provider. WSO2 Identity Server includes a Security Token Service (STS) which is capable of generating security tokens to build trust relationships. 
Once the consumer possesses the necesary security tokens, he presents those tokens to authenticate to a web service deployed in WSO2 Application Server.


1. Download and install WSO2 Identity Server 4.1.0 or later version
2. Download and install WSO2 Application Server 5.0.1 or later version
3. Any Java IDE (Eclipse, Idea)
4. soapUI (optional)

First, we need to figure out how we can start the end-to-end process. Obviously, the consumer initiates the process. But, how does the consumer know that he first needs to talk to a STS in order to consume the web service which is deployed in WSO2 application server? 
The service provider needs to advertise those requirements through a WS-Policy. So, the starting point of our use case is to configure a policy in the web service. 

Then, we need to look into the token provider, WSO2 Identity Server. As I have explained before, WSO2 IS comes with a Security Token Service. We need to configure it to issue tokens to the provider web service and secure STS using a WS-Security policy because the consumers should authenticate themselves to the STS when requesting tokens.

Finally, we will write a client/consumer (or just use a service invocation tool such as soapUI) to request a security token from STS and append it to the actual web service request message to authenticate to the web service which is deployed in WSO2 Application Server.

Step 1

Start WSO2 Application Server and log in to the management console as default admin user. Go to Manage --> Services  --> Add  --> AAR Service and deploy Axis2Service.aar which can be downloaded from here.

Once the service is deployed, we need to associate a security policy for our service. You will find a set of default web service security policies in security configuration wizard. However, we cannot use any of them for our use case. Why?

Our web service should be associated with a trust based security policy. A Security token  which is offered by STS represents a set of claims. A claim defines a specific information about a particular user. For example, firstname or email address of the user.  
In our example, Axis2Service (the web service deployed in service provider, wso2 application server) needs to be made as a claim-aware web service. So that, the users who are going to consume the service should present the claims defined in WS-Security policy.
Thus, we need to secure Axis2Service using a custom policy. You can download the complete policy from here. You might specially be interested in looking into the following element in the custom security policy.

 <sp:SupportingTokens xmlns:sp="http://schemas.xmlsoap.org/ws/2005/07/securitypolicy">
                            <Address xmlns="http://www.w3.org/2005/08/addressing">https://localhost:9444/services/wso2carbon-sts</Address>
                            <t:TokenType xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">urn:oasis:names:tc:SAML:2.0:assertion</t:TokenType>
                            <t:KeyType xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">http://schemas.xmlsoap.org/ws/2005/02/trust/Bearer</t:KeyType>
                            <t:KeySize xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">256</t:KeySize>
                            <t:Claims Dialect="http://wso2.org/claims"
                                <ic:ClaimType Uri="http://wso2.org/claims/givenname"/>
                                <ic:ClaimType Uri="http://wso2.org/claims/emailaddress"/>

SupportingTokens are used to provide additional claims for a requester (client). We will use SupportingToken policy assertion to describe claims which should be given by a consumer to access this particular web service.

The sp:IssuedToken element defines that the requester must talk to an STS to obtain the token.

The sp:IncludeToken attribute value is defined as ../AlwaysToRecipient, which says the token must be included as part of all messages sent from the initiator (service consumer) to the recipient.

The Address attribute of sp:Issuer element defines the token issuer whom the consumer must be contacted to get the security token. In other words, the endpoint address of the Security Token Service. In our case, it must be the STS url of IS.
[Note: I'm running WSO2 IS with port-offset 1, hence the https port is 9444]

The sp:RequestSecurityTokenTemplate element specifies the structure/type of the token which has to be issued by the STS. Later, you will observe the children of this element will be included as the part of RequestSecurityToken (RST) message which will be sent to the STS by consumer. 

The t:TokenType element represents the type of the token such as UserNameToken, SAML-1.1, SAML-2.0. In our example, we ask STS To generate SAML-2.0 token by specifying SAML-2.0 token type URL.

t:KeyType represents the type of the key desired in security token, can be either public key, symmetric key or bearer. We will ask STS to generate the relatively simple key, bearer token. Bearer token does not require a proof of possession hence it is quite easy to dealt with.

t:Claims element defines the claims that must be included in the security token. We will request First Name and email address claims from the consumer.

You may raise the question why the First Name claim has been mapped to "http://wso2.org/claims/givenname" URI. We will be able to clear ourselves once we configure the token provider in the next step.

OK, now we understood all elements of the SupportingToken assertion of our custom web service policy. Let's upload the policy to embedded registry in WSO2 Application Service.

Locate /_system/config collection in registry browser and click on Add Resource.
Then, browse the axis2service.policy.xml which you have downloaded from the above location and click on Add.

Now, we have our custom policy in WSO2 Application Server. We can associate the policy to Axis2Service.

Click on Unsecured link in Axis2Service in Deployed Services page. You will be directed to Security for the service page. Find Policy From Registry section which is at the bottom of the page. Click on Configuration Registry icon and select the axis2service.policy.xml from /_system/config collection.
Click on next to proceed with the wizard. You will be at the Activate Security page where you can specify a trusted key store for the web service.
We will select wso2carbon.jks as the trusted key store. We have already discussed that STS is responsible for issuing tokens. Therefore, the STS is trusted by both consumer as well as the service provider. In order to do so, the public key certificate of STS should be imported to the trusted key store defined here.  We will not explicitly do key exchanges in this example because we are using the same wso2carbon.jks keystore in both WSO2 Application Server and WSO2 Identity Server.

Now, we have our web service secured with a claim-aware security policy. Let's move in to the STS configurations.

Step 2

Start WSO2 Identity Server and log in to the management console using the default admin credentials. Select Secure Token Service from the left menu under the Manage section.

First, we should secure the STS. There are multiple ways to authenticate to the Secure Token Service. The consumer can just present his username and password as UserNameTokens or X509 certificates to the STS. In this example, we will configure the STS using UserNameToken Authentication policy so that the clients should talk to STS by presenting their credentials.

Click on Apply Security Policy in the above screen. Select UsernameToken from the list of default security policies in the Security for the service page. Click on Next.
Now, we need to select a user group to which the service consumer belongs. We will incorporate our consumer into the admin group and request tokens from STS by presenting the default admin user credentials. Therefore, select admin as the user group and click on finish.

Next, we need to configure STS to add Axis2Service as a trusted service. In the above screen, enter the http endpoint url of the Axis2Service.

Endpoint Address = http://localhost:9765/services/Axis2Service/

Usually, the security token is signed by STS. Thus, we need to select a certificate alias to sign the token. We will select the default wso2carbon certificate alias.

Now, let's look at the user profile of admin user who is going to authenticate to the STS. Click on My Profiles at the left menu. Update Profile form will be displayed where you can enter various user attributes such as First Name, Last Name etc.
Make sure to add some values to First Name and Email address fields since we are going to use those as the required claims.

Click on Configure in the left menu and select Claim Management. You will find a set of claim dialects associated with the internal user store in IS. Click on the default claim dialect, http://wso2.org/claims. Click on First Name claim mapping.
As you can see in the following screen shot, First Name is mapped to givenName attribute.

Now, you should be able to understand why we have specified http://wso2.org/claims/givenName as the claimuri for First Name attribute in the service policy.

Step 3

Now, all what left is to work on the service consumer. Basically, we need to generate the RequestSecurityToken programatically using a client, insert into the web service request and send to Axis2Service.
The wst:RequestSecurityToken element is used to request a security token from STS. 
It will be a child of SOAP body. At the minumum level, the RequestSecurityToken element will be similar to the following. 
<wst:RequestSecurityToken xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust">

            <wst:Claims xmlns:wsp="http://schemas.xmlsoap.org/ws/2005/02/trust"
                <wsid:ClaimType xmlns:wsid="http://schemas.xmlsoap.org/ws/2005/05/identity"
                <wsid:ClaimType xmlns:wsid="http://schemas.xmlsoap.org/ws/2005/05/identity"

You would like to compare this with the SupportingToken policy assertion which we defined in step 1. In this, we will request a SAML2 token from STS (see wst:TokenType element) and indicate that the keytype is bearer. We also define the two claims we would present to the web service, givenname and emailaddress.

You can include RequestSecurityToken element into a SOAP message body and send to STS. You can use soapUI SOAP request editor as shown below.

Since we are using usernametoken authentication when submitting token request to STS, specify username and password under "Aut" tab in soapUI. Also specify WSS-passwordType as plaintext and WSS timeto live to some integer value. You can find more information about securing requests with username token with soapUI in one of my blog posts. Also, make sure to enable WS-Addressing for the token request as explained in another blog post of mine.
With all these, when you submit the request to STS endpoint (in our case, https://localhost:9444/services/wso2carbon-sts), you will get a response with the generated token (RequestSecurityTokenResponse). You will notice it in the response view of the above screen shot.
Now, you can extract the saml2:Assertion element from the response and embed it with the actual web service request message. For that, you can try adding SAML WSS entry for soapUI request and copy the extracted saml2:Assertion element into the Enter SAML Assertion text area as shown in the following screen shot. However, I was unsuccessful sending a message with bearer confirmation method with soapUI-4.5.2. It seems soapUI still supports sender vouches confirmation method only.

Because of this limitation, we need to follow a programmatic approach to insert the token into the web service request and forward to Axis2Service.
You can find the complete working client in this sourceforge account

Steps to run the client

1. Check out the complete source from https://sourceforge.net/p/charithablogsam/code/ci/master/tree/  and import the source into your IDE
2. Go to IS_HOME/bin and type ant. This will copy the necessary client libraries in to IS_HOME/repository/lib directory
3. Add IS_HOME/repository/lib directory to your IDE class path
4. Also add IS_HOME/repository/components/lib/bcprov-jdk15-132.jar to the class path
5. Modify Axis2ServiceClient according to the file paths in your system wherever necessary.
Policy stsPolicy = loadSTSPolicy("/resources/policies/sts.ut.policy.xml");
6. Run the client

You should see the output similar to the following.

<saml2:Assertion xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xs="http://www.w3.org/2001/XMLSchema" ID="urn:uuid:D6511FA799FD47AC4D1373081003469" IssueInstant="2013-07-06T03:23:23.465Z" Version="2.0"><saml2:Issuer>localhost</saml2:Issuer><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
<ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"></ds:CanonicalizationMethod>
<ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"></ds:SignatureMethod>
<ds:Reference URI="#urn:uuid:D6511FA799FD47AC4D1373081003469">
<ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"></ds:Transform>
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"><ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"></ec:InclusiveNamespaces></ds:Transform>
<ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"></ds:DigestMethod>
O4d1DeGHT/YnIjs9JogRKv4XHECwLtIVdAbIdWHEtVZJyMSktcyysFcvuhPQK8Qc/E/Wq8uHSCo=</ds:X509Certificate></ds:X509Data></ds:KeyInfo></ds:Signature><saml2:Subject><saml2:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">admin</saml2:NameID><saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"></saml2:SubjectConfirmation></saml2:Subject><saml2:Conditions NotBefore="2013-07-06T03:23:23.465Z" NotOnOrAfter="2013-07-06T03:28:23.465Z"></saml2:Conditions><saml2:AuthnStatement AuthnInstant="2013-07-06T03:23:23.465Z"><saml2:AuthnContext><saml2:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:Password</saml2:AuthnContextClassRef></saml2:AuthnContext></saml2:AuthnStatement><saml2:AttributeStatement><saml2:Attribute Name="http://wso2.org/claims/emailaddress" NameFormat="http://wso2.org/claims/emailaddress"><saml2:AttributeValue xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">charitha@wso2.com</saml2:AttributeValue></saml2:Attribute><saml2:Attribute Name="http://wso2.org/claims/givenname" NameFormat="http://wso2.org/claims/givenname"><saml2:AttributeValue xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">charitha</saml2:AttributeValue></saml2:Attribute></saml2:AttributeStatement></saml2:Assertion>
Response  : <ns:echoStringResponse xmlns:ns="http://service.carbon.wso2.org"><ns:return>Hello world1</ns:return></ns:echoStringResponse>

Comment on or contact me directly at uoccharitha@gmail.com if this does not work for you!

Sunday, June 30, 2013

Helping hands in the world of enterprise middleware - Technical Support

Recently, I got an opportunity to work on a domain which is bit different from my subject matter specialization, software testing and quality assurance. I started to work as a lead of technical support team at WSO2 since November 2012. I thought to put together some key observations I made in technical support career in general and how I position it in terms of my beliefs.

Willingness to help - the most important characteristic of a support engineer

Are you really a helping hand regardless of your profession? How can I understand my willingness in helping others. I would simply list down some questions to evaluate your likelihood to help others.

Do you get frustrated when someone requests your help? If someone asks a question, what is your initial reaction?
In our life, we have seen people who do not even look at the person who requested help. Some people just route the request to some other party without even understanding what kind of help is required.

If you are really a helping hand, you just do not do it because of your professional obligations. You do it simply because you like to help others. You enjoy by observing how your help solves some other's problems.

If you are generously help people in need without hesitation, you possess one of the fundamental characteristics expected from a good technical support engineer.

Listening to others and clear in communication

A good support engineer always listen to others very carefully. She raises many follow-up queries to get the question clarified. She does not attempt to solve the problem blindly without completely understanding the issue at hand. She also communicates very clearly with the clients as well as the other stake holders to help in providing the the best answer.

Expertise in user perspective

Support engineers are clients advocates who should look at the problem through customer point of view. How painful is this issue to my clients? How does it affect their business transactions? When you approach problem from the customer point of view, you first look at what needs to be done before considering how you are going to address the concerns.

Enthusiastic in learning new technologies

To help needy people, you must have a good understanding about the relevant subject matter. You will be clueless if you do not know the problem domain. This is one of the most important attributes especially  in middleware technical support. You will not at least understand a basic question, if you do not step in to learn the technology. The technical support engineer should eager to learn various technologies and find out various solutions to address common problems that can occur.
At WSO2, technology becomes obsolete quite frequently. The technical support engineer must stay up-to-date with the latest features and enhancements introduced to the product platform.

Focus on SLA

Generally, technical support contracts are based on different Service Level Agreements (SLA). There can be various levels of SLA policies. A good support engineer always consider his SLA is now and act fast to find the best resolution for client's issue.

More experiences on technical support, specially related to WSO2 product platform are to be followed....

Friday, June 28, 2013

Invoking WSO2 Carbon admin services with soapUI

The management aspects of WSO2 Carbon platform are primarily achieved through SOAP web services interface known as admin services. All Carbon products ship with a management console (front-end user interface) which communicates with these web services and provides users with various administration capabilities.

In some situations, we need to by-pass the management UI and call the backend web services directly. Specially, in test automation, it is important to minimize the risk of frequent UI changes hence focusing on admin service interactions can be considered as a viable solution. WSO2 test automation framework is built upon this approach which programatically calls the backend web services to manage deployment, configuration and various other tasks.

These backend web services are secured to prevent anonymous invocations. WSO2 Carbon server secures these services through multiple methodologies. For example;

  • HTTP Basic Authentication over SSL
  • WS-Security username token
  • Session based authentication
You can use any SOAP client and communicates with the admin services by authenticating through above security protocols.

In this post, I will take  you through consuming an admin service using soapUI since soapUI is the most user-friendly service testing tool out there to test SOAP or RESTful web services.

We will use HTTP basic auth authentication mechanism out of the auth options described above. If you like to use a different approach such as carbon session based authentication, you may refer to Nandika's blog post.

soapUI 4.5.1 or later
WSO2 Carbon 4.X version (You can use any member product of WSO2 Carbon family)

Step 1

By default, the WSDLs of admin web services are hidden from consumers. Therefore, we need to enable them first to import admin service WSDL into a soapUI project.

Open CARBON_HOME/repository/conf/carbon.xml and set HideAdminServiceWSDLs property to false.


Start WSO2 Carbon server.

Step 2


There are large number of admin services which serve many administration and management functionalities such as ServiceAdmin, StatisticsAdmin, ProxyAdmin etc.. However, I will use a simplest admin service, UserAdmin for this example.

Open  https://localhost:9443/services/UserAdmin?wsdl in your browser and check whether it is accessible.

Step 3

Create a SOAP web service project in soapUI using the above WSDL

Step 4

We are going to invoke one of the web services operations exposed through UserAdmin web service. Select listAllUsers operation under UserAdminSoap11Binding interface and click on SOAP request.

Step 5

We are going to submit listAllUsers SOAP request using HTTP basic authentication headers. Therefore, click on Aut tab under the bottom of the request editor and specify the admin  user name and password of carbon server.

Enter the following values for the required parameters of the SOAP request.


Step 6

Now, we can submit the request to UserAdmin web service.You will get a SOAP response with all users available in the user store of Carbon server.

Friday, June 21, 2013

How to use SecureVault when WSO2 Carbon servers are started as background processes

SecureVault can be used to encrypt the plain-text passwords specified in various configurations files in WSO2 Carbon products. You can find more information about how to secure plain-text passwords using securevault in this blog written by Asela.

When you use SecureVault to encrypt the passwords as explained there, you are supposed to specify the primary keystore password at the server startup. However, this is not possible when you start the server as a background process.

This post summarizes the complete procedure of securing plain text passwords using secure vault and additional configurations when you start the server as a background process.

Suppose, we need to encrypt the LDAP ConnectionPassword value in CARBON_HOME/repository/conf/user-mgt.xml

  1. Locate cipher-text.properties which can be found at CARBON_HOME/repository/conf/security directory
  2.  Keep a back up of the cipher-text.properties file
  3.  Now, remove all key-value pairs which have there by default in cipher-text.properties file.(In this example, we just need to encrypt ConnectionPassword value)
  4. Add the following line. Make sure to include your plain_text LDAP connection password in [plain_text_ldap_password]

  5.  Locate ciphertool.sh script which can be found at CARBON_HOME/bin directory
  6.  Run ciphertool.sh as follows
      ciphertool.sh -Dconfigure

    This will prompt "[Please Enter Primary KeyStore Password of Carbon Server : ]" message. Enter "wso2carbon" as the primary keystore password
  7.  If the script execution completed successfully, you will see the following message.
    "Secret Configurations are written to the property file successfully"
  8.  Now, go back and look at the cipher-text.properties file. The plain text LDAP password will be replaced by a cipher value.
  9.  You will also look at CARBON_HOME/repository/conf/user-mgt.xml where we have specified the connection password for LDAP user.
    You will notice that it will be modified by the ciphertool script as follows.
    <Property name="ConnectionPassword" svns:secretAlias="UserStoreManager.Property.ConnectionPassword">password</Property>
  10. Now, you can start the server.
    e.g:- sh wso2server.sh
  11. This will prompt "[Enter KeyStore and Private Key Password :]" at the server startup because we need to decrypt the encrypted passwords to connect to LDAP.
    You can enter "wso2carbon" and the server will be started successfully.

    But you will not be able to provide this password value if you start WSO2 Carbon server as a background process.
    i.e:- ./wso2server.sh start
    In that case, you can follow a simple set of additional steps as explained below
  12. Have a file named "password-tmp" in CARBON_HOME/ directory. Add "wso2carbon" (the primary keystore password) to this file and save
  13. Now, start the server as a background process.
    ./wso2server.sh start
  14. Keystore password will be picked up from password-tmp file. Once the server is started, this fill will automatically be deleted from the file system. Make sure to add this temporary file back whenever you start the sever as a background process.
          NOTE : If you make the name of the password file as "password-persist" instead of "password-tmp" then the fie will not be deleted after reading. Then you don't need to provide the password in subsequent startups.


Saturday, May 11, 2013

The simplest possible way to simulate a backend service delay

When testing service integrations, we usually want to simulate the delayed responses from backend web services. For example, if you want to test how WSO2 ESB or Apache Synapse reacts when the backend service takes large amount of time for responding, there are many approaches to introduce delay to backend web service. I have observed that, most of the people modify the backend web services to add Thread.sleep() when they need to introduce a delay in service invocation.

If you are not testing the backend web service and just want to test the integration (e.g:- outgoing calls from ESB), I cannot think of a better solution than using soapUI mock services.

Step 1


Add a new mock service to soapUI project. This can be done at the time of creating the project or by selecting a particular interface (binding) of a wsdl based project.

Step 2


Select the mock service in soapUI navigator and open the mock service editor. In the mock service editor, select OnRequest script.

Step 3


Add the following groovy script inside the OnRequest Script editor to introduce 1 minute delay.


Now, send a request to the mock service. You will notice that all requests will respond back to the caller after 1 minute of delay.

Read chapter 6 (Web service simulation with soapUI) and 11 (Extending soapUI with Scripting) of Web Services Testing with soapUI book for more information about soapUI scripting capabilities.