Popular Posts

Friday, February 28, 2014

Common mistakes to avoid in WSO2 Carbon - 2 - "java.sql.SQLException: Total number of available connections are less than the total number of closed connections"

This is the second post of common mistakes blog series which I'm planning to share with you. In this post, we are looking into another common mistake which we do when working with WSO2 Carbon platform.

Registry mounting is a way of federating the registry space across multiple servers in a product cluster. For example, if you have a WSO2 ESB cluster, you can use a single registry space to store all configuration data common to cluster nodes.
There are 3 different registry spaces provided by each WSO2 Carbon product; local, configuration and governance. You can find more details about these spaces in here.

We have to keep in mind a few important concepts when building a shared registry setup. You cannot share the local registry space among multiple cluster nodes. The local registry space is used to store node-specific data hence it should not be shared among other nodes in the cluster. However, we mistakenly do this when configuring shared registry setups and experience many unexpected issues. The following weird startup error is one such occurrence due to incorrect mounting configurations (I removed some part of the complete stack trace for clarity).

ERROR - RegistryCoreServiceComponent Failed to activate Registry Core bundle
org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to close transaction.
    at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.endTransaction(JDBCTransactionManager.java:183)
--   

Caused by: java.sql.SQLException: Total number of available connections are less than the total number of closed connections
    at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCDatabaseTransaction$ManagedRegistryConnection.close(JDBCDatabaseTransaction.java:1349)
    at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.endTransaction(JDBCTransactionManager.java:178)


This error does not give any clue about a problem related to mounting. You may have spent many hours unnecessarily to tune up your DBMS since the error says about DB connections! 

Let's explore this error in detail.

Step 1

 

We are going to have a shared registry database (which is used as configuration and governance registry spaces in a ESB cluster). I will use mySQL and create a database first.

mysql> create database sharedreg_db;


Next, create the registry DB schema using mySQL database scripts available in CARBON_HOME/dbscripts directory.

mysql> use sharedreg_db;
mysql> source /home/charitha/products/esb/tmp/wso2esb-4.8.1/dbscripts/mysql.sql;


Step 2

 

We will register this new database in master-datasources.xml which can be found at
CARBON_HOME/repository/conf/datasources directory

<datasource>
            <name>WSO2_SHARED_REGISTRY_DB</name>
            <description>The datasource used for shared registry</description>
            <jndiConfig>
                <name>jdbc/WSO2SharedRegDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/sharedreg_db</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

Step 3

 


Now, we have a shared registry database. We need to mount the registry collections to this remote database. There are 3 mounting mechanisms; JDBC, Atom and WS. The commonly used pattern is JDBC mounting. Hence I will use the same.
Mounting configuration can be done in CARBON_HOME/repository/conf/registry.xml as shown below.

<remoteInstance url="https://localhost:9443/registry">

        <id>instanceid</id>

        <dbConfig>wso2registry</dbConfig>

        <readOnly>false</readOnly>

        <enableCache>true</enableCache>

        <registryRoot>/</registryRoot>

    </remoteInstance>

    <mount path="/_system/config" overwrite="true">

        <instanceId>instanceid</instanceId>

        <targetPath>/_system/nodes</targetPath>

    </mount>
    <mount path="/_system/governance" overwrite="true">

        <instanceId>instanceid</instanceId>

        <targetPath>/_system/governance</targetPath>

    </mount>

Make a note of the highlighted configuration parameter.

Now, the database configuration referred by the mount is defined at the top of registry.xml as follows.
We simply change the JNDI name of the default db config; jdbc/WSO2CarbonDB to the JNDI name of our shared registry database; jdbc/WSO2SharedRegDB

 <currentDBConfig>wso2registry</currentDBConfig>

    <readOnly>false</readOnly>

    <enableCache>true</enableCache>

    <registryRoot>/</registryRoot>


    <dbConfig name="wso2registry">

        <dataSource>jdbc/WSO2SharedRegDB</dataSource>

    </dbConfig>



OK. Assuming everything is configured correctly, we start Carbon server. Unfortunately, you will get the above meaningless error at the server startup.

What is wrong here?

 


By defining a common data source for mount configuration as well as local registry definition (under currentDBConfig element), we have done a big mistake. This will eventually leads to share local registry space among heterogeneous product cluster nodes which is theoretically incorrect.  

How can we fix this?

 

Simple. You can define a separate, unique database configuration for the shared registry db.

<dbConfig name="sharedregistry">

        <dataSource>jdbc/WSO2SharedRegDB</dataSource>

    </dbConfig>


Then, that will be referenced by the remote mounting configuration.

<remoteInstance url="https://localhost:9443/registry">

        <id>instanceid</id>

        <dbConfig>sharedregistry</dbConfig>

        <readOnly>false</readOnly>

        <enableCache>true</enableCache>

        <registryRoot>/</registryRoot>

    </remoteInstance>

Finally, make sure to change the local registry definition back to its default so that it will use the WSO2 Carbon DB (usually H2).


<dbConfig name="wso2registry">

        <dataSource>jdbc/WSO2CarbonDB</dataSource>

    </dbConfig>

Restart the server. The error will disappear!

Saturday, February 22, 2014

Common mistakes to avoid in WSO2 ESB - 1 - "org.apache.axis2.AxisFault: The system cannot infer the transport information from the URL"

In the next few weeks, you could expect a series of blog posts from me which explain the remedies to avoid some common mistakes which we do when working with WSO2 ESB. This is the first of many.

During the initial rounds of testing your integration solution or even in production systems, you may have come across the following error.

org.apache.axis2.AxisFault: The system cannot infer the transport information from the URL ....

Some of you may have struggled hours figuring out a solution to address this issue. Let me explain this error in detail and possible causes of this. So, next time when you see this error in your WSO2 ESB setup, you will not have to spend time unnecessarily googling everywhere.

There can be many different causes of this error. I have observed  four main reasons.

Cause 1


This is the simplest cause. Suppose, you have a proxy service similar to the following.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="MistakesTest"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <send/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>
                                

Look at the inSequence of the above proxy. We have defined send mediator without address information. Now, when you send a typical SOAP request over HTTP to this proxy service, ESB does have no way to find where to route the request. The HTTP transport sender which is supposed to route the HTTP request clueless where to forward the request and fails with the following error.

ERROR - ClientUtils The system cannot infer the transport information from the /services/MistakesTest URL.ERROR - Axis2Sender Unexpected error during sending message out
org.apache.axis2.AxisFault: The system cannot infer the transport information from the /services/MistakesTest URL.
    at org.apache.axis2.description.ClientUtils.inferOutTransport(ClientUtils.java:81)


How can we fix this? There are many approaches.

First, you can simply associate address information to the send mediator by defining an endpoint.

        <send>
            <endpoint>
               <address uri="http://localhost:8088/mockaxis2service"/>
            </endpoint>
         </send>

Or else, you can seek help of WS-Addressing framework. You can fix your client application to send your request with WS-Addressing To header as shown below. Then, Axis2 transport sender in WSO2 ESB, extracts the wsa:To header value of the request and forwards the message to back-end service.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://service.carbon.wso2.org">
   <soapenv:Header xmlns:wsa="http://www.w3.org/2005/08/addressing"><wsa:Action>urn:echoString</wsa:Action><wsa:MessageID>uuid:86b1f69b-f4a2-4f4c-b5c9-54fea095972e</wsa:MessageID><wsa:To>http://localhost:8088/mockaxis2service</wsa:To></soapenv:Header>
   <soapenv:Body>
      <ser:echoString>
          <ser:s>charitha</ser:s>
      </ser:echoString>
   </soapenv:Body>
</soapenv:Envelope>

We have discussed about the simplest cause for the error which we are trying to solve. Next, let's try to understand another common reason.

Cause 2


Suppose, you have a proxy service similar to the following.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="MistakesTest"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <send>
            <endpoint>
               <default/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>

Here, we have defined a default endpoint, which does not have an address URI associated with it and the address information is resolved through wsa:To header of the incoming request.
Now, if you send a SOAP request, without wsa:To header, you will get the above error.

How can we fix this?

Obviously, we can include wsa:To addressing header in the request (as explained before).
Or else, you can add an header mediator before send mediator in the above configuration which will set wsa:To header of the request.

<header name="To" value="http://localhost:9090/mockaxis2service"/>

Cause 3


As I explained above, axis2 transport sender, defined at axis2.xml is responsible for forwarding the incoming request.

e.g:-
 <transportSender name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpSender">

HTTP and HTTPS transport senders are enabled by default in axis2.xml. Think about a use case similar to the following.
You need to do a transport protocol switching through WSO2 ESB, which is a quite common use case in any ESB. In that case, suppose you have a proxy service similar to the following.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="MistakesTest"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <send>
            <endpoint>
               <address uri="jms:/queue1?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&amp;java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&amp;java.naming.provider.url=tcp://localhost:61616&amp;transport.jms.DestinationType=queue"/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>
                                

In this proxy configuration, the proxy service acts as a JMS producer and forwards the request coming through HTTP channel to a JMS queue (queue1). If you try this scenario, in vanilla version of WSO2 ESB without any modification, you will get the following error.

ERROR - Axis2Sender Unexpected error during sending message out
org.apache.axis2.AxisFault: The system cannot infer the transport information from the jms:/queue1?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&                 java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&java.naming.provider.url=tcp://localhost:61616&transport.jms.DestinationType=queue URL


How can we fix this?

In the default axis2 configuration, the HTTP transport senders are only enabled. Thus, when you we are trying to forward a message through non-HTTP transport such as JMS (or VFS, SAP, FiX, mail etc), ESB cannot finds the transport sender registered against the transport protocol define in endpoint url (in this example, "jms:/")
Thus, you need to stop ESB and enable the relevant transport sender in ESB_HOME/repository/conf/axis2/axis2.xml as shown below.

<transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender"/>

Cause 4


Let's conclude our discussion by looking at another common reason for "The system cannot infer transport..." error.

When you are working with scenarios related to blocking transport senders, for example, Callout mediator or message processors, you may have come across the same error. These elements use blocking transports to call the external services hence they cannot make use of the default axis configuration defined in ESB_HOME/repository/conf/axis2/axis2.xml.
Think about a scenario, where you use message processor to query a queue and forward the message to an external file server through VFS transport or route to a SAP endpoint.
In this case, even if you enabled, the relevant transportSenders in the default axis2.xml, message processor or Callout mediator is not aware of that. They read the axis configuration from the following locations by default.

In ESB-4.8.0 or above:
ESB_HOME/repository/conf/axis2/axis2_blocking_client.xml

In ESB-4.7.0 or below:
ESB_HOME/samples/axis2Client/client_repo/conf/axis2.xml

Thus, if you do not enable transport senders in those locations, you get the same error.


Back in blogosphere

I have been away from blogging for more than 2 months. Many of the readers of this blog, asked me if I stopped writing or what. The last few months were hectic due to a many reasons. First, I moved to Bloomington, a beautiful small town in Indiana, USA from my home country, Sri Lanka. Thus, I had to focus on many initial settlement stuff which allowed me very little time to scan through my blog. Next, I had to spend time on adjusting to the totally new environment specially crazy coldness in here.
I believe I'm back in business now. As the tag line of this blog says, "no nonsense", only SOA and testing. Therefore, I'm not going to write about what is happening around me. I will continue sticking to the objective of this blog. Helping at least one person who is stuck figuring out an answer for a problem related to Service-Oriented architecture, enterprise integration or software testing.