Sunday, July 15, 2012

Advantages of mod_cluster over mod_jk

Advantages of mod_cluster vs mod_jk

Hello All,

Maximum times I have heard about mod_jk. So, I was a little bit confused that which one better and better performer than others. So I have just searched and found really good and interesting answer and sharing this as below. So if anyone knows something interesting or new about the comparison of these both, then please share your thoughts.

Overview

mod_cluster is an httpd-based load balancer. Like mod_jk and mod_proxy, mod_cluster uses a communication channel to forward requests from httpd to one of a set of application server nodes. Unlike mod_jk and mod_proxy, mod_cluster leverages an additional connection between the application server nodes and httpd. The application server nodes use this connection to transmit server-side load balance factors and lifecycle events back to httpd via a custom set of HTTP methods, affectionately called the Mod-Cluster Management Protocol (MCMP). This additional feedback channel allows mod_cluster to offer a level of intelligence and granularity not found in other load balancing solutions.

Within httpd, mod_cluster is implemented as a set of modules for httpd with mod_proxy enabled. Much of the logic comes from mod_proxy, e.g. mod_proxy_ajp provides all the AJP logic needed by mod_cluster.

Advantages

mod_cluster boasts the following advantages over other httpd-based load balancers:

Dynamic configuration of httpd workers: Traditional httpd-based load balancers require explicit configuration of the workers available to a proxy. In mod_cluster, the bulk of the proxy's configuration resides on the application servers. The set of proxies to which an application server will communicate is determined either by a static list or using dynamic discovery via the advertise mechanism. The application server relays lifecycle events (e.g. server startup/shutdown) to the proxies allowing them to effectively auto-configure themselves. Notably, the graceful shutdown of a server will not result in a failover response by a proxy, as is the case with traditional httpd-based load balancers.
Server-side load balance factor calculation: In contrast with traditional httpd-based load balancers, mod_cluster uses load balance factors calculated and provided by the application servers, rather than computing these in the proxy. Consequently, mod_cluster offers a more robust and accurate set of load metrics than is available from the proxy. (see Load Metrics for more)
Fine grained web-app lifecycle control: Traditional httpd-based load balancers do not handle web application undeployments particularly well. From the proxy's perspective requests to an undeployed web application are indistinguishable from a request for an non-existent resource, and will result in 404 errors. In mod_cluster, each server forwards any web application context lifecycle events (e.g. web-app deploy/undeploy) to the proxy informing it to start/stop routing requests for a given context to that server.
AJP is optional: Unlike mod_jk, mod_cluster does not require AJP. httpd connections to application server nodes can use HTTP, HTTPS, or AJP.

Saturday, June 16, 2012

Configuring mod_cluster to have JBoss nodes disabled by default on join

Issue

When we start or restart a JBoss node, it joins with mod_cluster and mod_cluster automatically enables traffic to it. Can we configure mod_cluster so that the node is disabled by default on join, and then enabled once we specify?

Environment

mod_cluster
JBoss Enterprise Web Server (EWS) 1.0.x
JBoss Enterprise Application Platform (EAP) 5.x

Resolution

This functionality specifically is not yet available.
You can workaround this default behavior to only manually turn on the node or connect it with mod_cluster when you desire through the following steps:
- Turn off advertisement and communicate through the proxyList.
- Once you've disabled advertisement on the httpd and JBoss end, set the proxyList to be some ip/port that is not a valid one for it to join mod_cluster. You can set this in with the system property jboss.modcluster.proxyList or in deploy/mod-cluster.sar/META-INF/mod-cluster-jboss-beans.xml:
```
        <property name="proxyList">${jboss.modcluster.proxyList:127.0.0.1:6665}</property>
```
Now with apache/mod_cluster up, boot up your jboss instance. It will start but throw an error about not being able to connect to this nonsense proxylist.
After JBoss is started and you want it to join mod_cluster and become accessible, you can change the proxyList to the correct value. You can do this by going to the jmx-console and following the "service=ModCluster" link under the jboss.web section. From here, invoke the addProxy option. p1 should be the address and p2 the port. It should then be able to successfully connect to mod_cluster and become accessible. You should also then remove the initial dummy proxyList value through the removeProxy invocation.
Note that you can also invoke the addProxy and removeProxy operations through twiddle:
- twiddle.sh -s 127.0.0.1:1099 -u admin -p admin invoke "jboss.web:service=ModCluster" addProxy 127.0.0.1 6666 twiddle.sh -s 127.0.0.1:1099 -u admin -p admin invoke "jboss.web:service=ModCluster" removeProxy 127.0.0.1 6665
- Note that for this workaround to work you must use the ModClusterService listener and not the HAModClusterService listener. If you use the HAModClusterService listener, then when you bring a node back up it will communicate with mod_cluster via the cluster master's valid proxyList, thus enabling it by default.
Root Cause
JBoss nodes are enabled by default upon joining with mod_cluster.

Thanks,

Kuldeep Sharma

Enable friendly access log times in Apache and JBoss

How do I enable friendly access log times in Apache and JBoss?

Hi All,

For every troubleshooter, it will be great if provided logs have well identified Time Stamps. With well mentioned time stamps, we can analyse the issues easy and come to solution within less time and comfotably.

Environment

JBoss Enterprise Application Platform (EAP)
- 4.x
- 5.x
Apache Web Server

The default access log timing in Apache and JBoss are in milliseconds and are not conducive to quickly identifying high access times, or for easily tracing through the flow of a request from Apache to JBoss using mod_jk for example. One approach that has worked well in modifying the default configuration is to add a Time Taken string to the beginning of the log pattern followed by the request time in seconds, not milliseconds.

To enable the Time Taken in JBoss locate the following piece of code in $JBOSS_HOME/server/$CONFIG/deploy/$JBOSSWEB/server.xml

<!--
<Valve className="org.apache.catalina.valves.AccessLogValve"
       prefix="localhost_access_log." suffix=".log"
       pattern="common" directory="${jboss.server.home.dir}/log"
       resolveHosts="false" />
-->

and change the pattern="common" to pattern="Time Taken: %T %h %l %u %t %r %s %b" The resulting xml, will look like

<Valve className="org.apache.catalina.valves.AccessLogValve"
       prefix="localhost_access_log." suffix=".log"
       pattern="Time Taken: %T %h %l %u %t %r %s %b" directory="${jboss.server.home.dir}/log"
       resolveHosts="false" />

Now, in Apache's httpd.conf, on RHEL /etc/httpd/conf/httpd.conf, change LogFormat "%h %l %u %t \"%r\" %>s %b" common to LogFormat "Time Taken: %T %h %l %u %t \"%r\" %>s %b" common

After these changes, the access logs generated will be in a format that is much easier to digest at first sight.

%T logs response time in seconds. When greater precision is needed, use the %D option to log response time in microseconds.

!Hope This will Help

Kuldeep Sharma

Thursday, May 10, 2012

What Version of RESTEasy is included in JBOSS EAP?

RESTEasy Version Info for following JBOSS EAP.

Hi All,
This is small info about which version of RESTEasy is currently being used with JBOSS EAP.

Environment :
JBoss Enterprise Application Platform (EAP)

EAP Version	RESTEasy Version
5.0.0	1.1.CP01
5.1.0	1.2.1
5.1.1	1.2.1.GA_CP01
5.1.2	1.2.1.GA_CP02
6.0	2.3.1 (in definition)

Thanks!!

Kuldeep Sharma

Friday, March 2, 2012

Configuring JBoss clustering through JAVA_OPTS

Hi Friends,
There are different way to configure Jboss Clustering. Here you can see one on them...

Configuring JBoss clustering through JAVA_OPTS:

Issue

I need to configure my cluster through JAVA_OPTS. What settings can I use for this?

Environment

JBoss Enterprise Application Platform (EAP)
- 5.1.0

Resolution

The following all are the JAVA_OPTS needed to set up clustering:

-Djboss.messaging.ServerPeerID=
-Djboss.bind.address=
-Djboss.partition.name=
-Djgroups.udp.mcast_addr=
-Djboss.partition.udpGroup=

Regards,

Kuldeep Sharma

Jboss Monitoring Tips

Hi Friends,

Below is the some information about memory and threads monitoring in Jboss. For this you can use jmx-console of Jboss. Just open your jmx-console (e.g http://localhost:8080/jmx-console). Then on left panel you will find jboss.system. After this just click on jboss.system:type=ServerInfo on right panel.

Memory and Thread Monitoring:

The jboss.system:type=ServerInfo MBean provides several attributes that can be used to monitor the thread and memory usage in a JBoss instance. These attributes can be monitored in many ways: through the JMX Console, from a third-party JMX management tool, from shell scripts using the twiddle command, etc... The most interesting attributes are shown below

FreeMemory

This is the current free memory available in the JVM.

ActiveThreadCount

This is the number of active threads in the JVM.

ActiveThreadGroupCount

This is the number of active thread groups in the JVM.

These are useful metrics for monitoring and alerting, but developers and administrators need a little more insight than this. The Java 5 JVMs from Sun provide more detailed information about the current state of the JVM. Some of these details are exposed by JBoss through operations on the SystemInfo MBean.

listMemoryPools

This operations shows the size and current usage of all JVM memory pools. This operation is only available when using Java 5.

listThreadDump

This operations shows all threads currently running in the JVM. When using Java 5, JBoss will display a complete stack trace for each thread, showing you exactly what code each thread is executing.

listThreadCpuUtilization

This operations shows all threads currently running in the JVM along with the total CPU time each thread has used. The operation is only available in Java 5.

!Hope this will help you

Regards,

Kuldeep Sharma

About Ports Used in Jboss

Hi All,
This is small information about different ports used by jboss. Usually we know we get familiar with only few commonly used ports(e.g. 8080, 8009 etc). But actually there are some more which are internally configured for different services.

Supposing we have bound JBoss AS to the IP Address 10.2.20.156, here's the list of ports engaged by the application server:
TCP    10.2.20.156:1090       <--- RMI/JRMP socket for connecting to the JMX MBeanServer
TCP    10.2.20.156:1091       <--- RMI server socket
TCP    10.2.20.156:1099       <--- JNDI Service
TCP    10.2.20.156:1098       <--- RMI Port for JNDI Service
TCP    10.2.20.156:4446       <--- JBoss Remoting Connector
TCP    10.2.20.156:4712       <--- JBossTS Service
TCP    10.2.20.156:4713       <--- JBossTS Service
TCP    10.2.20.156:5445       <--- HornetQ JMS Service
TCP    10.2.20.156:5455       <--- HornetQ Netty port
TCP    10.2.20.156:5500       <--- JBoss Remoting
TCP    10.2.20.156:5501       <--- JBoss Remoting ssl
TCP    10.2.20.156:8009       <--- Web server AjpConnector
TCP    10.2.20.156:8080       <--- Web server HTTP Connector
TCP    10.2.20.156:8083       <--- Web services

Clustered servers will additionally open the following ports:

TCP    10.2.20.156:1100       <--- HA JNDI Service
TCP    10.2.20.156:1101       <--- HA JNDI Service
TCP    10.2.20.156:3528       <--- Corba IIOP Service

The Service Binding Manager
The Service Binding Manager co-ordinates which ports the JBoss Application Server instance listens on when it starts up and the Administration Console provides a mechanism for configuring this service to change which ports get used.
The Service Binding Manager can be used to change the default port sets, without changing the single service ports.
Out of the box, the JBoss Application Server ships with four port sets defined as:
* ports-default: The standard ports, for instance JBoss HTTP is on 8080.
* ports-01: The standard ports incremented by 100. For example, HTTP is on 8180
* ports-02: The standard ports incremented by 200. For example, HTTP is on 8280
* ports-03: The standard ports incremented by 300. For example, HTTP is on 8380
In order to change the port set you can edit the file bindings-jboss-beans.xml and set for example the ports-01
<parameter>${jboss.service.binding.set:ports-01}</parameter>
You can as well change the Service Binding ports from the start-up script of the application server. For example:
run -Djboss.service.binding.set=ports-01

Note : This is pretty useful if you don't have to maintain a separate copy of the all folder if you want to, say, start up multiple instances of JBoss running as a cluster.

Regards,
Kuldeep Sharma

JDBC connection pool running out of connections in JBoss EAP

Issue

Application Server runs out of available connections in the connection pool
Message appears in the server.log file

·         No ManagedConnections available within configured blocking timeout

Server hangs

Environment

JBoss Enterprise Application Platform (EAP)
- 4.x
- 5.x

Resolution

Increase your connection pool size. Add the following

<max-pool-size>whatever</max-pool-size>

to the *-ds.xml file to increase the number of connections available. The default is 20.

The following attribute of the ManagedConnectionPool MBean in the jmx-concole, for the datasource in question, can help understand how many connections are really needed.

MaxConnectionsInUse

Tell your threads to wait longer for a connection to open up. Add the following

<blocking-timeout-millis>whatever</blocking-timeout-millis>

to the *-ds.xml file to increase the length of time to wait. The default is 30000 milli-seconds.

Fix the offending code so that connections are closed. Especially if you see -

Closing a connection for you. Please close them yourself

Diagnostic Steps

Ensure the CachedConnectionValve is installed and CachedConnectionManager debugging is enabled in order to show where the connection leak is coming from. The CachedConnectionValve is installed by default in EAP. CachedConnectionManager debugging depends on the server profile (enabled in default and all, disabled in production).
Configuration details is as follows:

EAP 4.x
- CachedConnectionValve: JBOSS_HOME/server/$PROFILE/deploy/deploy/jboss-web.deployer/server.xml
- CachedConnectionManager is configured in JBOSS_HOME/server/$PROFILE/deploy/jbossjca-service.xml . Debugging is enabled as follows:

  <mbean code="org.jboss.resource.connectionmanager.CachedConnectionManager"

         name="jboss.jca:service=CachedConnectionManager">

    <depends optional-attribute-name="TransactionManagerServiceName">jboss:service=TransactionManager</depends>

    <!-- Enable connection close debug monitoring -->

    <attribute name="Debug">true</attribute>

EAP 5.x
- CachedConnectionValve: JBOSS_HOME/server/$PROFILE/deploy/deploy/jbossweb.sar/server.xml
- CachedConnectionManager is configured in JBOSS_HOME/server/$PROFILE/deploy/jca-jboss-beans.xml. Debugging is enabled as follows:

  <bean name="CachedConnectionManager" class="org.jboss.resource.connectionmanager.CachedConnectionManager">

      <!-- Whether to track unclosed connections and close them -->

     <property name="debug">true</property>

Search server.log for the following:

INFO [CachedConnectionManager] Closing a connection for you. Please close them yourself

Review associated stack trace to find the code that's not releasing connections.
Identify the code which is leaking the connection. By using the CachedConnectionManager from the JMX Console:

Log into the JMX Console (e.g. http://jboss-host:8080/jmx-console).
Scroll down to the "jboss.jca" section.
Click the "service=CachedConnectionManager" link.
Scroll down and invoke the "listInUseConnections" operation.
A stack trace for each in-use connection will be printed. (Only if CachedConnectionManager is in debug mode)

Take subsequent readings from listInUseConnections() say after the interval of 10 seconds and figure out from the trace which application code is holding the connections for longer time. This will help in detecting the application code which probably would be leaking the connections.
Look at the database connections from the database side to see what SQL is running on the connections. Make sure there is not any long running SQL or recursion bugs in the SQL that could be causing the connections to never be released.

Regards

Kuldeep Sharma