Hi,
I've been having issues with tomcat session replication. I have 20 tomcat servers running in a cluster mode, behind an Apache load balancer. The tomcat version is 6.0.18 on CentOS 5.1. Running the cluster using the DeltaManager seems to be working fine, however when I try to use the BackupManager for session replication, I get the following entries in the logs:
Apr 1, 2009 3:28:42 AM org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl <at> 62af9d74 last access:2009-04
-01 03:28:35.969
Apr 1, 2009 3:28:42 AM org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no
interest ops for the last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl <at> 4c4947d3 last access:2009-04-01 03:28:35.969
Apr 1, 2009 3:29:04 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 99, 86, 47}:4000,{10, 99, 86, 47},4000, alive=1380182,id={-121 25 -2 -7 81 -1 76 3 -92 -20 122 69 67 102 -31 -15 }, payload={}, command={}, domain={}, ]] message. Will verify.
Apr 1, 2009 3:29:04 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 99, 86, 47}:4000,{10, 99, 86, 47},4000, alive=1380182,id={-121 25 -2 -7
81 -1 76 3 -92 -20 122 69 67 102 -31 -15 }, payload={}, command={}, domain={}, ]]
Apr 1, 2009 3:29:04 AM org.apache.catalina.tribes.tipis.AbstractReplicatedMap
heartbeat
SEVERE: Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Operation has timed out(60000 ms.).; Faulty members:tcp://{10, 99, 86, 47}:4000;
at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
&n
bsp; at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at org.apache.catalina.tr
ibes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:89)
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:253)
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:793)
at
org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:153)
at
org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:661)
Of course the above entry is just one of many, for the different hosts. Searching the mailing lists, I found this post http://markmail.org/message/jv4dykh7fdhr4mvp which looks like the same problem I am having. The outcome of that thread states that the problem is fixed by a patch in revision 618823, so I compiled a version of the current 6.x trunk (rev 759722) and deployed it to all the servers. However, the problem is still appearing. I've attached a copy of the current server.xml ( it is common to all tomcat instances ).I've done a thread dump on one of the servers when these errors started appear
ing, and the output is attached, thread_dump.txt (removed threads that were running by our application).
This problem is reproducable each time I restart the
servers. At this stage, I'm clueless on what to try next, so I'm looking forward to your replies.
Regards,
Jim.
Attached: server.xml, thread_dump.txt