Mantis - Resin
Viewing Issue Advanced Details
4499 minor always 04-12-11 14:26 06-02-11 17:37
ferg  
ferg  
normal  
closed 4.0.14  
fixed  
none    
none 4.0.19  
0004499: bam/heartbeat synchronization
(rep by William Kong)

Last night, our system produced these where exception messages. We have two load balanced servers running.

Here's what I did:

1. Login / Authenticate into our Web Application. (i.e. Our logic to authenticate with the user using Resin and Oracle Database.)
2. Authenticated successfully and redirected to our landing page.
3. <Error occurs here> Search for an account (i.e. Our logic to search accounts using Resin Oracle Database.) I waited and waited and eventually get a one liner Services Not Available (sorry, don't have a screen shot).
4. <Debugging Steps> I went to look at both server logs and found the exception below.
5. <Debugging Steps> I just stop only one of the Resin Web Server
6. <Debugging Steps> I go back to step 3 above to hit the refresh button to resubmit my post request. However, the application re-routes to be the login page. From the step, I assuming there was a session load balancing issue.
7. <Debugging Steps> I did not restart the other resin server instance and it seems like I have not gotten any new complaints similar to below:

Is there a session load balancing issue? What can I do to prevent this issue from re-occurring again?




[04/12/11 00:35:26] [11-04-12 00:35:26.155] {server://10.8.9.28:6800-431} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@1a8c6fdb query timeout: 60000
                                                                           to: cluster-cache@aaa.estmt.admin.resin
                                                                           query: CacheGet[key=88ca0aae,version=0,server=0]
                                                                                 at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65)
                                                                                 at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179)
                                                                                 at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642)
                                                                                 at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126)
                                                                                 at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137)
                                                                                 at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247)
                                                                                 at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106)
                                                                                 at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496)
                                                                                 at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611)
                                                                                 at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198)
                                                                                 at _jsp._foo__jsp._jspService(_acctdetails__jsp.java:160)
                                                                                 at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
                                                                                 at com.caucho.jsp.Page.pageservice(Page.java:542)
                                                                                 at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
                                                                                 at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184)
                                                                                 at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162)
                                                                                 at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
                                                                                 at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339)
                                                                                 at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613)
                                                                                 at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81)
                                                                                 at com.caucho.network.listen.AcceptTask.run(AcceptTask.java:67)
                                                                                 at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)
                                                                                 at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)
[04/12/11 00:36:41] [11-04-12 00:36:41.191] {server://10.8.9.28:6800-466} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@4ecf3c56 query timeout: 60000
                                                                           to: cluster-cache@aaa.estmt.admin.resin
                                                                           query: CacheGet[key=88ca0aae,version=0,server=0]
                                                                                 at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65)
                                                                                 at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179)
                                                                                 at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642)
                                                                                 at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126)
                                                                                 at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137)
                                                                                 at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247)
                                                                                 at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106)
                                                                                 at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496)
                                                                                 at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611)
                                                                                 at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198)
                                                                                 at _jsp._foo__jsp._jspService(_acctdetails__jsp.java:160)
                                                                                 at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
                                                                                 at com.caucho.jsp.Page.pageservice(Page.java:542)
                                                                                 at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
                                                                                 at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184)
                                                                                 at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162)
                                                                                 at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
                                                                                 at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339)
                                                                                 at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613)
                                                                                 at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81)
                                                                                 at com.caucho.network.listen.AcceptTask.run(AcceptTask.java:67)
                                                                                 at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)
                                                                                 at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)
[04/12/11 00:37:56] [11-04-12 00:37:56.227] {server://10.8.9.28:6800-467} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@d2b24c7 query timeout: 60000
                                                                           to: cluster-cache@aaa.foo.admin.resin
                                                                           query: CacheGet[key=88ca0aae,version=0,server=0]
                                                                                 at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65)
                                                                                 at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179)
                                                                                 at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642)
                                                                                 at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126)
                                                                                 at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137)
                                                                                 at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247)
                                                                                 at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106)
                                                                                 at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496)
                                                                                 at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611)
                                                                                 at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198)
                                                                                 at _jsp._statements._acctdetails__jsp._jspService(_acctdetails__jsp.java:160)
                                                                                 at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
                                                                                 at com.caucho.jsp.Page.pageservice(Page.java:542)
                                                                                 at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
                                                                                 at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184)
                                                                                 at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162)
                                                                                 at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
                                                                                 at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339)
                                                                                 at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613)
                                                                                 at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81)
 

Notes
(0005290)
ferg   
06-02-11 17:37   
cloud/60o1