Mantis Bugtracker
Main | My View | View Issues | Change Log | Docs
  

Viewing Issue Advanced Details [ Jump to Notes ] [ View Simple ] [ Issue History ] [ Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0004499 [Resin] minor always 04-12-11 14:26 06-02-11 17:37
Reporter ferg View Status public  
Assigned To ferg
Priority normal Resolution fixed Platform
Status closed   OS
Projection none   OS Version
ETA none Fixed in Version 4.0.19 Product Version 4.0.14
  Product Build
Summary 0004499: bam/heartbeat synchronization
Description (rep by William Kong)

Last night, our system produced these where exception messages. We have two load balanced servers running.

Here's what I did:

1. Login / Authenticate into our Web Application. (i.e. Our logic to authenticate with the user using Resin and Oracle Database.)
2. Authenticated successfully and redirected to our landing page.
3. <Error occurs here> Search for an account (i.e. Our logic to search accounts using Resin Oracle Database.) I waited and waited and eventually get a one liner Services Not Available (sorry, don't have a screen shot).
4. <Debugging Steps> I went to look at both server logs and found the exception below.
5. <Debugging Steps> I just stop only one of the Resin Web Server
6. <Debugging Steps> I go back to step 3 above to hit the refresh button to resubmit my post request. However, the application re-routes to be the login page. From the step, I assuming there was a session load balancing issue.
7. <Debugging Steps> I did not restart the other resin server instance and it seems like I have not gotten any new complaints similar to below:

Is there a session load balancing issue? What can I do to prevent this issue from re-occurring again?




[04/12/11 00:35:26] [11-04-12 00:35:26.155] {server://10.8.9.28:6800-431} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@1a8c6fdb query timeout: 60000
                                                                           to: cluster-cache@aaa.estmt.admin.resin
                                                                           query: CacheGet[key=88ca0aae,version=0,server=0]
                                                                                 at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65)
                                                                                 at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179)
                                                                                 at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642)
                                                                                 at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126)
                                                                                 at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137)
                                                                                 at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247)
                                                                                 at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106)
                                                                                 at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496)
                                                                                 at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611)
                                                                                 at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198)
                                                                                 at _jsp._foo__jsp._jspService(_acctdetails__jsp.java:160)
                                                                                 at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
                                                                                 at com.caucho.jsp.Page.pageservice(Page.java:542)
                                                                                 at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
                                                                                 at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184)
                                                                                 at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162)
                                                                                 at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
                                                                                 at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339)
                                                                                 at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613)
                                                                                 at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81)
                                                                                 at com.caucho.network.listen.AcceptTask.run(AcceptTask.java:67)
                                                                                 at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)
                                                                                 at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)
[04/12/11 00:36:41] [11-04-12 00:36:41.191] {server://10.8.9.28:6800-466} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@4ecf3c56 query timeout: 60000
                                                                           to: cluster-cache@aaa.estmt.admin.resin
                                                                           query: CacheGet[key=88ca0aae,version=0,server=0]
                                                                                 at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65)
                                                                                 at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179)
                                                                                 at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642)
                                                                                 at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126)
                                                                                 at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137)
                                                                                 at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247)
                                                                                 at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106)
                                                                                 at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496)
                                                                                 at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611)
                                                                                 at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198)
                                                                                 at _jsp._foo__jsp._jspService(_acctdetails__jsp.java:160)
                                                                                 at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
                                                                                 at com.caucho.jsp.Page.pageservice(Page.java:542)
                                                                                 at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
                                                                                 at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184)
                                                                                 at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162)
                                                                                 at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
                                                                                 at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339)
                                                                                 at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613)
                                                                                 at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81)
                                                                                 at com.caucho.network.listen.AcceptTask.run(AcceptTask.java:67)
                                                                                 at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)
                                                                                 at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)
[04/12/11 00:37:56] [11-04-12 00:37:56.227] {server://10.8.9.28:6800-467} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@d2b24c7 query timeout: 60000
                                                                           to: cluster-cache@aaa.foo.admin.resin
                                                                           query: CacheGet[key=88ca0aae,version=0,server=0]
                                                                                 at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65)
                                                                                 at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179)
                                                                                 at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642)
                                                                                 at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126)
                                                                                 at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137)
                                                                                 at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271)
                                                                                 at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247)
                                                                                 at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106)
                                                                                 at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496)
                                                                                 at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611)
                                                                                 at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255)
                                                                                 at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198)
                                                                                 at _jsp._statements._acctdetails__jsp._jspService(_acctdetails__jsp.java:160)
                                                                                 at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
                                                                                 at com.caucho.jsp.Page.pageservice(Page.java:542)
                                                                                 at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
                                                                                 at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184)
                                                                                 at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162)
                                                                                 at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
                                                                                 at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373)
                                                                                 at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339)
                                                                                 at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665)
                                                                                 at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613)
                                                                                 at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98)
                                                                                 at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81)
 
Steps To Reproduce
Additional Information
Attached Files

- Relationships

- Notes
(0005290)
ferg
06-02-11 17:37

 cloud/60o1
 

- Issue History
Date Modified Username Field Change
04-12-11 14:26 ferg New Issue
06-02-11 17:37 ferg Note Added: 0005290
06-02-11 17:37 ferg Assigned To  => ferg
06-02-11 17:37 ferg Status new => closed
06-02-11 17:37 ferg Resolution open => fixed
06-02-11 17:37 ferg Fixed in Version  => 4.0.19

Mantis 1.0.0rc3[^]
Copyright © 2000 - 2005 Mantis Group
28 total queries executed.
25 unique queries executed.
Powered by Mantis Bugtracker