Mantis - Resin
|
|||||
Viewing Issue Advanced Details | |||||
|
|||||
ID: | Category: | Severity: | Reproducibility: | Date Submitted: | Last Update: |
4499 | minor | always | 04-12-11 14:26 | 06-02-11 17:37 | |
|
|||||
Reporter: | ferg | Platform: | |||
Assigned To: | ferg | OS: | |||
Priority: | normal | OS Version: | |||
Status: | closed | Product Version: | 4.0.14 | ||
Product Build: | Resolution: | fixed | |||
Projection: | none | ||||
ETA: | none | Fixed in Version: | 4.0.19 | ||
|
|||||
Summary: | 0004499: bam/heartbeat synchronization | ||||
Description: |
(rep by William Kong) Last night, our system produced these where exception messages. We have two load balanced servers running. Here's what I did: 1. Login / Authenticate into our Web Application. (i.e. Our logic to authenticate with the user using Resin and Oracle Database.) 2. Authenticated successfully and redirected to our landing page. 3. <Error occurs here> Search for an account (i.e. Our logic to search accounts using Resin Oracle Database.) I waited and waited and eventually get a one liner Services Not Available (sorry, don't have a screen shot). 4. <Debugging Steps> I went to look at both server logs and found the exception below. 5. <Debugging Steps> I just stop only one of the Resin Web Server 6. <Debugging Steps> I go back to step 3 above to hit the refresh button to resubmit my post request. However, the application re-routes to be the login page. From the step, I assuming there was a session load balancing issue. 7. <Debugging Steps> I did not restart the other resin server instance and it seems like I have not gotten any new complaints similar to below: Is there a session load balancing issue? What can I do to prevent this issue from re-occurring again? [04/12/11 00:35:26] [11-04-12 00:35:26.155] {server://10.8.9.28:6800-431} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@1a8c6fdb query timeout: 60000 to: cluster-cache@aaa.estmt.admin.resin query: CacheGet[key=88ca0aae,version=0,server=0] at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65) at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179) at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642) at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126) at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137) at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53) at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311) at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271) at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247) at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106) at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496) at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611) at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308) at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255) at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198) at _jsp._foo__jsp._jspService(_acctdetails__jsp.java:160) at com.caucho.jsp.JavaPage.service(JavaPage.java:64) at com.caucho.jsp.Page.pageservice(Page.java:542) at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194) at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184) at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162) at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95) at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287) at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468) at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373) at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339) at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723) at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685) at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665) at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613) at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104) at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98) at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81) at com.caucho.network.listen.AcceptTask.run(AcceptTask.java:67) at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164) at com.caucho.env.thread.ResinThread.run(ResinThread.java:130) [04/12/11 00:36:41] [11-04-12 00:36:41.191] {server://10.8.9.28:6800-466} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@4ecf3c56 query timeout: 60000 to: cluster-cache@aaa.estmt.admin.resin query: CacheGet[key=88ca0aae,version=0,server=0] at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65) at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179) at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642) at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126) at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137) at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53) at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311) at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271) at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247) at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106) at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496) at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611) at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308) at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255) at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198) at _jsp._foo__jsp._jspService(_acctdetails__jsp.java:160) at com.caucho.jsp.JavaPage.service(JavaPage.java:64) at com.caucho.jsp.Page.pageservice(Page.java:542) at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194) at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184) at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162) at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95) at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287) at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468) at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373) at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339) at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723) at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685) at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665) at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613) at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104) at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98) at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81) at com.caucho.network.listen.AcceptTask.run(AcceptTask.java:67) at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164) at com.caucho.env.thread.ResinThread.run(ResinThread.java:130) [04/12/11 00:37:56] [11-04-12 00:37:56.227] {server://10.8.9.28:6800-467} [^] com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@d2b24c7 query timeout: 60000 to: cluster-cache@aaa.foo.admin.resin query: CacheGet[key=88ca0aae,version=0,server=0] at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:65) at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:179) at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:642) at com.caucho.server.distcache.CacheMnodeActor.get(CacheMnodeActor.java:126) at com.caucho.server.distcache.CacheMnodeManager.get(CacheMnodeManager.java:137) at com.caucho.server.distcache.CacheClusterBackingImpl.loadClusterValue(CacheClusterBackingImpl.java:53) at com.caucho.server.distcache.AbstractCacheManager.loadExpiredValue(AbstractCacheManager.java:311) at com.caucho.server.distcache.AbstractCacheManager.reloadValue(AbstractCacheManager.java:271) at com.caucho.server.distcache.AbstractCacheManager.getMnodeValue(AbstractCacheManager.java:247) at com.caucho.server.distcache.ProCacheEntry.getMnodeValue(ProCacheEntry.java:106) at com.caucho.distcache.AbstractCache.getExtCacheEntry(AbstractCache.java:496) at com.caucho.server.session.SessionImpl.load(SessionImpl.java:611) at com.caucho.server.session.SessionManager.createSession(SessionManager.java:1308) at com.caucho.server.http.AbstractCauchoRequest.createSession(AbstractCauchoRequest.java:255) at com.caucho.server.http.AbstractCauchoRequest.getSession(AbstractCauchoRequest.java:198) at _jsp._statements._acctdetails__jsp._jspService(_acctdetails__jsp.java:160) at com.caucho.jsp.JavaPage.service(JavaPage.java:64) at com.caucho.jsp.Page.pageservice(Page.java:542) at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194) at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:184) at com.caucho.server.cache.ProxyCacheFilterChain.doFilter(ProxyCacheFilterChain.java:162) at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95) at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:287) at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:468) at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:373) at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:339) at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:723) at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:685) at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:665) at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:613) at com.caucho.network.listen.AcceptTask.doTask(AcceptTask.java:104) at com.caucho.network.listen.ConnectionReadTask.runThread(ConnectionReadTask.java:98) at com.caucho.network.listen.ConnectionReadTask.run(ConnectionReadTask.java:81) |
||||
Steps To Reproduce: | |||||
Additional Information: | |||||
Relationships | |||||
Attached Files: |
Notes | |||||
|
|||||
|
|