Mantis - Resin
Viewing Issue Advanced Details
ID: Category: Severity: Reproducibility: Date Submitted: Last Update:
5336 minor always 01-17-13 07:13 01-17-13 10:05
Reporter: cowan Platform:  
Assigned To: ferg OS:  
Priority: high OS Version:  
Status: closed Product Version: 4.0.33  
Product Build: Resolution: fixed  
Projection: none      
ETA: none Fixed in Version: 4.0.34  
Summary: 0005336: Startup delay with multiple clusters: cannot open heartbeat connection to TriadServer: future timeout: 120000ms
Description: Previously it took about one minute to start resin, and now it takes about five minutes. Interestingly, the startup time goes back down if I misconfigure it so the front-end server cannot find the back end server, for example by making the back end resin run on a different port to that declared in the cluster configuration for the front end. With finer logging enabled I can see errors like the following in the logs while it is starting up. If I change resin-embed to listen on a different port the startup time goes back down to one minute (although obviously the cluster is not available).

[2013-01-11 18:23:40.372] AllMethodCallback[3,AllMethodScoreboard[1,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.risk-dashboard.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:24:40.212] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.212] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[impersonation,0,127.0.0.1:6801]
[2013-01-11 18:24:40.213] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.213] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[administration,0,127.0.0.1:6802]
[2013-01-11 18:24:40.213] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.213] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[risk-dashboard,0,127.0.0.1:6803]
[2013-01-11 18:25:40.214] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.215] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[impersonation,0,127.0.0.1:6801]
[2013-01-11 18:25:40.215] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.215] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[administration,0,127.0.0.1:6802]
[2013-01-11 18:25:40.215] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.216] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[risk-dashboard,0,127.0.0.1:6803]
[2013-01-11 18:25:40.368] java.lang.IllegalStateException: com.caucho.bam.query.QueryFutureCallback@146ccf3e future timeout: 120000ms
                               at com.caucho.bam.query.QueryFutureCallback.get(QueryFutureCallback.java:108)
                               at com.caucho.bam.proxy.BamProxyHandler$QueryCall.invoke(BamProxyHandler.java:160)
                               at com.caucho.bam.proxy.BamProxyHandler.invoke(BamProxyHandler.java:117)
                               at $Proxy43.ping(Unknown Source)
                               at com.caucho.cloud.globalcache.GlobalCacheManager.sendPing(GlobalCacheManager.java:330)
                               at com.caucho.cloud.globalcache.GlobalCacheManager.start(GlobalCacheManager.java:134)
                               at com.caucho.cloud.globalcache.GlobalCacheSystem.start(GlobalCacheSystem.java:117)
                               at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:529)
                               at com.caucho.env.service.ResinSystem.start(ResinSystem.java:496)
                               at com.caucho.server.resin.Resin.start(Resin.java:849)
                               at com.caucho.server.resin.Resin.initMain(Resin.java:981)
                               at com.caucho.server.resin.Resin.main(Resin.java:1437)
[2013-01-11 18:25:40.370] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->administration,127.0.0.1:6802]
                           to: global-cache@aaa.administration.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=14,to=global-cache@aaa.administration.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->impersonation,127.0.0.1:6801]
                           to: global-cache@aaa.impersonation.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=12,to=global-cache@aaa.impersonation.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803]
                           to: global-cache@aaa.risk-dashboard.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=16,to=global-cache@aaa.risk-dashboard.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] AllMethodCallback[1,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.impersonation.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:25:40.370] AllMethodCallback[2,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.administration.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:25:40.371] AllMethodCallback[3,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.risk-dashboard.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
Steps To Reproduce:
Additional Information: Rep by D. Shearer
Relationships
Attached Files:

There are no notes attached to this issue.