Mantis Bugtracker
  

Viewing Issue Simple Details Jump to Notes ] View Advanced ] Issue History ] Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0005336 [Resin] minor always 01-17-13 07:13 01-17-13 10:05
Reporter cowan View Status public  
Assigned To ferg
Priority high Resolution fixed  
Status closed   Product Version 4.0.33
Summary 0005336: Startup delay with multiple clusters: cannot open heartbeat connection to TriadServer: future timeout: 120000ms
Description Previously it took about one minute to start resin, and now it takes about five minutes. Interestingly, the startup time goes back down if I misconfigure it so the front-end server cannot find the back end server, for example by making the back end resin run on a different port to that declared in the cluster configuration for the front end. With finer logging enabled I can see errors like the following in the logs while it is starting up. If I change resin-embed to listen on a different port the startup time goes back down to one minute (although obviously the cluster is not available).

[2013-01-11 18:23:40.372] AllMethodCallback[3,AllMethodScoreboard[1,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.risk-dashboard.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:24:40.212] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.212] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[impersonation,0,127.0.0.1:6801]
[2013-01-11 18:24:40.213] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.213] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[administration,0,127.0.0.1:6802]
[2013-01-11 18:24:40.213] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.213] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[risk-dashboard,0,127.0.0.1:6803]
[2013-01-11 18:25:40.214] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.215] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[impersonation,0,127.0.0.1:6801]
[2013-01-11 18:25:40.215] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.215] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[administration,0,127.0.0.1:6802]
[2013-01-11 18:25:40.215] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.216] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[risk-dashboard,0,127.0.0.1:6803]
[2013-01-11 18:25:40.368] java.lang.IllegalStateException: com.caucho.bam.query.QueryFutureCallback@146ccf3e future timeout: 120000ms
                               at com.caucho.bam.query.QueryFutureCallback.get(QueryFutureCallback.java:108)
                               at com.caucho.bam.proxy.BamProxyHandler$QueryCall.invoke(BamProxyHandler.java:160)
                               at com.caucho.bam.proxy.BamProxyHandler.invoke(BamProxyHandler.java:117)
                               at $Proxy43.ping(Unknown Source)
                               at com.caucho.cloud.globalcache.GlobalCacheManager.sendPing(GlobalCacheManager.java:330)
                               at com.caucho.cloud.globalcache.GlobalCacheManager.start(GlobalCacheManager.java:134)
                               at com.caucho.cloud.globalcache.GlobalCacheSystem.start(GlobalCacheSystem.java:117)
                               at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:529)
                               at com.caucho.env.service.ResinSystem.start(ResinSystem.java:496)
                               at com.caucho.server.resin.Resin.start(Resin.java:849)
                               at com.caucho.server.resin.Resin.initMain(Resin.java:981)
                               at com.caucho.server.resin.Resin.main(Resin.java:1437)
[2013-01-11 18:25:40.370] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->administration,127.0.0.1:6802]
                           to: global-cache@aaa.administration.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=14,to=global-cache@aaa.administration.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->impersonation,127.0.0.1:6801]
                           to: global-cache@aaa.impersonation.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=12,to=global-cache@aaa.impersonation.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803]
                           to: global-cache@aaa.risk-dashboard.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=16,to=global-cache@aaa.risk-dashboard.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] AllMethodCallback[1,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.impersonation.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:25:40.370] AllMethodCallback[2,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.administration.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:25:40.371] AllMethodCallback[3,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.risk-dashboard.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]

Additional Information Rep by D. Shearer
Attached Files

- Relationships

There are no notes attached to this issue.

- Issue History
Date Modified Username Field Change
01-17-13 07:13 cowan New Issue
01-17-13 10:05 ferg Assigned To  => ferg
01-17-13 10:05 ferg Status new => closed
01-17-13 10:05 ferg Resolution open => fixed
01-17-13 10:05 ferg Fixed in Version  => 4.0.34


Mantis 1.0.0rc3[^]
Copyright © 2000 - 2005 Mantis Group
27 total queries executed.
25 unique queries executed.
Powered by Mantis Bugtracker