0005336: Startup delay with multiple clusters: cannot open heartbeat connection to TriadServer: future timeout: 120000ms

Viewing Issue Simple Details [ Jump to Notes ]

[ View Advanced ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0005336

[Resin]

minor

always

01-17-13 07:13

01-17-13 10:05

Reporter

cowan

View Status

public

Assigned To

ferg

Priority

high

Resolution

fixed

Status

closed

Product Version

4.0.33

Summary

0005336: Startup delay with multiple clusters: cannot open heartbeat connection to TriadServer: future timeout: 120000ms

Description

Previously it took about one minute to start resin, and now it takes about five minutes. Interestingly, the startup time goes back down if I misconfigure it so the front-end server cannot find the back end server, for example by making the back end resin run on a different port to that declared in the cluster configuration for the front end. With finer logging enabled I can see errors like the following in the logs while it is starting up. If I change resin-embed to listen on a different port the startup time goes back down to one minute (although obviously the cluster is not available).

[2013-01-11 18:23:40.372] AllMethodCallback[3,AllMethodScoreboard[1,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.risk-dashboard.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:24:40.212] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.212] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[impersonation,0,127.0.0.1:6801]
[2013-01-11 18:24:40.213] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.213] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[administration,0,127.0.0.1:6802]
[2013-01-11 18:24:40.213] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.213] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[risk-dashboard,0,127.0.0.1:6803]
[2013-01-11 18:25:40.214] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.215] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[impersonation,0,127.0.0.1:6801]
[2013-01-11 18:25:40.215] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.215] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[administration,0,127.0.0.1:6802]
[2013-01-11 18:25:40.215] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.216] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[risk-dashboard,0,127.0.0.1:6803]
[2013-01-11 18:25:40.368] java.lang.IllegalStateException: com.caucho.bam.query.QueryFutureCallback@146ccf3e future timeout: 120000ms
                               at com.caucho.bam.query.QueryFutureCallback.get(QueryFutureCallback.java:108)
                               at com.caucho.bam.proxy.BamProxyHandler$QueryCall.invoke(BamProxyHandler.java:160)
                               at com.caucho.bam.proxy.BamProxyHandler.invoke(BamProxyHandler.java:117)
                               at $Proxy43.ping(Unknown Source)
                               at com.caucho.cloud.globalcache.GlobalCacheManager.sendPing(GlobalCacheManager.java:330)
                               at com.caucho.cloud.globalcache.GlobalCacheManager.start(GlobalCacheManager.java:134)
                               at com.caucho.cloud.globalcache.GlobalCacheSystem.start(GlobalCacheSystem.java:117)
                               at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:529)
                               at com.caucho.env.service.ResinSystem.start(ResinSystem.java:496)
                               at com.caucho.server.resin.Resin.start(Resin.java:849)
                               at com.caucho.server.resin.Resin.initMain(Resin.java:981)
                               at com.caucho.server.resin.Resin.main(Resin.java:1437)
[2013-01-11 18:25:40.370] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->administration,127.0.0.1:6802]
                           to: global-cache@aaa.administration.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=14,to=global-cache@aaa.administration.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->impersonation,127.0.0.1:6801]
                           to: global-cache@aaa.impersonation.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=12,to=global-cache@aaa.impersonation.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803]
                           to: global-cache@aaa.risk-dashboard.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=16,to=global-cache@aaa.risk-dashboard.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] AllMethodCallback[1,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.impersonation.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:25:40.370] AllMethodCallback[2,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.administration.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:25:40.371] AllMethodCallback[3,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.risk-dashboard.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]

Additional Information

Rep by D. Shearer

Attached Files

Relationships

There are no notes attached to this issue.

Issue History
Date Modified	Username	Field	Change
01-17-13 07:13	cowan	New Issue
01-17-13 10:05	ferg	Assigned To	=> ferg
01-17-13 10:05	ferg	Status	new => closed
01-17-13 10:05	ferg	Resolution	open => fixed
01-17-13 10:05	ferg	Fixed in Version	=> 4.0.34

Mantis 1.0.0rc3[^]

27 total queries executed.
25 unique queries executed.