0005336: Startup delay with multiple clusters: cannot open heartbeat connection to TriadServer: future timeout: 120000ms

Viewing Issue Advanced Details [ Jump to Notes ]

[ View Simple ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0005336

[Resin]

minor

always

01-17-13 07:13

01-17-13 10:05

Reporter

cowan

View Status

public

Assigned To

ferg

Priority

high

Resolution

fixed

Platform

Status

closed

Projection

none

OS Version

ETA

none

Fixed in Version

4.0.34

Product Version

4.0.33

Product Build

Summary

0005336: Startup delay with multiple clusters: cannot open heartbeat connection to TriadServer: future timeout: 120000ms

Description

Previously it took about one minute to start resin, and now it takes about five minutes. Interestingly, the startup time goes back down if I misconfigure it so the front-end server cannot find the back end server, for example by making the back end resin run on a different port to that declared in the cluster configuration for the front end. With finer logging enabled I can see errors like the following in the logs while it is starting up. If I change resin-embed to listen on a different port the startup time goes back down to one minute (although obviously the cluster is not available).

[2013-01-11 18:23:40.372] AllMethodCallback[3,AllMethodScoreboard[1,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.risk-dashboard.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:24:40.212] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.212] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[impersonation,0,127.0.0.1:6801]
[2013-01-11 18:24:40.213] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.213] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[administration,0,127.0.0.1:6802]
[2013-01-11 18:24:40.213] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:24:40.213] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[risk-dashboard,0,127.0.0.1:6803]
[2013-01-11 18:25:40.214] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.215] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[impersonation,0,127.0.0.1:6801]
[2013-01-11 18:25:40.215] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.215] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[administration,0,127.0.0.1:6802]
[2013-01-11 18:25:40.215] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.216] HeartbeatActor[cluster-heartbeat@aaa.app-tier.admin.resin] cannot open heartbeat connection to TriadServer[risk-dashboard,0,127.0.0.1:6803]
[2013-01-11 18:25:40.368] java.lang.IllegalStateException: com.caucho.bam.query.QueryFutureCallback@146ccf3e future timeout: 120000ms
                               at com.caucho.bam.query.QueryFutureCallback.get(QueryFutureCallback.java:108)
                               at com.caucho.bam.proxy.BamProxyHandler$QueryCall.invoke(BamProxyHandler.java:160)
                               at com.caucho.bam.proxy.BamProxyHandler.invoke(BamProxyHandler.java:117)
                               at $Proxy43.ping(Unknown Source)
                               at com.caucho.cloud.globalcache.GlobalCacheManager.sendPing(GlobalCacheManager.java:330)
                               at com.caucho.cloud.globalcache.GlobalCacheManager.start(GlobalCacheManager.java:134)
                               at com.caucho.cloud.globalcache.GlobalCacheSystem.start(GlobalCacheSystem.java:117)
                               at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:529)
                               at com.caucho.env.service.ResinSystem.start(ResinSystem.java:496)
                               at com.caucho.server.resin.Resin.start(Resin.java:849)
                               at com.caucho.server.resin.Resin.initMain(Resin.java:981)
                               at com.caucho.server.resin.Resin.main(Resin.java:1437)
[2013-01-11 18:25:40.370] ClientSocketFactory[default->administration,127.0.0.1:6802] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] ClientSocketFactory[default->impersonation,127.0.0.1:6801] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803] java.net.ConnectException: Connection refused
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->administration,127.0.0.1:6802]
                           to: global-cache@aaa.administration.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=14,to=global-cache@aaa.administration.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->impersonation,127.0.0.1:6801]
                           to: global-cache@aaa.impersonation.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=12,to=global-cache@aaa.impersonation.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] HmuxQueue[hmtp-aaa-to-aaa] cannot connect to ClientSocketFactory[default->risk-dashboard,127.0.0.1:6803]
                           to: global-cache@aaa.risk-dashboard.admin.resin from: cluster-router-app-tier-main@aaa.app-tier.admin.resin
                           Query[id=16,to=global-cache@aaa.risk-dashboard.admin.resin,from=cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.bam.proxy.CallPayload]
                           heartbeat: false
[2013-01-11 18:25:40.370] AllMethodCallback[1,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.impersonation.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:25:40.370] AllMethodCallback[2,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.administration.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]
[2013-01-11 18:25:40.371] AllMethodCallback[3,AllMethodScoreboard[10,cluster-router-app-tier-main@aaa.app-tier.admin.resin,SimpleActorSender[QueryActorFilter[SkeletonActorFilter[cluster-router-app-tier-main@aaa.app-tier.admin.resin,com.caucho.cloud.bam.ClusterRouteActor]]]]] BamError[type=cancel,group=remote-connection-failed,text=hmtp-aaa-to-aaa failed connection CallPayload[ping] {to:global-cache@aaa.risk-dashboard.admin.resin, from:cluster-router-app-tier-main@aaa.app-tier.admin.resin}]

Steps To Reproduce

Additional Information

Rep by D. Shearer

Attached Files

Relationships

There are no notes attached to this issue.

Issue History
Date Modified	Username	Field	Change
01-17-13 07:13	cowan	New Issue
01-17-13 10:05	ferg	Assigned To	=> ferg
01-17-13 10:05	ferg	Status	new => closed
01-17-13 10:05	ferg	Resolution	open => fixed
01-17-13 10:05	ferg	Fixed in Version	=> 4.0.34

Mantis 1.0.0rc3[^]

27 total queries executed.
25 unique queries executed.