Mantis - Resin
Viewing Issue Advanced Details
4657 minor always 07-07-11 08:57 07-12-11 09:39
ferg  
ferg  
normal  
closed  
fixed  
none    
none 4.0.20  
0004657: heartbeat/cache sync issues
(rep by William Kong)


I've asked my team to test out the 4.0.19 resin system to see if the issue is occurring again. It seems like it is. The test systems we are using to test 4.0.19 has very low traffic. The following is the error message:


[11-07-06 14:33:21.932] {resin-149} ClusterCacheManagerImpl[estmt2] cannot load data for HashKey[a7200497] from triad
[11-07-06 14:33:21.932] {resin-149} Missing or corrupted data in get for MnodeValue[value=HashKey[a7200497],flags=0x6,version=1309987974522,lease=-1] ProCacheEntry[key=null,keyHash=19444d00,owner=C_A]
[11-07-06 15:28:21.222] {resin-250} com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@3d2ca4ef query timeout: 60000
                                     to: global-cache@baa.estmt.admin.resin
                                     query: GlobalCacheHeartbeat[de3cf789,94e37997,1309987957386]
                                           at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:78)
                                           at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:189)
                                           at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:675)
                                           at com.caucho.cloud.globalcache.GlobalCacheActor.sendHeartbeat(GlobalCacheActor.java:223)
                                           at com.caucho.cloud.globalcache.GlobalCacheManager.sendHeartbeatImpl(GlobalCacheManager.java:143)
                                           at com.caucho.cloud.globalcache.GlobalCacheManager$HeartbeatTask.run(GlobalCacheManager.java:199)
                                           at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)
                                           at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)

Notes
(0005374)
ferg   
07-11-11 11:01   
Also reported by Orlando Geronimo:

[11-07-11 12:56:39.642] {resin-24} com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@54ede19e query timeout: 60000

                                     to: global-cache@aaa.web-tier.admin.resin

                                     query: GlobalCacheHeartbeat[2ff0dbf6,c0b9aa4b,1310399740417]

                                        at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:78)

                                        at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:189)

                                        at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:675)

                                        at com.caucho.cloud.globalcache.GlobalCacheActor.sendHeartbeat(GlobalCacheActor.java:223)

                                        at com.caucho.cloud.globalcache.GlobalCacheManager.sendHeartbeatImpl(GlobalCacheManager.java:143)

                                        at com.caucho.cloud.globalcache.GlobalCacheManager$HeartbeatTask.run(GlobalCacheManager.java:199)

                                        at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)

                                        at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)

[11-07-11 12:58:39.649] {resin-111} com.caucho.bam.TimeoutException: com.caucho.cloud.bam.TriadFuture@1958aeed query timeout: 60000

                                      to: global-cache@aaa.web-tier.admin.resin

                                      query: GlobalCacheHeartbeat[2ff0dbf6,c0b9aa4b,1310399740417]

                                        at com.caucho.cloud.bam.TriadFuture.get(TriadFuture.java:78)

                                        at com.caucho.cloud.bam.TriadFirstQuery.get(TriadFirstQuery.java:189)

                                        at com.caucho.cloud.bam.BamTriadSender.queryTriadFirstRemote(BamTriadSender.java:675)

                                        at com.caucho.cloud.globalcache.GlobalCacheActor.sendHeartbeat(GlobalCacheActor.java:223)

                                        at com.caucho.cloud.globalcache.GlobalCacheManager.sendHeartbeatImpl(GlobalCacheManager.java:143)

                                        at com.caucho.cloud.globalcache.GlobalCacheManager$HeartbeatTask.run(GlobalCacheManager.java:199)

                                        at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)

                                        at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)
(0005375)
ferg   
07-12-11 09:39   
cloud/0480