Mantis - Resin
Viewing Issue Advanced Details
ID: Category: Severity: Reproducibility: Date Submitted: Last Update:
5177 minor always 08-15-12 16:37 09-07-12 18:38
Reporter: rickHigh Platform:  
Assigned To: ferg OS:  
Priority: normal OS Version:  
Status: closed Product Version: 4.0.30  
Product Build: Resolution: fixed  
Projection: none      
ETA: none Fixed in Version: 4.0.31  
Summary: 0005177: EC2 Elastic Join: "unable to join cluster app: No triad servers" dynamic/elastic join not working using Elastic IPs under EC2.
Description: Using external IPs/Elastic IPs, I am able to join three Triad servers running in Amazon EC2.

When an elastic server is spun up, it reports this error:

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800

Setup details are in additional information.
Steps To Reproduce:
Additional Information: 1) Created triad0 server. m1.small.

2) Install dev software on triad0 (internal instead of yum install resin for snapshots)

Steps:

$ sudo yum groupinstall "Development Tools"
$ sudo yum install java-1.6.0-openjdk-devel
$ sudo ln -s /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/include/ /usr/lib/jvm/jre/include
$ sudo yum install openssl-devel

Download tar

./configure make make install resin tar snapshot


3) Create three elastic IPs

4) Edit resin.properties.

Steps:
Create an admin_password and a system_key too.
$ resinctl generate-password admin somepass
admin_user : admin
admin_password : {SSHA}pass+uxce

Properties for triad0:

elastic_cloud_enable : true
home_cluster : app
home_server : app-0
app.https : 8443
admin_user : admin
admin_password : {SSHA}pass/XJCE
web_admin_enable : true
remote_cli_enable : true
web_admin_external : true
web_admin_ssl : true
app_servers : ext:23.21.106.227 ext:23.21.121.216 ext:23.21.195.83
cluster_system_key : changeme


Table:

triad0 = app-0 = ext:23.21.106.227
triad1 = app-1 = ext:23.21.121.216
triad2 = app-2 = ext:23.21.195.83

5) Install the license file

6) Clone triad0 then make triad1 and triad2 from the clone

7) Assign the triad servers the correct Elastic IPs

8) Modify home_server for triad1 and triad2 to app-1 and app-2 respectively.

9) Go to resin-admin of triad0 and see if it sees triad1 and triad2
https://23.21.106.227:8443/resin-admin/ [^]

10) Create another clone called elastic0

Add this
elastic_server : true

Remove home_server (comment it out)


11) Go to resin-admin of triad0 and see if it sees the elastic server

FAIL.
TURNED UP LOGGING, GOT THE ABOVE MESSAGE. (caps for emphasis not yelling)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800



Background:


I recall this feature working in 4.0.28 as I tested it whilst writing this guide:

http://wiki4.caucho.com/EC2:_Java_EE_Cloud_Deployment,_Clustering,_Session_Replication,_and_Setting_up_Amazon_Load_Balancer [^]

This feature has since seem to stop working. There has been some work in this area to simplify and clarify setup.
Relationships
Attached Files:

Notes
(0006033)
rickHigh   
08-31-12 15:50   
With minimum logging and remote_cli not enabled we get this:

Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 22:28:08 +0000 (UTC)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 22:28:41 +0000 (UTC)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 22:29:13 +0000 (UTC)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 22:29:45 +0000 (UTC
(0006034)
rickHigh   
08-31-12 16:28   
$ sudo resinctl start
Resin/4.0.30 launching watchdog at 127.0.0.1:6600
Resin/4.0.30 started -server 'dyn-10.207.55.13:6830' with watchdog at 127.0.0.1:6600
[ec2-user@domU-12-31-39-15-34-FF resin]$ cd /var/log/resin/
jvm-dyn-10.207.55.13_6830.log watchdog-manager.log
[ec2-user@domU-12-31-39-15-34-FF resin]$ cd /var/log/resin/jvm-dyn-10.207.55.13_6830.log
-bash: cd: /var/log/resin/jvm-dyn-10.207.55.13_6830.log: Not a directory
[ec2-user@domU-12-31-39-15-34-FF resin]$ cat /var/log/resin/jvm-dyn-10.207.55.13_6830.log
Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 23:03:06 +0000 (UTC)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 23:03:38 +0000 (UTC)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
[ec2-user@domU-12-31-39-15-34-FF resin]$ clear

[ec2-user@domU-12-31-39-15-34-FF resin]$ cat /var/log/resin/jvm-dyn-10.207.55.13_6830.log
Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 23:03:06 +0000 (UTC)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 23:03:38 +0000 (UTC)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
Resin Professional 4.0.30 (built Mon, 20 Aug 2012 10:01:17 PDT)
Copyright(c) 1998-2012 Caucho Technology. All rights reserved.

  1013792.license -- 1 Resin server Caucho

Starting Resin Professional on Fri, 31 Aug 2012 23:04:10 +0000 (UTC)

com.caucho.config.ConfigException: ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
    at com.caucho.server.resin.Resin.joinCluster(Resin.java:1112)
    at com.caucho.server.resin.Resin.initTopology(Resin.java:1047)
    at com.caucho.server.resin.Resin.initServletSystem(Resin.java:1241)
    at com.caucho.server.resin.Resin.configure(Resin.java:967)
    at com.caucho.server.resin.Resin.initMain(Resin.java:949)
    at com.caucho.server.resin.Resin.main(Resin.java:1403)
ProResinDelegate[] unable to join cluster app: No triad servers were reachable.
  Triad servers are ext:23.21.106.227:6800, ext:23.21.121.216:6800, ext:23.21.195.83:6800
(0006044)
ferg   
09-07-12 18:38   
cloud/12c9