0006097: Cluster error after multiple re-deploys of .war file

Viewing Issue Simple Details [ Jump to Notes ]

[ View Advanced ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0006097

[Resin]

minor

always

09-22-17 10:40

10-18-17 16:13

Reporter

wileysaw

View Status

public

Assigned To

ferg

Priority

normal

Resolution

unable to reproduce

Status

closed

Product Version

4.0.53

Summary

0006097: Cluster error after multiple re-deploys of .war file

Description

In this set up we had 3 vms, 2 actively serving web requests. The vm not serving the web requests in this instance received the error.

ClusterRepository[resin/repository/root] validate 9666afd7370059ded0a4d060a67b5741e20c2161 java.io.FileNotFoundException: /opt/resin/resin-data/app-2/.git/objects/96/66afd7370059ded0a4d060a67b5741e20c2161
[17-09-18 14:18:47.758] {resin-5377} java.io.FileNotFoundException: /opt/resin/resin-data/app-2/.git/objects/96/66afd7370059ded0a4d060a67b5741e20c2161
                           at com.caucho.vfs.JniFilePathImpl.openReadImpl(JniFilePathImpl.java:429)
                           at com.caucho.vfs.Path.openRead(Path.java:1156)
                           at com.caucho.env.git.GitSystem.openRawGitFile(GitSystem.java:630)
                           at com.caucho.env.repository.FileRepository.openRawGitFile(FileRepository.java:236)
                           at com.caucho.server.repository.ClusterRepository.openRawGitFile(ClusterRepository.java:599)
                           at com.caucho.env.repository.AbstractRepository.validateRawHash(AbstractRepository.java:641)
                           at com.caucho.env.repository.AbstractRepository.validateHash(AbstractRepository.java:593)
                           at com.caucho.env.repository.AbstractRepository.validateHash(AbstractRepository.java:614)
                           at com.caucho.env.repository.AbstractRepository.validateHash(AbstractRepository.java:614)
                           at com.caucho.env.repository.AbstractRepository.validateHash(AbstractRepository.java:608)
                           at com.caucho.env.repository.RepositoryTagMap.<init>(RepositoryTagMap.java:85)
                           at com.caucho.env.repository.AbstractRepository.updateTagMap(AbstractRepository.java:159)
                           at com.caucho.server.repository.ClusterRepository.access$100(ClusterRepository.java:60)
                           at com.caucho.server.repository.ClusterRepository$UpdateCallback.onSuccess(ClusterRepository.java:659)
                           at com.caucho.server.repository.RepositoryLoadTree.loadTreeSuccess(RepositoryLoadTree.java:120)
                           at com.caucho.server.repository.RepositoryLoadTree.access$200(RepositoryLoadTree.java:54)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:204)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.addHashChildren(RepositoryLoadTree.java:233)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:195)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.addHashChildren(RepositoryLoadTree.java:227)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:195)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.start(RepositoryLoadTree.java:157)
                           at com.caucho.server.repository.RepositoryLoadTree.loadTree(RepositoryLoadTree.java:93)
                           at com.caucho.server.repository.RepositoryActor.loadTree(RepositoryActor.java:94)
                           at com.caucho.server.repository.RepositoryManager.loadTree(RepositoryManager.java:144)
                           at com.caucho.server.repository.ClusterRepository.updateLoad(ClusterRepository.java:221)
                           at com.caucho.env.repository.AbstractRepository.update(AbstractRepository.java:136)
                           at com.caucho.server.repository.ClusterRepository.checkForUpdate(ClusterRepository.java:199)
                           at com.caucho.env.repository.AbstractRepository.checkForUpdate(AbstractRepository.java:105)
                           at com.caucho.server.repository.ClusterRepository$RepositoryCacheListener$1.run(ClusterRepository.java:726)
                           at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
                           at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

[17-09-18 14:18:47.758] {resin-5377} ClusterRepository[resin/repository/root] invalid 9666afd7370059ded0a4d060a67b5741e20c2161
[17-09-18 14:18:47.758] {resin-5377} ClusterRepository[resin/repository/root] invalid GitTree.Entry[ecompanies.war,100664,9666afd7370059ded0a4d060a67b5741e20c2161]
[17-09-18 14:18:47.758] {resin-5377} ClusterRepository[resin/repository/root] invalid GitTree.Entry[b2648e45a1ee2d00fed7f5b6c67757b48dd3131e,40000,b2648e45a1ee2d00fed7f5b6c67757b48dd3131e]
[17-09-18 14:18:47.758] {resin-5377} ClusterRepository[resin/repository/root] updating deployment RepositoryTagMap[seq=1505708327743,3c2829425608be56a6d2f2dab9b5d6899700ae1f]
[17-09-18 14:18:47.759] {resin-5377} java.io.IOException: Cannot create directory: /opt/resin/hosts/default
                           at com.caucho.vfs.FilePath.mkdirs(FilePath.java:532)
                           at com.caucho.server.host.HostWebAppDeployListener.update(HostWebAppDeployListener.java:89)
                           at com.caucho.server.host.HostWebAppDeployListener.onTagChange(HostWebAppDeployListener.java:99)
                           at com.caucho.env.repository.AbstractRepository.onTagChange(AbstractRepository.java:519)
                           at com.caucho.env.repository.AbstractRepository.onTagChange(AbstractRepository.java:508)
                           at com.caucho.env.repository.AbstractRepository.onTagChange(AbstractRepository.java:508)
                           at com.caucho.env.repository.AbstractRepository.notifyTagListeners(AbstractRepository.java:456)
                           at com.caucho.env.repository.AbstractRepository.setTagMap(AbstractRepository.java:433)
                           at com.caucho.env.repository.AbstractRepository.updateTagMap(AbstractRepository.java:162)
                           at com.caucho.server.repository.ClusterRepository.access$100(ClusterRepository.java:60)
                           at com.caucho.server.repository.ClusterRepository$UpdateCallback.onSuccess(ClusterRepository.java:659)
                           at com.caucho.server.repository.RepositoryLoadTree.loadTreeSuccess(RepositoryLoadTree.java:120)
                           at com.caucho.server.repository.RepositoryLoadTree.access$200(RepositoryLoadTree.java:54)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:204)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.addHashChildren(RepositoryLoadTree.java:233)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:195)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.addHashChildren(RepositoryLoadTree.java:227)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:195)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.start(RepositoryLoadTree.java:157)
                           at com.caucho.server.repository.RepositoryLoadTree.loadTree(RepositoryLoadTree.java:93)
                           at com.caucho.server.repository.RepositoryActor.loadTree(RepositoryActor.java:94)
                           at com.caucho.server.repository.RepositoryManager.loadTree(RepositoryManager.java:144)
                           at com.caucho.server.repository.ClusterRepository.updateLoad(ClusterRepository.java:221)
                           at com.caucho.env.repository.AbstractRepository.update(AbstractRepository.java:136)
                           at com.caucho.server.repository.ClusterRepository.checkForUpdate(ClusterRepository.java:199)
                           at com.caucho.env.repository.AbstractRepository.checkForUpdate(AbstractRepository.java:105)
                           at com.caucho.server.repository.ClusterRepository$RepositoryCacheListener$1.run(ClusterRepository.java:726)
                           at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
                           at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

Additional Information

Similar issue reported had a root cause of permissions error. Asked that subdirectories under resin-data have their permissions verified to be owned by resin. -- Pending update --

Attached Files

resin.log [^] (432,795 bytes) 09-22-17 10:40

Relationships

Notes
(0006800) ferg 10-18-17 16:13	Because of the second exception from the FilePath.mkdirs, there's either a permissions problem or the disk is full. Since the mkdirs failed, the git replication would have also failed for the same reason.

Issue History
Date Modified	Username	Field	Change
09-22-17 10:40	wileysaw	New Issue
09-22-17 10:40	wileysaw	File Added: resin.log
10-18-17 16:13	ferg	Note Added: 0006800
10-18-17 16:13	ferg	Assigned To	=> ferg
10-18-17 16:13	ferg	Status	new => closed
10-18-17 16:13	ferg	Resolution	open => unable to reproduce

Mantis 1.0.0rc3[^]

30 total queries executed.
26 unique queries executed.