0006097: Cluster error after multiple re-deploys of .war file

Viewing Issue Advanced Details [ Jump to Notes ]

[ View Simple ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0006097

[Resin]

minor

always

09-22-17 10:40

10-18-17 16:13

Reporter

wileysaw

View Status

public

Assigned To

ferg

Priority

normal

Resolution

unable to reproduce

Platform

Status

closed

Projection

none

OS Version

ETA

none

Fixed in Version

Product Version

4.0.53

Product Build

Summary

0006097: Cluster error after multiple re-deploys of .war file

Description

In this set up we had 3 vms, 2 actively serving web requests. The vm not serving the web requests in this instance received the error.

ClusterRepository[resin/repository/root] validate 9666afd7370059ded0a4d060a67b5741e20c2161 java.io.FileNotFoundException: /opt/resin/resin-data/app-2/.git/objects/96/66afd7370059ded0a4d060a67b5741e20c2161
[17-09-18 14:18:47.758] {resin-5377} java.io.FileNotFoundException: /opt/resin/resin-data/app-2/.git/objects/96/66afd7370059ded0a4d060a67b5741e20c2161
                           at com.caucho.vfs.JniFilePathImpl.openReadImpl(JniFilePathImpl.java:429)
                           at com.caucho.vfs.Path.openRead(Path.java:1156)
                           at com.caucho.env.git.GitSystem.openRawGitFile(GitSystem.java:630)
                           at com.caucho.env.repository.FileRepository.openRawGitFile(FileRepository.java:236)
                           at com.caucho.server.repository.ClusterRepository.openRawGitFile(ClusterRepository.java:599)
                           at com.caucho.env.repository.AbstractRepository.validateRawHash(AbstractRepository.java:641)
                           at com.caucho.env.repository.AbstractRepository.validateHash(AbstractRepository.java:593)
                           at com.caucho.env.repository.AbstractRepository.validateHash(AbstractRepository.java:614)
                           at com.caucho.env.repository.AbstractRepository.validateHash(AbstractRepository.java:614)
                           at com.caucho.env.repository.AbstractRepository.validateHash(AbstractRepository.java:608)
                           at com.caucho.env.repository.RepositoryTagMap.<init>(RepositoryTagMap.java:85)
                           at com.caucho.env.repository.AbstractRepository.updateTagMap(AbstractRepository.java:159)
                           at com.caucho.server.repository.ClusterRepository.access$100(ClusterRepository.java:60)
                           at com.caucho.server.repository.ClusterRepository$UpdateCallback.onSuccess(ClusterRepository.java:659)
                           at com.caucho.server.repository.RepositoryLoadTree.loadTreeSuccess(RepositoryLoadTree.java:120)
                           at com.caucho.server.repository.RepositoryLoadTree.access$200(RepositoryLoadTree.java:54)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:204)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.addHashChildren(RepositoryLoadTree.java:233)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:195)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.addHashChildren(RepositoryLoadTree.java:227)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:195)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.start(RepositoryLoadTree.java:157)
                           at com.caucho.server.repository.RepositoryLoadTree.loadTree(RepositoryLoadTree.java:93)
                           at com.caucho.server.repository.RepositoryActor.loadTree(RepositoryActor.java:94)
                           at com.caucho.server.repository.RepositoryManager.loadTree(RepositoryManager.java:144)
                           at com.caucho.server.repository.ClusterRepository.updateLoad(ClusterRepository.java:221)
                           at com.caucho.env.repository.AbstractRepository.update(AbstractRepository.java:136)
                           at com.caucho.server.repository.ClusterRepository.checkForUpdate(ClusterRepository.java:199)
                           at com.caucho.env.repository.AbstractRepository.checkForUpdate(AbstractRepository.java:105)
                           at com.caucho.server.repository.ClusterRepository$RepositoryCacheListener$1.run(ClusterRepository.java:726)
                           at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
                           at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

[17-09-18 14:18:47.758] {resin-5377} ClusterRepository[resin/repository/root] invalid 9666afd7370059ded0a4d060a67b5741e20c2161
[17-09-18 14:18:47.758] {resin-5377} ClusterRepository[resin/repository/root] invalid GitTree.Entry[ecompanies.war,100664,9666afd7370059ded0a4d060a67b5741e20c2161]
[17-09-18 14:18:47.758] {resin-5377} ClusterRepository[resin/repository/root] invalid GitTree.Entry[b2648e45a1ee2d00fed7f5b6c67757b48dd3131e,40000,b2648e45a1ee2d00fed7f5b6c67757b48dd3131e]
[17-09-18 14:18:47.758] {resin-5377} ClusterRepository[resin/repository/root] updating deployment RepositoryTagMap[seq=1505708327743,3c2829425608be56a6d2f2dab9b5d6899700ae1f]
[17-09-18 14:18:47.759] {resin-5377} java.io.IOException: Cannot create directory: /opt/resin/hosts/default
                           at com.caucho.vfs.FilePath.mkdirs(FilePath.java:532)
                           at com.caucho.server.host.HostWebAppDeployListener.update(HostWebAppDeployListener.java:89)
                           at com.caucho.server.host.HostWebAppDeployListener.onTagChange(HostWebAppDeployListener.java:99)
                           at com.caucho.env.repository.AbstractRepository.onTagChange(AbstractRepository.java:519)
                           at com.caucho.env.repository.AbstractRepository.onTagChange(AbstractRepository.java:508)
                           at com.caucho.env.repository.AbstractRepository.onTagChange(AbstractRepository.java:508)
                           at com.caucho.env.repository.AbstractRepository.notifyTagListeners(AbstractRepository.java:456)
                           at com.caucho.env.repository.AbstractRepository.setTagMap(AbstractRepository.java:433)
                           at com.caucho.env.repository.AbstractRepository.updateTagMap(AbstractRepository.java:162)
                           at com.caucho.server.repository.ClusterRepository.access$100(ClusterRepository.java:60)
                           at com.caucho.server.repository.ClusterRepository$UpdateCallback.onSuccess(ClusterRepository.java:659)
                           at com.caucho.server.repository.RepositoryLoadTree.loadTreeSuccess(RepositoryLoadTree.java:120)
                           at com.caucho.server.repository.RepositoryLoadTree.access$200(RepositoryLoadTree.java:54)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:204)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.addHashChildren(RepositoryLoadTree.java:233)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:195)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.addHashChildren(RepositoryLoadTree.java:227)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.completeLoad(RepositoryLoadTree.java:195)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.loadFile(RepositoryLoadTree.java:257)
                           at com.caucho.server.repository.RepositoryLoadTree$LoadTree.start(RepositoryLoadTree.java:157)
                           at com.caucho.server.repository.RepositoryLoadTree.loadTree(RepositoryLoadTree.java:93)
                           at com.caucho.server.repository.RepositoryActor.loadTree(RepositoryActor.java:94)
                           at com.caucho.server.repository.RepositoryManager.loadTree(RepositoryManager.java:144)
                           at com.caucho.server.repository.ClusterRepository.updateLoad(ClusterRepository.java:221)
                           at com.caucho.env.repository.AbstractRepository.update(AbstractRepository.java:136)
                           at com.caucho.server.repository.ClusterRepository.checkForUpdate(ClusterRepository.java:199)
                           at com.caucho.env.repository.AbstractRepository.checkForUpdate(AbstractRepository.java:105)
                           at com.caucho.server.repository.ClusterRepository$RepositoryCacheListener$1.run(ClusterRepository.java:726)
                           at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
                           at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

Steps To Reproduce

Additional Information

Similar issue reported had a root cause of permissions error. Asked that subdirectories under resin-data have their permissions verified to be owned by resin. -- Pending update --

Attached Files

resin.log [^] (432,795 bytes) 09-22-17 10:40

Relationships

Notes
(0006800) ferg 10-18-17 16:13	Because of the second exception from the FilePath.mkdirs, there's either a permissions problem or the disk is full. Since the mkdirs failed, the git replication would have also failed for the same reason.

Issue History
Date Modified	Username	Field	Change
09-22-17 10:40	wileysaw	New Issue
09-22-17 10:40	wileysaw	File Added: resin.log
10-18-17 16:13	ferg	Note Added: 0006800
10-18-17 16:13	ferg	Assigned To	=> ferg
10-18-17 16:13	ferg	Status	new => closed
10-18-17 16:13	ferg	Resolution	open => unable to reproduce

Mantis 1.0.0rc3[^]

30 total queries executed.
26 unique queries executed.