0005457: ArrayIndexOutOfBoundsException in BlockStore after Database corruption

Viewing Issue Advanced Details [ Jump to Notes ]

[ View Simple ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0005457

[Resin]

minor

always

06-14-13 06:00

07-23-13 11:31

Reporter

cowan

View Status

public

Assigned To

ferg

Priority

normal

Resolution

fixed

Platform

Status

closed

Projection

none

OS Version

ETA

none

Fixed in Version

4.0.37

Product Version

4.0.36

Product Build

Summary

0005457: ArrayIndexOutOfBoundsException in BlockStore after Database corruption

Description

[13-06-09 06:12:54.824] {resin-148} java.lang.ArrayIndexOutOfBoundsException: 102708
                           at com.caucho.db.block.BlockStore.getAllocation(BlockStore.java:927)
                           at com.caucho.db.block.BlockStore.getAllocationByAddress(BlockStore.java:917)
                           at com.caucho.db.block.BlockStore.isRowBlock(BlockStore.java:620)
                           at com.caucho.db.table.TableIterator.isValidRow(TableIterator.java:294)
                           at com.caucho.db.sql.IdentityIndexExpr.initRow(IdentityIndexExpr.java:102)
                           at com.caucho.db.sql.Query$TailInitRow.initBlockRow(Query.java:952)
                           at com.caucho.db.sql.Query.start(Query.java:727)
                           at com.caucho.db.sql.SelectQuery.execute(SelectQuery.java:209)
                           at com.caucho.db.sql.SelectQuery.execute(SelectQuery.java:171)
                           at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:357)
                           at com.caucho.db.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:325)
                           at com.caucho.server.distcache.DataStore.loadBlob(DataStore.java:289)
                           at com.caucho.server.distcache.DataStore.load(DataStore.java:241)
                           at com.caucho.server.distcache.CacheDataBackingImpl.loadData(CacheDataBackingImpl.java:259)
                           at com.caucho.server.distcache.LocalDataManager.readData(LocalDataManager.java:172)
                           at com.caucho.server.distcache.DistCacheEntry.get(DistCacheEntry.java:905)
                           at com.caucho.server.distcache.DistCacheEntry.get(DistCacheEntry.java:163)
                           at com.caucho.server.distcache.CacheImpl.get(CacheImpl.java:312)
                           at com.caucho.cloud.globalcache.GlobalCacheManager.get(GlobalCacheManager.java:170)
                           at com.caucho.cloud.globalcache.AbstractGlobalCache.get(AbstractGlobalCache.java:97)
                           at com.caucho.cloud.elastic.ScalingManager$PeerPod.getScalingPod(ScalingManager.java:398)
                           at com.caucho.cloud.elastic.ScalingManager.updatePod(ScalingManager.java:248)
                           at com.caucho.cloud.elastic.ScalingManager$ScalingCacheListener.onCacheChange(ScalingManager.java:411)
                           at com.caucho.cloud.globalcache.GlobalCacheManager.notifyListeners(GlobalCacheManager.java:249)
                           at com.caucho.cloud.globalcache.GlobalCacheManager.access$000(GlobalCacheManager.java:61)
                           at com.caucho.cloud.globalcache.GlobalCacheManager$ClusterCacheListener.entryUpdated(GlobalCacheManager.java:398)
                           at com.caucho.server.distcache.CacheImpl$UpdatedListener.entryUpdated(CacheImpl.java:1323)
                           at com.caucho.server.distcache.CacheImpl.entryUpdate(CacheImpl.java:1122)
                           at com.caucho.server.distcache.CacheImpl.access$200(CacheImpl.java:85)
                           at com.caucho.server.distcache.CacheImpl$1.run(CacheImpl.java:1105)
                           at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
                           at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

[13-06-09 06:12:54.825] {global-cache@baa.app-tier.admin.resin-139} Missing or corrupted data in get for MnodeEntry[value=668bf04ad8a4a104,flags=0x6,version=1370780541908,lease=-1] DistCacheEntry[key=resin:scaling:services-tier:main,keyHash=496c390b,owner=A_C]
                        {resin-148} Missing or corrupted data in get for MnodeEntry[value=668bf04ad8a4a104,flags=0x6,version=1370780541908,lease=-1] DistCacheEntry[key=resin:scaling:services-tier:main,keyHash=496c390b,owner=A_C]
[13-06-09 06:12:54.825] {resin-143} Missing or corrupted data in get for MnodeEntry[value=668bf04ad8a4a104,flags=0x6,version=1370780541908,lease=-1] DistCacheEntry[key=resin:scaling:services-tier:main,keyHash=496c390b,owner=A_C]
[13-06-09 06:12:54.826] {global-cache@baa.app-tier.admin.resin-140} Missing or corrupted data in get for MnodeEntry[value=668bf04ad8a4a104,flags=0x6,version=1370780541908,lease=-1] DistCacheEntry[key=resin:scaling:services-tier:main,keyHash=496c390b,owner=A_C]
[13-06-09 06:13:03.835] {main} Table[log_name:6] validating indexes due to unclean shutdown.
[13-06-09 06:13:03.836] {main} java.lang.IllegalStateException: LongColumn[id,index]: invalid index (key=89, index value=0, row addr=8f85, Table[log_name:6])
                           at com.caucho.db.table.Column.validateIndex(Column.java:600)
                           at com.caucho.db.table.Table.validateIndexByRow(Table.java:779)
                           at com.caucho.db.table.Table.validateIndexes(Table.java:743)
                           at com.caucho.db.table.Table.validateIndexesSafe(Table.java:707)
                           at com.caucho.db.table.Table.loadFromFile(Table.java:338)
                           at com.caucho.db.Database.getTable(Database.java:214)
                           at com.caucho.db.sql.Parser.parseFromItem(Parser.java:530)
                           at com.caucho.db.sql.Parser.parseFromItems(Parser.java:373)
                           at com.caucho.db.sql.Parser.parseSelect(Parser.java:249)
                           at com.caucho.db.sql.Parser.parseSelect(Parser.java:213)
                           at com.caucho.db.sql.Parser.parse(Parser.java:178)
                           at com.caucho.db.sql.Parser.parse(Parser.java:151)
                           at com.caucho.db.Database.parseQuery(Database.java:314)
                           at com.caucho.db.jdbc.StatementImpl.executeQuery(StatementImpl.java:108)
                           at com.caucho.env.log.NameDatabase.createDatabases(NameDatabase.java:211)
                           at com.caucho.env.log.NameDatabase.<init>(NameDatabase.java:55)
                           at com.caucho.env.log.ProLogSystem.start(ProLogSystem.java:397)
                           at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:544)
                           at com.caucho.env.service.ResinSystem.start(ResinSystem.java:511)
                           at com.caucho.server.resin.Resin.start(Resin.java:849)
                           at com.caucho.server.resin.Resin.initMain(Resin.java:981)
                           at com.caucho.server.resin.Resin.main(Resin.java:1437)

[13-06-09 06:13:03.836] {main} rebuilding indexes for 'Table[log_name:6]' because they did not properly validate on startup.

[13-06-09 06:12:52.767] {main} java.lang.IllegalStateException: BinaryColumn[id,index]: invalid index (key=BinaryKeyCompare[da57dac7e0405ea873f92cfd1473069d7882cdeb0442c8f2594af2310df0b377], index value=0, row addr=9a696e, Table[mnode:4])
                           at com.caucho.db.table.Column.validateIndex(Column.java:600)
                           at com.caucho.db.table.Table.validateIndexByRow(Table.java:779)
                           at com.caucho.db.table.Table.validateIndexes(Table.java:743)
                           at com.caucho.db.table.Table.validateIndexesSafe(Table.java:707)
                           at com.caucho.db.table.Table.loadFromFile(Table.java:338)
                           at com.caucho.db.Database.getTable(Database.java:214)
                           at com.caucho.db.sql.Parser.parseFromItem(Parser.java:530)
                           at com.caucho.db.sql.Parser.parseFromItems(Parser.java:373)
                           at com.caucho.db.sql.Parser.parseSelect(Parser.java:249)
                           at com.caucho.db.sql.Parser.parseSelect(Parser.java:213)
                           at com.caucho.db.sql.Parser.parse(Parser.java:178)
                           at com.caucho.db.sql.Parser.parse(Parser.java:151)
                           at com.caucho.db.Database.parseQuery(Database.java:314)
                           at com.caucho.db.jdbc.StatementImpl.executeQuery(StatementImpl.java:108)
                           at com.caucho.server.distcache.MnodeStore.initDatabase(MnodeStore.java:267)
                           at com.caucho.server.distcache.MnodeStore.init(MnodeStore.java:237)
                           at com.caucho.server.distcache.CacheDataBackingImpl.start(CacheDataBackingImpl.java:380)
                           at com.caucho.server.distcache.CacheStoreManager.start(CacheStoreManager.java:364)
                           at com.caucho.server.distcache.DistCacheSystem.start(DistCacheSystem.java:162)
                           at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:544)
                           at com.caucho.env.service.ResinSystem.start(ResinSystem.java:511)
                           at com.caucho.server.resin.Resin.start(Resin.java:849)
                           at com.caucho.server.resin.Resin.initMain(Resin.java:981)
                           at com.caucho.server.resin.Resin.main(Resin.java:1437)

[13-06-09 06:12:52.768] {main} rebuilding indexes for 'Table[mnode:4]' because they did not properly validate on startup.
[13-06-09 06:12:54.351] {main} java.sql.SQLException: table data is not a Resin DB. Version '7'
[13-06-09 06:12:54.561] {main} removing /var/resin/resin-data/site/distcache/data.db due to restart corruption
java.lang.ArrayIndexOutOfBoundsException: 124774
    at com.caucho.db.block.BlockStore.getAllocation(BlockStore.java:927)
    at com.caucho.db.block.BlockStore.getAllocationByAddress(BlockStore.java:917)
    at com.caucho.db.block.BlockStore.isRowBlock(BlockStore.java:620)
    at com.caucho.db.table.TableIterator.isValidRow(TableIterator.java:294)
    at com.caucho.db.sql.IdentityIndexExpr.initRow(IdentityIndexExpr.java:102)
    at com.caucho.db.sql.LeftOuterIndexExpr.initRow(LeftOuterIndexExpr.java:97)
    at com.caucho.db.sql.Query$TailInitRow.initBlockRow(Query.java:952)
    at com.caucho.db.sql.Query$NonTailInitRow.initBlockRow(Query.java:989)
    at com.caucho.db.sql.Query.start(Query.java:727)
    at com.caucho.db.sql.SelectQuery.execute(SelectQuery.java:209)
    at com.caucho.db.sql.SelectQuery.execute(SelectQuery.java:171)
    at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:357)
    at com.caucho.db.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:325)
    at com.caucho.server.distcache.DataStore.deleteOrphans(DataStore.java:539)
    at com.caucho.server.distcache.DataStore.access$100(DataStore.java:61)
    at com.caucho.server.distcache.DataStore$DeleteAlarm.handleAlarm(DataStore.java:690)
    at com.caucho.util.Alarm.handleAlarm(Alarm.java:523)
    at com.caucho.util.Alarm.run(Alarm.java:495)
    at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
    at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

Steps To Reproduce

Additional Information

Rep by E Larson

Attached Files

Relationships

There are no notes attached to this issue.

Issue History
Date Modified	Username	Field	Change
06-14-13 06:00	cowan	New Issue
07-23-13 11:31	ferg	Assigned To	=> ferg
07-23-13 11:31	ferg	Status	new => closed
07-23-13 11:31	ferg	Resolution	open => fixed
07-23-13 11:31	ferg	Fixed in Version	=> 4.0.37
10-16-14 21:33	bun_a	Issue Monitored: bun_a

Mantis 1.0.0rc3[^]

29 total queries executed.
26 unique queries executed.