Bug 1026471

Summary: Cassandra - Duplicate hard link, cannot clear
Product: [Other] RHQ Project Reporter: Elias Ross <genman>
Component: Core ServerAssignee: John Sanda <jsanda>
Status: CLOSED NOTABUG QA Contact: Mike Foley <mfoley>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.9CC: hrupp, jsanda
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-09 21:11:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Elias Ross 2013-11-04 17:50:42 UTC
Description of problem:

My Cassandra is unable to perform maintenance. It may be a Cassandra bug.

The error I see it:

17:43:01,355 ERROR [org.rhq.enterprise.server.operation.OperationServerServiceImpl] (http-/0.0.0.0:7080-73) The failed operation [rhq-resource-427988--1783761
045-1383586979530_=_rhq-resource-427988_=_1383586979598] had an error of: org.mc4j.ems.connection.EmsInvocationException: Exception on invocation of [takeSnap
shot]javax.management.RuntimeMBeanException: java.lang.RuntimeException: Tried to create duplicate hard link to /data06/rhq/data/system/NodeIdInfo/snapshots/1
383586980828/system-NodeIdInfo-ic-1-Digest.sha1
...
Caused by: java.lang.RuntimeException: Tried to create duplicate hard link to /data06/rhq/data/system/NodeIdInfo/snapshots/1383586980828/system-NodeIdInfo-ic-1-Digest.sha1
        at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:70)
        at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1081)
        at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1567)
        at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1612)
        at org.apache.cassandra.db.Table.snapshot(Table.java:194)
        at org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2203)


Version-Release number of selected component (if applicable): 4.9


How reproducible: Unclear. My installation may be corrupted in some way.


Additional info: Clearing the existing snapshots doesn't seem to fix the issue. E.g.:

./nodetool -p 7299 clearsnapshot

doesn't fix the problem, even on all 3 nodes.

Comment 1 Elias Ross 2014-01-09 21:11:54 UTC
Not an issue. It seems due to corruption in my side.