Bug 1356065

Summary:	data loss after rhgs node reboot
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Neha <nerawat>
Component:	heketi	Assignee:	Humble Chirammal <hchiramm>
Status:	CLOSED NOTABUG	QA Contact:	Anoop <annair>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	rhgs-3.1	CC:	amainkar, annair, hchiramm, lpabon, mliyazud, nerawat, rcyriac, sashinde
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-07-15 07:16:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Neha 2016-07-13 10:58:59 UTC

Rebooted two of the rhgs node, now dont see heketi.db file under heketidb volume

due to that heketi container is crashing.


docker logs -f 87fcbdb9da33
Heketi 2.0.2
[heketi] INFO 2016/07/13 06:43:13 Loaded kubernetes executor
[heketi] ERROR 2016/07/13 06:43:13 /src/github.com/heketi/heketi/apps/glusterfs/app.go:149: write /var/lib/heketi/heketi.db: read-only file system
ERROR: Unable to start application


heketi container status : CrashLoopBackOff

Comment 3 Luis Pabón 2016-07-13 12:47:26 UTC

Humble, could you please investigate why a volume would go Read Only?

Comment 4 Humble Chirammal 2016-07-14 18:49:16 UTC

(In reply to Luis Pabón from comment #3)
> Humble, could you please investigate why a volume would go Read Only?

Luis, below error message clearly says 'The FS is READONLY' which is different from 'No Such File or Directory' reported here in the bugzilla. Can you confirm in this error scenario,  the path exist, but its READONLY causing this issue? 


[heketi] INFO 2016/07/13 06:43:13 Loaded kubernetes executor
[heketi] ERROR 2016/07/13 06:43:13 /src/github.com/heketi/heketi/apps/glusterfs/app.go:149: write /var/lib/heketi/heketi.db: read-only file system
ERROR: Unable to start application

Comment 5 Humble Chirammal 2016-07-14 18:52:17 UTC

@Neha, once again FS goes readonly in this scenario is expected as mentioned here https://bugzilla.redhat.com/show_bug.cgi?id=1355689. 

I dont know why you say the DB file *does not* exist eventhough there is a clear error message says FS is READONLY. I am confused with the bug report and the error message captured here in this bz as it does not match.

Comment 8 Neha 2016-07-15 05:26:42 UTC

Regarding read only volume , it might be because few of the brick process are not up and not because of data loss.

Comment 9 Neha 2016-07-15 07:16:14 UTC

closing this bug as heketi issue is being tracked in BZ 1355689. For data loss I will create a new BZ to avoid confusion.