Bug 1500955

Summary:	heketi.db in read-only when heketi pod is rescheduled
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Alexis Solanas <asolanas>
Component:	kubernetes	Assignee:	Humble Chirammal <hchiramm>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Prasanth <pprakash>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.0	CC:	aos-bugs, aos-storage-staff, asolanas, hchiramm, pprakash, rhs-bugs
Target Milestone:	---	Flags:	asolanas: needinfo-
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-11-15 13:35:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alexis Solanas 2017-10-11 20:24:49 UTC

Description of problem:

 When the heketi pod is killed, and a new one is started, heketi.db in the new pod is in read only mode

Version-Release number of selected component (if applicable):

  rhgs3/rhgs-server-rhel7:3.2.0-7
  rhgs3/rhgs-volmanager-rhel7:3.2.0-11

How reproducible:

 Always

Steps to Reproduce:

1.  Kill the current heketi pod:

# oc delete pod heketi-1-9x3s5 
pod "heketi-1-9x3s5" deleted

# oc get pods 
NAME                                  READY     STATUS              RESTARTS   AGE
glusterblock-provisioner-dc-1-4n2gl   1/1       Running             0          4m
glusterfs-6h4v9                       1/1       Running             0          16m
glusterfs-kh6n3                       1/1       Running             0          11m
glusterfs-m3r91                       1/1       Running             0          8m
heketi-1-9x3s5                        1/1       Terminating         0          21m
heketi-1-dhbbh                        0/1       ContainerCreating   0          2s


2. Wait for the new heketi pod to be ready

3. Try to create a new volume:

# heketi-cli volume create --size=1 --persistent-volume --persistent-volume-endpoint=test-storage-project > test.json 
Error: Error calling v.allocBricksInCluster: database is in read-only mode

Actual results:

  heketi.db is in read-only mode 

Expected results:

 The new pod is not started until the previous one is completely removed, and heketi.db is in read-write mode

Master Log:


Node Log (of failed PODs):


PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

The issue doesn't happen with the latest CNS version:

  rhgs3/rhgs-server-rhel7:3.3.0-362
  rhgs3/rhgs-volmanager-rhel7:3.3.0-362


As a workaround the heketi replication controller can be scaled down to 0, and scaled up to 1,
and heketi.db will be in read-write mode

Comment 1 Prasanth 2017-10-12 17:02:13 UTC

Looks like it's a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1490980 which has been fixed and available in the latest CNS 3.6 bits.

For information on the advisory, and where to find the updated
files, follow the link below.

https://access.redhat.com/errata/RHEA-2017:2879

However, I would request Humble to check and confirm the same.