Description of problem:

 When the heketi pod is killed, and a new one is started, heketi.db in the new pod is in read only mode

Version-Release number of selected component (if applicable):

  rhgs3/rhgs-server-rhel7:3.2.0-7
  rhgs3/rhgs-volmanager-rhel7:3.2.0-11

How reproducible:

 Always

Steps to Reproduce:

1.  Kill the current heketi pod:

# oc delete pod heketi-1-9x3s5 
pod "heketi-1-9x3s5" deleted

# oc get pods 
NAME                                  READY     STATUS              RESTARTS   AGE
glusterblock-provisioner-dc-1-4n2gl   1/1       Running             0          4m
glusterfs-6h4v9                       1/1       Running             0          16m
glusterfs-kh6n3                       1/1       Running             0          11m
glusterfs-m3r91                       1/1       Running             0          8m
heketi-1-9x3s5                        1/1       Terminating         0          21m
heketi-1-dhbbh                        0/1       ContainerCreating   0          2s


2. Wait for the new heketi pod to be ready

3. Try to create a new volume:

# heketi-cli volume create --size=1 --persistent-volume --persistent-volume-endpoint=test-storage-project > test.json 
Error: Error calling v.allocBricksInCluster: database is in read-only mode

Actual results:

  heketi.db is in read-only mode 

Expected results:

 The new pod is not started until the previous one is completely removed, and heketi.db is in read-write mode

Master Log:


Node Log (of failed PODs):


PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

The issue doesn't happen with the latest CNS version:

  rhgs3/rhgs-server-rhel7:3.3.0-362
  rhgs3/rhgs-volmanager-rhel7:3.3.0-362


As a workaround the heketi replication controller can be scaled down to 0, and scaled up to 1,
and heketi.db will be in read-write mode