Description of problem:
On a 5 node gluster cluster, gluster pods were brought down on 2 nodes by removing 'glusterfs: storage-host'. Now effectively, there are 3 gluster pods up and running.
sh-4.2# heketi-cli volume create --size=20
Error: Unable to find a GlusterFS pod on host dhcp46-45.lab.eng.blr.redhat.com with a label key glusterfs-node
heketi volume create operation on such a system failed.
Version-Release number of selected component (if applicable):
rpm -qa | grep 'heketi'
2/2, this should definitely be reproducible
Steps to Reproduce:
1. create a 5 node cns setup
2. on 2 of the nodes, remove the label - glusterfs: storage-host
3. Try to create heketi volume
volume creation fails
heketi should pick up the nodes on which pods are up
heketi logs and topology info shall be attached
Created attachment 1419998 [details]
Created attachment 1419999 [details]
Two items I noticed looking through the logs:
1) The node health monitor thread has not been started. This is probably due to an "old" heketi config that lacks the parameter needed to enable this thread. With the monitor on the volume create operation will not try to use nodes it knows to be unavailable.
2) The volume create operation retried correctly, but must have never hit a combination of nodes where all nodes were up. We may need to tweak the number of retries performed to increase the chances of a working node selection.
But before we work on #2, we should retest with #1 working.
This issue was due to the node health monitoring not enabled. With rhgs-volmanager-container-3.3.1-8.3, this is enabled by default.
[heketi] INFO 2018/04/16 14:34:34 Loaded kubernetes executor
[heketi] ERROR 2018/04/16 14:34:34 /src/github.com/heketi/heketi/apps/glusterfs/app.go:100: invalid log level:
[heketi] INFO 2018/04/16 14:34:34 Block: Auto Create Block Hosting Volume set to true
[heketi] INFO 2018/04/16 14:34:34 Block: New Block Hosting Volume size 100 GB
[heketi] INFO 2018/04/16 14:34:34 GlusterFS Application Loaded
[heketi] INFO 2018/04/16 14:34:34 Started Node Health Cache Monitor
Listening on port 8080
Verified the bug in rhgs-volmanager-container-3.3.1-8.4.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.