After upgrading Gluster storage nodes from glusterfs-3.8.10-1.el7 to glusterfs-3.8.11-1.el7 (this is the community build) followed by a system reboot. All storage nodes were updated and rebooted without waiting for Gluster to recover. The latter is a failure in our procedure and to be adressed separately. Shortly after the reboot pods started to report failure events:

    mkdir /var/lib/origin/openshift.local.volumes/pods/…/volumes/kubernetes.io~glusterfs/gluster-…: file exists

Only a pod restart brought them back into working order.

The logs written by the Gluster client started for the affected pods by OpenShift/K8s, /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/gluster-*/*.log, contained a single line:

    E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: 172.16.110.… (No data available)

Digging this we see Kubernetes uses a single endpoint IP address[1] when mounting a volume. The "backup-volfile-server" option was only added in commit 43c0a686, to be shipped with Kubernetes 1.7 . This effectively makes every Gluster server a single point of failure for pods using it as a volume server. Should that server fail permanently or become unreachable for longer periods of time services running in pods will be impacted.

Gluster client option "backup-volfile-server" should be implemented from [2] and backported to 3.4/3.5

[1] https://github.com/kubernetes/kubernetes/blob/v1.6.2/pkg/volume/glusterfs/glusterfs.go#L336
[2] https://github.com/kubernetes/kubernetes/commit/43c0a6869d35eaa8f762ee1e223f3e14c507120c