After upgrading Gluster storage nodes from glusterfs-3.8.10-1.el7 to glusterfs-3.8.11-1.el7 (this is the community build) followed by a system reboot. All storage nodes were updated and rebooted without waiting for Gluster to recover. The latter is a failure in our procedure and to be adressed separately. Shortly after the reboot pods started to report failure events:
mkdir /var/lib/origin/openshift.local.volumes/pods/…/volumes/kubernetes.io~glusterfs/gluster-…: file exists
Only a pod restart brought them back into working order.
The logs written by the Gluster client started for the affected pods by OpenShift/K8s, /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/gluster-*/*.log, contained a single line:
E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: 172.16.110.… (No data available)
Digging this we see Kubernetes uses a single endpoint IP address[1] when mounting a volume. The "backup-volfile-server" option was only added in commit 43c0a686, to be shipped with Kubernetes 1.7 . This effectively makes every Gluster server a single point of failure for pods using it as a volume server. Should that server fail permanently or become unreachable for longer periods of time services running in pods will be impacted.
Gluster client option "backup-volfile-server" should be implemented from [2] and backported to 3.4/3.5
[1] https://github.com/kubernetes/kubernetes/blob/v1.6.2/pkg/volume/glusterfs/glusterfs.go#L336
[2] https://github.com/kubernetes/kubernetes/commit/43c0a6869d35eaa8f762ee1e223f3e14c507120c
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2018:2985
After upgrading Gluster storage nodes from glusterfs-3.8.10-1.el7 to glusterfs-3.8.11-1.el7 (this is the community build) followed by a system reboot. All storage nodes were updated and rebooted without waiting for Gluster to recover. The latter is a failure in our procedure and to be adressed separately. Shortly after the reboot pods started to report failure events: mkdir /var/lib/origin/openshift.local.volumes/pods/…/volumes/kubernetes.io~glusterfs/gluster-…: file exists Only a pod restart brought them back into working order. The logs written by the Gluster client started for the affected pods by OpenShift/K8s, /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/gluster-*/*.log, contained a single line: E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: 172.16.110.… (No data available) Digging this we see Kubernetes uses a single endpoint IP address[1] when mounting a volume. The "backup-volfile-server" option was only added in commit 43c0a686, to be shipped with Kubernetes 1.7 . This effectively makes every Gluster server a single point of failure for pods using it as a volume server. Should that server fail permanently or become unreachable for longer periods of time services running in pods will be impacted. Gluster client option "backup-volfile-server" should be implemented from [2] and backported to 3.4/3.5 [1] https://github.com/kubernetes/kubernetes/blob/v1.6.2/pkg/volume/glusterfs/glusterfs.go#L336 [2] https://github.com/kubernetes/kubernetes/commit/43c0a6869d35eaa8f762ee1e223f3e14c507120c