Description of problem: Create n pods and each of them uses a PVC. The target is 1000, and in this run, it is less than 600. It has been tested with at least 3 clusters. Problem occurs between 500 - 600. Version-Release number of selected component (if applicable): # oc get pod -n glusterfs -o yaml | grep "image:" | sort -u image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-gluster-block-prov-rhel7:3.3.1-10 image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-server-rhel7:3.3.1-13 image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-volmanager-rhel7:3.3.1-10 # yum list installed | grep openshift atomic-openshift.x86_64 3.10.0-0.50.0.git.0.db6dfd6.el7 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: The logs will be attched.
Are those logs normal? heketi [negroni] Completed 401 Unauthorized in 58.847µs master-controller E0523 16:21:25.001057 1 glusterfs.go:708] failed to create volume: failed to create volume: Token used before issued After reading comments from https://bugzilla.redhat.com/show_bug.cgi?id=1541323 I tried to delete the heketi pod and check if the time on nodes is synced. It did not solve the problem. Those failures still show in heketi and controller logs. The last time we got 1000 is with the same cns images and atomic-openshift.x86_64 3.10.0-0.27.0.git.0.baf1ec4.el7
patch posted upstream at https://github.com/heketi/heketi/pull/1223
Fixed in version : rhgs-volmanager-rhel7:3.3.1-20
Not seeing this bug after 1000 gluster.file PVCs were created Tested with # oc get pod -n glusterfs -o yaml | grep "image:" | sort -u image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-gluster-block-prov-rhel7:3.3.1-20 image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-server-rhel7:3.3.1-27 image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-volmanager-rhel7:3.3.1-21 # yum list installed | grep openshift atomic-openshift.x86_64 3.10.18-1.git.0.13dc4a0.el7
Updated doc text in the Doc Text field. Please review for technical accuracy.
Doc Text looks OK
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2686