Description of problem: On scaling up a CNS setup by creating more than 200 PVCs (file) with a sleep time of 15 seconds between each create request, it was observed that the PVCs after 200 were still in pending state 18 mins after the create request. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: More details will be added soon
https://github.com/heketi/heketi/pull/1267 https://github.com/heketi/heketi/pull/1271
sosreport and logs -> http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1601904/ After 600 volumes heketi is kind of stuck and it is giving a server busy error, hence marking this as failed QA. I was giving a 15 second of delay and after every 100 pvc i was waiting for all pvcs to come in bound state. [root@dhcp47-153 ~]# oc describe pvc c799 Name: c799 Namespace: glusterfs StorageClass: glusterfs-sc Status: Pending Volume: Labels: <none> Annotations: volume.beta.kubernetes.io/storage-class=glusterfs-sc volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 2h (x300 over 3h) persistentvolume-controller Failed to provision volume with StorageClass "glusterfs-sc": failed to create volume: failed to create volume: Server busy. Retry operation later. All pvc which are in pending state [root@dhcp47-153 ~]# oc get pvc | grep Pending c709 Pending glusterfs-sc 4h c710 Pending glusterfs-sc 4h c712 Pending glusterfs-sc 4h c713 Pending glusterfs-sc 4h c714 Pending glusterfs-sc 4h c717 Pending glusterfs-sc 4h c718 Pending glusterfs-sc 4h c719 Pending glusterfs-sc 4h c720 Pending glusterfs-sc 4h c721 Pending glusterfs-sc 4h c723 Pending glusterfs-sc 4h c726 Pending glusterfs-sc 3h c727 Pending glusterfs-sc 3h c728 Pending glusterfs-sc 3h c729 Pending glusterfs-sc 3h c733 Pending glusterfs-sc 3h c740 Pending glusterfs-sc 3h c742 Pending glusterfs-sc 3h c743 Pending glusterfs-sc 3h c744 Pending glusterfs-sc 3h c745 Pending glusterfs-sc 3h c746 Pending glusterfs-sc 3h c747 Pending glusterfs-sc 3h c748 Pending glusterfs-sc 3h c749 Pending glusterfs-sc 3h c753 Pending glusterfs-sc 3h c755 Pending glusterfs-sc 3h c760 Pending glusterfs-sc 3h c761 Pending glusterfs-sc 3h c762 Pending glusterfs-sc 3h c764 Pending glusterfs-sc 3h c767 Pending glusterfs-sc 3h c770 Pending glusterfs-sc 3h c772 Pending glusterfs-sc 3h c773 Pending glusterfs-sc 3h c775 Pending glusterfs-sc 3h c777 Pending glusterfs-sc 3h c778 Pending glusterfs-sc 3h c780 Pending glusterfs-sc 3h c781 Pending glusterfs-sc 3h c782 Pending glusterfs-sc 3h c785 Pending glusterfs-sc 3h c786 Pending glusterfs-sc 3h c787 Pending glusterfs-sc 3h c791 Pending glusterfs-sc 3h c793 Pending glusterfs-sc 3h c794 Pending glusterfs-sc 3h c795 Pending glusterfs-sc 3h c796 Pending glusterfs-sc 3h c797 Pending glusterfs-sc 3h c798 Pending glusterfs-sc 3h c799 Pending glusterfs-sc 3h
Hi Johh and Talur, I did like John mentioned in comment 13 and provide logs hope it will help. http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1601904/logs1/ These 44 pvcs are in pending state for 3 hours. [root@dhcp46-180 ~]# oc get pvc | grep Pending c411 Pending glusterfs-sc 4h c414 Pending glusterfs-sc 4h c418 Pending glusterfs-sc 4h c420 Pending glusterfs-sc 4h c423 Pending glusterfs-sc 4h c425 Pending glusterfs-sc 4h c430 Pending glusterfs-sc 3h c432 Pending glusterfs-sc 3h c437 Pending glusterfs-sc 3h c441 Pending glusterfs-sc 3h c442 Pending glusterfs-sc 3h c444 Pending glusterfs-sc 3h c446 Pending glusterfs-sc 3h c448 Pending glusterfs-sc 3h c449 Pending glusterfs-sc 3h c451 Pending glusterfs-sc 3h c453 Pending glusterfs-sc 3h c455 Pending glusterfs-sc 3h c457 Pending glusterfs-sc 3h c459 Pending glusterfs-sc 3h c461 Pending glusterfs-sc 3h c463 Pending glusterfs-sc 3h c464 Pending glusterfs-sc 3h c465 Pending glusterfs-sc 3h c466 Pending glusterfs-sc 3h c467 Pending glusterfs-sc 3h c469 Pending glusterfs-sc 3h c470 Pending glusterfs-sc 3h c473 Pending glusterfs-sc 3h c475 Pending glusterfs-sc 3h c476 Pending glusterfs-sc 3h c477 Pending glusterfs-sc 3h c478 Pending glusterfs-sc 3h c480 Pending glusterfs-sc 3h c484 Pending glusterfs-sc 3h c485 Pending glusterfs-sc 3h c486 Pending glusterfs-sc 3h c487 Pending glusterfs-sc 3h c489 Pending glusterfs-sc 3h c492 Pending glusterfs-sc 3h c494 Pending glusterfs-sc 3h c496 Pending glusterfs-sc 3h c497 Pending glusterfs-sc 3h c498 Pending glusterfs-sc 3h
I am facing this issue in block pvcs also so created a new bug for block as well. Bug -> https://bugzilla.redhat.com/show_bug.cgi?id=1612013
John, I tried delete heketi pod on one of my setup but it is not working.