Bug 1601904
| Summary: | Creation of file PVC is very slow when scaled up to more than 200 | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Rachael <rgeorge> |
| Component: | heketi | Assignee: | John Mulligan <jmulligan> |
| Status: | CLOSED WORKSFORME | QA Contact: | Nitin Goyal <nigoyal> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | cns-3.9 | CC: | hchiramm, jmulligan, kramdoss, madam, nigoyal, pprakash, rhs-bugs, rtalur, sankarshan, storage-qa-internal |
| Target Milestone: | --- | ||
| Target Release: | CNS 3.10 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-08-24 10:22:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1568862 | ||
|
Description
Rachael
2018-07-17 12:31:18 UTC
sosreport and logs -> http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1601904/ After 600 volumes heketi is kind of stuck and it is giving a server busy error, hence marking this as failed QA. I was giving a 15 second of delay and after every 100 pvc i was waiting for all pvcs to come in bound state. [root@dhcp47-153 ~]# oc describe pvc c799 Name: c799 Namespace: glusterfs StorageClass: glusterfs-sc Status: Pending Volume: Labels: <none> Annotations: volume.beta.kubernetes.io/storage-class=glusterfs-sc volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 2h (x300 over 3h) persistentvolume-controller Failed to provision volume with StorageClass "glusterfs-sc": failed to create volume: failed to create volume: Server busy. Retry operation later. All pvc which are in pending state [root@dhcp47-153 ~]# oc get pvc | grep Pending c709 Pending glusterfs-sc 4h c710 Pending glusterfs-sc 4h c712 Pending glusterfs-sc 4h c713 Pending glusterfs-sc 4h c714 Pending glusterfs-sc 4h c717 Pending glusterfs-sc 4h c718 Pending glusterfs-sc 4h c719 Pending glusterfs-sc 4h c720 Pending glusterfs-sc 4h c721 Pending glusterfs-sc 4h c723 Pending glusterfs-sc 4h c726 Pending glusterfs-sc 3h c727 Pending glusterfs-sc 3h c728 Pending glusterfs-sc 3h c729 Pending glusterfs-sc 3h c733 Pending glusterfs-sc 3h c740 Pending glusterfs-sc 3h c742 Pending glusterfs-sc 3h c743 Pending glusterfs-sc 3h c744 Pending glusterfs-sc 3h c745 Pending glusterfs-sc 3h c746 Pending glusterfs-sc 3h c747 Pending glusterfs-sc 3h c748 Pending glusterfs-sc 3h c749 Pending glusterfs-sc 3h c753 Pending glusterfs-sc 3h c755 Pending glusterfs-sc 3h c760 Pending glusterfs-sc 3h c761 Pending glusterfs-sc 3h c762 Pending glusterfs-sc 3h c764 Pending glusterfs-sc 3h c767 Pending glusterfs-sc 3h c770 Pending glusterfs-sc 3h c772 Pending glusterfs-sc 3h c773 Pending glusterfs-sc 3h c775 Pending glusterfs-sc 3h c777 Pending glusterfs-sc 3h c778 Pending glusterfs-sc 3h c780 Pending glusterfs-sc 3h c781 Pending glusterfs-sc 3h c782 Pending glusterfs-sc 3h c785 Pending glusterfs-sc 3h c786 Pending glusterfs-sc 3h c787 Pending glusterfs-sc 3h c791 Pending glusterfs-sc 3h c793 Pending glusterfs-sc 3h c794 Pending glusterfs-sc 3h c795 Pending glusterfs-sc 3h c796 Pending glusterfs-sc 3h c797 Pending glusterfs-sc 3h c798 Pending glusterfs-sc 3h c799 Pending glusterfs-sc 3h Hi Johh and Talur, I did like John mentioned in comment 13 and provide logs hope it will help. http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1601904/logs1/ These 44 pvcs are in pending state for 3 hours. [root@dhcp46-180 ~]# oc get pvc | grep Pending c411 Pending glusterfs-sc 4h c414 Pending glusterfs-sc 4h c418 Pending glusterfs-sc 4h c420 Pending glusterfs-sc 4h c423 Pending glusterfs-sc 4h c425 Pending glusterfs-sc 4h c430 Pending glusterfs-sc 3h c432 Pending glusterfs-sc 3h c437 Pending glusterfs-sc 3h c441 Pending glusterfs-sc 3h c442 Pending glusterfs-sc 3h c444 Pending glusterfs-sc 3h c446 Pending glusterfs-sc 3h c448 Pending glusterfs-sc 3h c449 Pending glusterfs-sc 3h c451 Pending glusterfs-sc 3h c453 Pending glusterfs-sc 3h c455 Pending glusterfs-sc 3h c457 Pending glusterfs-sc 3h c459 Pending glusterfs-sc 3h c461 Pending glusterfs-sc 3h c463 Pending glusterfs-sc 3h c464 Pending glusterfs-sc 3h c465 Pending glusterfs-sc 3h c466 Pending glusterfs-sc 3h c467 Pending glusterfs-sc 3h c469 Pending glusterfs-sc 3h c470 Pending glusterfs-sc 3h c473 Pending glusterfs-sc 3h c475 Pending glusterfs-sc 3h c476 Pending glusterfs-sc 3h c477 Pending glusterfs-sc 3h c478 Pending glusterfs-sc 3h c480 Pending glusterfs-sc 3h c484 Pending glusterfs-sc 3h c485 Pending glusterfs-sc 3h c486 Pending glusterfs-sc 3h c487 Pending glusterfs-sc 3h c489 Pending glusterfs-sc 3h c492 Pending glusterfs-sc 3h c494 Pending glusterfs-sc 3h c496 Pending glusterfs-sc 3h c497 Pending glusterfs-sc 3h c498 Pending glusterfs-sc 3h I am facing this issue in block pvcs also so created a new bug for block as well. Bug -> https://bugzilla.redhat.com/show_bug.cgi?id=1612013 John, I tried delete heketi pod on one of my setup but it is not working. |