Bug 1601904 - Creation of file PVC is very slow when scaled up to more than 200
Summary: Creation of file PVC is very slow when scaled up to more than 200
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: cns-3.9
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: CNS 3.10
Assignee: John Mulligan
QA Contact: Nitin Goyal
URL:
Whiteboard:
Depends On:
Blocks: 1568862
TreeView+ depends on / blocked
 
Reported: 2018-07-17 12:31 UTC by Rachael
Modified: 2018-09-18 08:02 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-24 10:22:07 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Rachael 2018-07-17 12:31:18 UTC
Description of problem:
On scaling up a CNS setup by creating more than 200 PVCs (file) with a sleep time of 15 seconds between each create request, it was observed that the PVCs after 200 were still in pending state 18 mins after the create request.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:

More details will be added soon

Comment 10 Nitin Goyal 2018-07-30 12:24:11 UTC
sosreport and logs ->
http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1601904/

After 600 volumes heketi is kind of stuck and it is giving a server busy error, hence marking this as failed QA.
I was giving a 15 second of delay and after every 100 pvc i was waiting for all pvcs to come in bound state.

[root@dhcp47-153 ~]# oc describe pvc c799
Name:          c799
Namespace:     glusterfs
StorageClass:  glusterfs-sc
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class=glusterfs-sc
               volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type     Reason              Age                 From                         Message
  ----     ------              ----                ----                         -------
  Warning  ProvisioningFailed  2h (x300 over 3h)   persistentvolume-controller  Failed to provision volume with StorageClass "glusterfs-sc": failed to create volume: failed to create volume: Server busy. Retry operation later.

All pvc which are in pending state
[root@dhcp47-153 ~]# oc get pvc | grep Pending
c709      Pending                                                                        glusterfs-sc   4h
c710      Pending                                                                        glusterfs-sc   4h
c712      Pending                                                                        glusterfs-sc   4h
c713      Pending                                                                        glusterfs-sc   4h
c714      Pending                                                                        glusterfs-sc   4h
c717      Pending                                                                        glusterfs-sc   4h
c718      Pending                                                                        glusterfs-sc   4h
c719      Pending                                                                        glusterfs-sc   4h
c720      Pending                                                                        glusterfs-sc   4h
c721      Pending                                                                        glusterfs-sc   4h
c723      Pending                                                                        glusterfs-sc   4h
c726      Pending                                                                        glusterfs-sc   3h
c727      Pending                                                                        glusterfs-sc   3h
c728      Pending                                                                        glusterfs-sc   3h
c729      Pending                                                                        glusterfs-sc   3h
c733      Pending                                                                        glusterfs-sc   3h
c740      Pending                                                                        glusterfs-sc   3h
c742      Pending                                                                        glusterfs-sc   3h
c743      Pending                                                                        glusterfs-sc   3h
c744      Pending                                                                        glusterfs-sc   3h
c745      Pending                                                                        glusterfs-sc   3h
c746      Pending                                                                        glusterfs-sc   3h
c747      Pending                                                                        glusterfs-sc   3h
c748      Pending                                                                        glusterfs-sc   3h
c749      Pending                                                                        glusterfs-sc   3h
c753      Pending                                                                        glusterfs-sc   3h
c755      Pending                                                                        glusterfs-sc   3h
c760      Pending                                                                        glusterfs-sc   3h
c761      Pending                                                                        glusterfs-sc   3h
c762      Pending                                                                        glusterfs-sc   3h
c764      Pending                                                                        glusterfs-sc   3h
c767      Pending                                                                        glusterfs-sc   3h
c770      Pending                                                                        glusterfs-sc   3h
c772      Pending                                                                        glusterfs-sc   3h
c773      Pending                                                                        glusterfs-sc   3h
c775      Pending                                                                        glusterfs-sc   3h
c777      Pending                                                                        glusterfs-sc   3h
c778      Pending                                                                        glusterfs-sc   3h
c780      Pending                                                                        glusterfs-sc   3h
c781      Pending                                                                        glusterfs-sc   3h
c782      Pending                                                                        glusterfs-sc   3h
c785      Pending                                                                        glusterfs-sc   3h
c786      Pending                                                                        glusterfs-sc   3h
c787      Pending                                                                        glusterfs-sc   3h
c791      Pending                                                                        glusterfs-sc   3h
c793      Pending                                                                        glusterfs-sc   3h
c794      Pending                                                                        glusterfs-sc   3h
c795      Pending                                                                        glusterfs-sc   3h
c796      Pending                                                                        glusterfs-sc   3h
c797      Pending                                                                        glusterfs-sc   3h
c798      Pending                                                                        glusterfs-sc   3h
c799      Pending                                                                        glusterfs-sc   3h

Comment 14 Nitin Goyal 2018-08-02 14:20:29 UTC
Hi Johh and Talur,

I did like John mentioned in comment 13 and provide logs hope it will help.

http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1601904/logs1/

These 44 pvcs are in pending state for 3 hours.

[root@dhcp46-180 ~]# oc get pvc | grep Pending
c411      Pending                                                                        glusterfs-sc   4h
c414      Pending                                                                        glusterfs-sc   4h
c418      Pending                                                                        glusterfs-sc   4h
c420      Pending                                                                        glusterfs-sc   4h
c423      Pending                                                                        glusterfs-sc   4h
c425      Pending                                                                        glusterfs-sc   4h
c430      Pending                                                                        glusterfs-sc   3h
c432      Pending                                                                        glusterfs-sc   3h
c437      Pending                                                                        glusterfs-sc   3h
c441      Pending                                                                        glusterfs-sc   3h
c442      Pending                                                                        glusterfs-sc   3h
c444      Pending                                                                        glusterfs-sc   3h
c446      Pending                                                                        glusterfs-sc   3h
c448      Pending                                                                        glusterfs-sc   3h
c449      Pending                                                                        glusterfs-sc   3h
c451      Pending                                                                        glusterfs-sc   3h
c453      Pending                                                                        glusterfs-sc   3h
c455      Pending                                                                        glusterfs-sc   3h
c457      Pending                                                                        glusterfs-sc   3h
c459      Pending                                                                        glusterfs-sc   3h
c461      Pending                                                                        glusterfs-sc   3h
c463      Pending                                                                        glusterfs-sc   3h
c464      Pending                                                                        glusterfs-sc   3h
c465      Pending                                                                        glusterfs-sc   3h
c466      Pending                                                                        glusterfs-sc   3h
c467      Pending                                                                        glusterfs-sc   3h
c469      Pending                                                                        glusterfs-sc   3h
c470      Pending                                                                        glusterfs-sc   3h
c473      Pending                                                                        glusterfs-sc   3h
c475      Pending                                                                        glusterfs-sc   3h
c476      Pending                                                                        glusterfs-sc   3h
c477      Pending                                                                        glusterfs-sc   3h
c478      Pending                                                                        glusterfs-sc   3h
c480      Pending                                                                        glusterfs-sc   3h
c484      Pending                                                                        glusterfs-sc   3h
c485      Pending                                                                        glusterfs-sc   3h
c486      Pending                                                                        glusterfs-sc   3h
c487      Pending                                                                        glusterfs-sc   3h
c489      Pending                                                                        glusterfs-sc   3h
c492      Pending                                                                        glusterfs-sc   3h
c494      Pending                                                                        glusterfs-sc   3h
c496      Pending                                                                        glusterfs-sc   3h
c497      Pending                                                                        glusterfs-sc   3h
c498      Pending                                                                        glusterfs-sc   3h

Comment 20 Nitin Goyal 2018-08-03 09:14:13 UTC
I am facing this issue in block pvcs also so created a new bug for block as well.

Bug -> https://bugzilla.redhat.com/show_bug.cgi?id=1612013

Comment 23 Nitin Goyal 2018-08-09 04:55:25 UTC
John,

I tried delete heketi pod on one of my setup but it is not working.


Note You need to log in before you can comment on or make changes to this bug.