Bug 1613781

Description of problem:


Scalability issues with external provisioner controller. While we scale up our ( gluster block PVCs served by the external provisioner in CNS) setup to >100 PVC requests, we are seeing congestion in serving PVC requests. The issues are mainly on leader election and api throttling which is known and getting addressed in external storage repo. 

--snip-- from another bug

I requested >100 PVCs via:
for i in $(seq 0 100); do sed s/PVCNAME/block-1-$i/ bc.yaml | oc apply -f- ; done

There's an initial burst of activity in the block provisioner logs and then activity decreases and does not appear to resume. Heketi logs indicate that it is not getting any requests for new block volumes.

The logging from the block provisioner mainly appears to log lines like:
I0728 14:27:56.953340       1 controller.go:1167] scheduleOperation[lock-provision-cns/block-1-20[c06a4991-926f-11e8-b008-5254005f433e]]
I0728 14:27:56.953352       1 controller.go:1167] scheduleOperation[lock-provision-cns/block-1-33[c2759f71-926f-11e8-b008-5254005f433e]]
I0728 14:27:56.953365       1 controller.go:1167] scheduleOperation[lock-provision-cns/block-1-26[c1525238-926f-11e8-b008-5254005f433e]]
I0728 14:27:56.953377       1 controller.go:1167] scheduleOperation[lock-provision-cns/block-1-43[c4598d59-926f-11e8-b008-5254005f433e]]
I0728 14:27:56.953387       1 controller.go:1167] scheduleOperation[lock-provision-cns/block-1-78[ca3d61dc-926f-11e8-b008-5254005f433e]]
I0728 14:27:56.953400       1 controller.go:1167] scheduleOperation[lock-provision-cns/block-1-64[c8048cb3-926f-11e8-b008-5254005f433e]]
I0728 14:27:56.953411       1 controller.go:1167] scheduleOperation[lock-provision-cns/block-1-84[cb4a2c2a-926f-11e8-b008-5254005f433e]]
I0728 14:27:56.953422       1 controller.go:1167] scheduleOperation[lock-provision-cns/block-1-90[cc45933f-926f-11e8-b008-5254005f433e]]


I did some searching around and found the following links:
  https://github.com/kubernetes-csi/external-provisioner/issues/68

  https://github.com/kubernetes-incubator/external-storage/pull/837

  https://github.com/kubernetes-csi/external-provisioner/pull/104

  https://github.com/kubernetes-incubator/external-storage/pull/825

  https://github.com/kubernetes-incubator/external-storage/commit/126c9ffc8ef125460f6bd75d903ea1d26051b730#diff-634ce7794d379f3ba2119e2413074537


They indicate that there are potentially scalability issues with how the external provisioners communicate with the k8s api server. Suspiciously, one of the above links mentions "100 PVCs created at same time" which is quite similar to how our PVCs were created.

--/snip--




Version-Release number of selected component (if applicable):

OCP 3.10

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info: