Bug 1443103

Summary: [Scale Testing] Scaling of gluster volumes beyond 300 failed as the PVC's got stuck in Pending state indefinitely
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Prasanth <pprakash>
Component: heketiAssignee: Humble Chirammal <hchiramm>
Status: CLOSED ERRATA QA Contact: Prasanth <pprakash>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: cns-3.5CC: hchiramm, kramdoss, mliyazud, pprakash, rcyriac, rhs-bugs, rreddy, storage-qa-internal, vinug
Target Milestone: ---   
Target Release: CNS 3.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
This is a build, docker image related issue.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-11 07:07:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1427156, 1451471, 1451474    
Bug Blocks: 1445447    

Description Prasanth 2017-04-18 13:16:10 UTC 
Description of problem:

Scaling of gluster volumes beyond 300 failed as the PVC's are stuck in Pending state indefinitely

#########
mongodb-300   Bound     pvc-555d619d-235a-11e7-941a-005056b35fb4   1Gi        RWO           1d
mongodb-301   Bound     pvc-5c19e370-235a-11e7-941a-005056b35fb4   1Gi        RWO           1d
mongodb-302   Pending                                                                       1d
mongodb-303   Pending                                                                       1d
mongodb-304   Pending                                                                       1d
mongodb-305   Pending                                                                       1d
mongodb-306   Pending                                                                       1d
mongodb-307   Pending                                                                       1d
mongodb-308   Pending                                                                       1d
mongodb-309   Pending                                                                       1d
#########

Version-Release number of selected component (if applicable):
heketi-client-4.0.0-7.el7rhgs.x86_64
cns-deploy-4.0.0-15.el7rhgs.x86_64

openshift v3.5.5.5
kubernetes v1.5.2+43a9be4

docker-1.12.6-16.el7.x86_64

How reproducible: 1/1


Steps to Reproduce:
1. Create a 3 node CNS 3.5 cluster (Memory allocated for each of the work nodes: 48 GB)
2. Create around 300 volumes using dynamic provisioning after rebooting one of the gluster node every 100 volumes
3. Ensure all the 300 volumes are created successfully, 3 RHGS nodes are in READY Staus and the corresponding gluster pods are in Running Status
4. Now create another 100 volumes in a loop


Actual results: Creation of volumes post 300 didn't happen successfully as the PVC's appears to be in "Pending" state. Any docker commands appears to be in hung state indefinitely on the rebooted node and the following errors are seen in the logs:

############
Apr 17 18:43:35 dhcp46-53 dockerd-current: time="2017-04-17T18:43:35.701375494+05:30" level=error msg="Error running exec in container: rpc error: code = 2 desc = shim error: context deadline exceeded"
Apr 17 18:43:35 dhcp46-53 dockerd-current: time="2017-04-17T18:43:35.701579923+05:30" level=error msg="Create container failed with error: shim error: context deadline exceeded"
Apr 17 18:43:35 dhcp46-53 dockerd-current: time="2017-04-17T18:43:35.701667909+05:30" level=error msg="Create container failed with error: shim error: context deadline exceeded"
Apr 17 18:43:35 dhcp46-53 dockerd-current: time="2017-04-17T18:43:35.701680254+05:30" level=error msg="Create container failed with error: shim error: context deadline exceeded"
Apr 17 18:43:35 dhcp46-53 atomic-openshift-node: I0417 18:43:35.703678   14716 prober.go:106] Readiness probe for "mongodb-66-1-6fgxh_storage-project(d1fd6d7d-234b-11e7-941a-005056b35fb4):mongodb" failed (failure): rpc error: code = 2 desc = shim error: context deadline exceeded
Apr 17 18:43:36 dhcp46-53 dockerd-current: time="2017-04-17T18:43:36.108326786+05:30" level=error msg="Handler for POST /v1.24/containers/7b930d6970ff721e84af5ff859a44c605942917878bb0835f767beffcab31b4c/start returned error: shim error: context deadline exceeded
############

Expected results: Scaling of volumes should be successful 


Additional info:

-------------------------------
[root@dhcp46-53 ~]# docker ps |grep rhgs
^C
[root@dhcp46-53 ~]# docker ps |grep rhgs
^C
[root@dhcp46-53 ~]# 
[root@dhcp46-53 ~]# 
[root@dhcp46-53 ~]# systemctl status docker -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─custom.conf
   Active: active (running) since Mon 2017-04-17 17:15:31 IST; 1h 34min ago
     Docs: http://docs.docker.com
 Main PID: 5659 (dockerd-current)
   Memory: 980.7M
   CGroup: /system.slice/docker.service
           ├─ 1444 /usr/bin/docker-containerd-shim-current f2cd25921cddd8709cacb0eab41f571ccbb00a5acfac4140ae2723d125e3a669 /var/run/docker/libcontainerd/f2cd25921cddd8709cacb0eab41f571ccbb00a5acfac4140ae2723d125e3a669 /usr/libexec/docker/docker-runc-current
           ├─ 1783 /usr/bin/docker-containerd-shim-current 4c4faf2bcce29d370ce2414b6b0bd15276eb22efe529c35e201e80c00325eb3b /var/run/docker/libcontainerd/4c4faf2bcce29d370ce2414b6b0bd15276eb22efe529c35e201e80c00325eb3b /usr/libexec/docker/docker-runc-current
           ├─ 2050 /usr/bin/docker-containerd-shim-current a0a1e3b7ed3d862052cfddff045e2335ad4c852c2b70fa7168164cd17eafc101 /var/run/docker/libcontainerd/a0a1e3b7ed3d862052cfddff045e2335ad4c852c2b70fa7168164cd17eafc101 /usr/libexec/docker/docker-runc-current
           ├─ 2209 /usr/bin/docker-containerd-shim-current 0f8cf5a5a6a7f405b344b6be73a48c5c43f82fd253ba196dff2818af7d176331 /var/run/docker/libcontainerd/0f8cf5a5a6a7f405b344b6be73a48c5c43f82fd253ba196dff2818af7d176331 /usr/libexec/docker/docker-runc-current
           ├─ 2547 /usr/bin/docker-containerd-shim-current 1fd1a898c09f9952c9f37da49dd9fe5de327a558a4d44e49658d8c08ecbb3d4b /var/run/docker/libcontainerd/1fd1a898c09f9952c9f37da49dd9fe5de327a558a4d44e49658d8c08ecbb3d4b /usr/libexec/docker/docker-runc-current
           ├─ 2728 /usr/bin/docker-containerd-shim-current 63918e08257f85444d5ac9b4f5f05f09a323130882c518401f6802bf55025b13 /var/run/docker/libcontainerd/63918e08257f85444d5ac9b4f5f05f09a323130882c518401f6802bf55025b13 /usr/libexec/docker/docker-runc-current
           ├─ 3189 /usr/bin/docker-containerd-shim-current 6b0d93d0d9b0ff7cc3fdfce0a300a27e63fc250c7435756d39c30a5587c91145 /var/run/docker/libcontainerd/6b0d93d0d9b0ff7cc3fdfce0a300a27e63fc250c7435756d39c30a5587c91145 /usr/libexec/docker/docker-runc-current
           ├─ 3541 /usr/bin/docker-containerd-shim-current 0e0039b0e16e9e6598dea50bf3fa8179d9b695a810f34b061f876041b6078f39 /var/run/docker/libcontainerd/0e0039b0e16e9e6598dea50bf3fa8179d9b695a810f34b061f876041b6078f39 /usr/libexec/docker/docker-runc-current
           ├─ 4003 /usr/bin/docker-containerd-shim-current a256027a1634cb082db68ae656e21a5d8ee52e6047ce616db8ef11d4589abc3d /var/run/docker/libcontainerd/a256027a1634cb082db68ae656e21a5d8ee52e6047ce616db8ef11d4589abc3d /usr/libexec/docker/docker-runc-current
           ├─ 4579 /usr/bin/docker-containerd-shim-current 6cc586efcf5bb06b0fb6ea98baff04119ad392b7c2074c2dfe81e2fb524c9f2b /var/run/docker/libcontainerd/6cc586efcf5bb06b0fb6ea98baff04119ad392b7c2074c2dfe81e2fb524c9f2b /usr/libexec/docker/docker-runc-current
           ├─ 4945 /usr/bin/docker-containerd-shim-current 918abdb0734dea511b785539f4b9722e6bebf919b41437337d1d7605114f1e1b /var/run/docker/libcontainerd/918abdb0734dea511b785539f4b9722e6bebf919b41437337d1d7605114f1e1b /usr/libexec/docker/docker-runc-current
           ├─ 5411 /usr/bin/docker-containerd-shim-current e5837454f5312e7d4a5dc75aebcd064630e1777d3b80935251bbea9ef64d9516 /var/run/docker/libcontainerd/e5837454f5312e7d4a5dc75aebcd064630e1777d3b80935251bbea9ef64d9516 /usr/libexec/docker/docker-runc-current
           ├─ 5659 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/docker--vg-docker--pool --storage-opt dm.use_deferred_removal=true --add-registry registry.access.stage.redhat.com --insecure-registry registry.access.stage.redhat.com
           ├─ 5775 /usr/bin/docker-containerd-shim-current 5d0eeea601cb2432b25bf2c79f24154cb275fdd989773950f81b60788f7a5130 /var/run/docker/libcontainerd/5d0eeea601cb2432b25bf2c79f24154cb275fdd989773950f81b60788f7a5130 /usr/libexec/docker/docker-runc-current
           ├─ 5952 /usr/bin/docker-containerd-shim-current c04838055cf01c69b6d765276c228c7c2f69ceb632803f569c8efee60dccfb11 /var/run/docker/libcontainerd/c04838055cf01c69b6d765276c228c7c2f69ceb632803f569c8efee60dccfb11 /usr/libexec/docker/docker-runc-current
           ├─ 6197 /usr/bin/docker-containerd-shim-current f0b96987f36dfb7cb88d2665e466050cef915a2386b1d744895b343edac9d1e3 /var/run/docker/libcontainerd/f0b96987f36dfb7cb88d2665e466050cef915a2386b1d744895b343edac9d1e3 /usr/libexec/docker/docker-runc-current
           ├─ 6461 /usr/bin/docker-containerd-shim-current 36d7bcedcec64d186d2ac252be61e41ee9e2481fd6ca30231cd3add65bc4257f /var/run/docker/libcontainerd/36d7bcedcec64d186d2ac252be61e41ee9e2481fd6ca30231cd3add65bc4257f /usr/libexec/docker/docker-runc-current
           ├─ 6664 /usr/bin/docker-containerd-shim-current cf2dd589acbec64f634229fee07de13eacac45dd7bedbf3f00a73e86624d7e3a /var/run/docker/libcontainerd/cf2dd589acbec64f634229fee07de13eacac45dd7bedbf3f00a73e86624d7e3a /usr/libexec/docker/docker-runc-current
           ├─ 6948 /usr/bin/docker-containerd-shim-current c5ce737761d1571b85172ad8f006636d6f0e3874d0fedb0b4290d484b5450f59 /var/run/docker/libcontainerd/c5ce737761d1571b85172ad8f006636d6f0e3874d0fedb0b4290d484b5450f59 /usr/libexec/docker/docker-runc-current
           ├─ 7213 /usr/bin/docker-containerd-shim-current dd9dbede54eae911b0f2547d27d05b54a30ecfd72895de4b9d1e6842b37d3a6d /var/run/docker/libcontainerd/dd9dbede54eae911b0f2547d27d05b54a30ecfd72895de4b9d1e6842b37d3a6d /usr/libexec/docker/docker-runc-current
           ├─ 7617 /usr/bin/docker-containerd-shim-current 9e2ff82021e8836b0d2a723e6ec57b6a3cf1f2fbf5422e45d6bc0683aa4b4370 /var/run/docker/libcontainerd/9e2ff82021e8836b0d2a723e6ec57b6a3cf1f2fbf5422e45d6bc0683aa4b4370 /usr/libexec/docker/docker-runc-current
           ├─ 7654 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc --runtime-args --systemd-cgroup=true
           ├─ 7885 /usr/bin/docker-containerd-shim-current b33b37d545534527676bbced289dea768d38643a80ac1f24a7aad4913ad21c32 /var/run/docker/libcontainerd/b33b37d545534527676bbced289dea768d38643a80ac1f24a7aad4913ad21c32 /usr/libexec/docker/docker-runc-current
           ├─ 8202 /usr/bin/docker-containerd-shim-current 2a6550e62b88ba7101dbf28ebbeaf54e1c775677118a2b1ba1c8d68ea7f0c341 /var/run/docker/libcontainerd/2a6550e62b88ba7101dbf28ebbeaf54e1c775677118a2b1ba1c8d68ea7f0c341 /usr/libexec/docker/docker-runc-current
           ├─ 8581 /usr/bin/docker-containerd-shim-current faac5783536c32354c25d805f9f5455276bb404ef3d8a10983d919d6162af44a /var/run/docker/libcontainerd/faac5783536c32354c25d805f9f5455276bb404ef3d8a10983d919d6162af44a /usr/libexec/docker/docker-runc-current
           ├─ 8815 /usr/bin/docker-containerd-shim-current 7b71f34e0789820b2d2b9278c0b26879b2d2c418f91e6d5714fc9a86da171e2c /var/run/docker/libcontainerd/7b71f34e0789820b2d2b9278c0b26879b2d2c418f91e6d5714fc9a86da171e2c /usr/libexec/docker/docker-runc-current
           ├─ 9069 /usr/bin/docker-containerd-shim-current 39564335a46c70680003c874d560ab5218fa1b655a386f7ad1adb1ddb1af2c66 /var/run/docker/libcontainerd/39564335a46c70680003c874d560ab5218fa1b655a386f7ad1adb1ddb1af2c66 /usr/libexec/docker/docker-runc-current
           ├─ 9353 /usr/bin/docker-containerd-shim-current e4f103be251b8b2d26716cf2a18f4e21f18a7b2d748ed2234395a81832f3c239 /var/run/docker/libcontainerd/e4f103be251b8b2d26716cf2a18f4e21f18a7b2d748ed2234395a81832f3c239 /usr/libexec/docker/docker-runc-current
           ├─ 9624 /usr/bin/docker-containerd-shim-current fb50e86c51ecaf674617e93eab425f1a2eccb5241ad5ce240003f1ac29f93940 /var/run/docker/libcontainerd/fb50e86c51ecaf674617e93eab425f1a2eccb5241ad5ce240003f1ac29f93940 /usr/libexec/docker/docker-runc-current
           ├─10218 /usr/bin/docker-containerd-shim-current e0e6ae99a7137d2d129c891bc93dea6ad5e74329d2965a018de93bfe6bac623e /var/run/docker/libcontainerd/e0e6ae99a7137d2d129c891bc93dea6ad5e74329d2965a018de93bfe6bac623e /usr/libexec/docker/docker-runc-current
           ├─10525 /usr/bin/docker-containerd-shim-current e59aee723581b832c2786f96ee64067e501c8f67c3406b263e47ca24b0a9ce11 /var/run/docker/libcontainerd/e59aee723581b832c2786f96ee64067e501c8f67c3406b263e47ca24b0a9ce11 /usr/libexec/docker/docker-runc-current
           ├─11821 /usr/bin/docker-containerd-shim-current 81fd5ee2a34478731c55953e81d57d1407b9a9daf45b6e156678fa492e479436 /var/run/docker/libcontainerd/81fd5ee2a34478731c55953e81d57d1407b9a9daf45b6e156678fa492e479436 /usr/libexec/docker/docker-runc-current
           ├─12636 /usr/bin/docker-containerd-shim-current d7611a5fa27d7513c39d35dac933534b385710f6b9908e3d0f4c4ab4793e750d /var/run/docker/libcontainerd/d7611a5fa27d7513c39d35dac933534b385710f6b9908e3d0f4c4ab4793e750d /usr/libexec/docker/docker-runc-current
           ├─12956 /usr/bin/docker-containerd-shim-current bc0f284a64de196f10b0b30cecab13509d55f7bc77446c7b781f9d3a1246b04d /var/run/docker/libcontainerd/bc0f284a64de196f10b0b30cecab13509d55f7bc77446c7b781f9d3a1246b04d /usr/libexec/docker/docker-runc-current
           ├─13686 /usr/bin/docker-containerd-shim-current 5fc918c423f9d77317abe076fd7dab3f562f5d754fc16261f4c07825a288a87f /var/run/docker/libcontainerd/5fc918c423f9d77317abe076fd7dab3f562f5d754fc16261f4c07825a288a87f /usr/libexec/docker/docker-runc-current
           ├─14161 /usr/bin/docker-containerd-shim-current d91754283a13060621a587b70301665a6f513683ae7c9928a860e3e8714fcd76 /var/run/docker/libcontainerd/d91754283a13060621a587b70301665a6f513683ae7c9928a860e3e8714fcd76 /usr/libexec/docker/docker-runc-current
           ├─14596 /usr/bin/docker-containerd-shim-current 63767098b3e7ae67a2358174c1e9dc77e7e7ec037e08726470043e25d71b0f02 /var/run/docker/libcontainerd/63767098b3e7ae67a2358174c1e9dc77e7e7ec037e08726470043e25d71b0f02 /usr/libexec/docker/docker-runc-current
           ├─14782 /usr/bin/docker-containerd-shim-current ca69858bb65148799bfd2993036bb847c54e671794762578f281d473ed93be54 /var/run/docker/libcontainerd/ca69858bb65148799bfd2993036bb847c54e671794762578f281d473ed93be54 /usr/libexec/docker/docker-runc-current
           ├─15497 /usr/bin/docker-containerd-shim-current cd92b03f7f66e47985658f1f36dbd86f706ffff07b1dc50f902a7806410f1245 /var/run/docker/libcontainerd/cd92b03f7f66e47985658f1f36dbd86f706ffff07b1dc50f902a7806410f1245 /usr/libexec/docker/docker-runc-current
           ├─15560 /usr/bin/docker-containerd-shim-current 0bd941c57a4f3942d993c287a7512d12b9345e68ee8a3b666148610de77af70f /var/run/docker/libcontainerd/0bd941c57a4f3942d993c287a7512d12b9345e68ee8a3b666148610de77af70f /usr/libexec/docker/docker-runc-current
           ├─15923 /usr/bin/docker-containerd-shim-current 0d14753c6fd12a34be289fb2e678883cb46f9a2f05d03757fb035b49729b3e33 /var/run/docker/libcontainerd/0d14753c6fd12a34be289fb2e678883cb46f9a2f05d03757fb035b49729b3e33 /usr/libexec/docker/docker-runc-current
           ├─16295 /usr/bin/docker-containerd-shim-current 8e92160bb3f632654db84bc09dbd7d5fe61d43f107d01f1660412c3db33d5779 /var/run/docker/libcontainerd/8e92160bb3f632654db84bc09dbd7d5fe61d43f107d01f1660412c3db33d5779 /usr/libexec/docker/docker-runc-current
           ├─16850 /usr/bin/docker-containerd-shim-current d6871c372b3d9b4261aa8da7fcd6b44e150bb3bc51d2f536608a2a81c1e34458 /var/run/docker/libcontainerd/d6871c372b3d9b4261aa8da7fcd6b44e150bb3bc51d2f536608a2a81c1e34458 /usr/libexec/docker/docker-runc-current
           ├─17125 /usr/bin/docker-containerd-shim-current 214bae1fe89ac76da2575475e0f6a974e7cdae0069435f5c9892818a23ed6257 /var/run/docker/libcontainerd/214bae1fe89ac76da2575475e0f6a974e7cdae0069435f5c9892818a23ed6257 /usr/libexec/docker/docker-runc-current
           ├─17543 /usr/bin/docker-containerd-shim-current 6fd24e11fc317b263c4a34195775a230cc22fdcc387a993f08dcd695f0bf0a5e /var/run/docker/libcontainerd/6fd24e11fc317b263c4a34195775a230cc22fdcc387a993f08dcd695f0bf0a5e /usr/libexec/docker/docker-runc-current
           ├─17795 /usr/bin/docker-containerd-shim-current 9877de2e4f73361ee561c73fe5b11251c9daa18640dfab10cd2e8d1823f7060a /var/run/docker/libcontainerd/9877de2e4f73361ee561c73fe5b11251c9daa18640dfab10cd2e8d1823f7060a /usr/libexec/docker/docker-runc-current
           ├─18143 /usr/bin/docker-containerd-shim-current 4c588bec6569307788cd1f7c8352380f5b49a1fe09877a07000b6e2f00b87c2b /var/run/docker/libcontainerd/4c588bec6569307788cd1f7c8352380f5b49a1fe09877a07000b6e2f00b87c2b /usr/libexec/docker/docker-runc-current
           ├─18466 /usr/bin/docker-containerd-shim-current 4e2c1aef4ec35d53b01cceced463626b687487a56020202f4ada8752103de430 /var/run/docker/libcontainerd/4e2c1aef4ec35d53b01cceced463626b687487a56020202f4ada8752103de430 /usr/libexec/docker/docker-runc-current
           ├─18593 /usr/bin/docker-containerd-shim-current 4f81610ff3684c06d14257f6c7e5795692271d49857fc162fa741e199f17884d /var/run/docker/libcontainerd/4f81610ff3684c06d14257f6c7e5795692271d49857fc162fa741e199f17884d /usr/libexec/docker/docker-runc-current
           ├─19202 /usr/bin/docker-containerd-shim-current 2403fe038f8d0043fcb3bb2bb51b295873abbed704e6903546e8e4d223862837 /var/run/docker/libcontainerd/2403fe038f8d0043fcb3bb2bb51b295873abbed704e6903546e8e4d223862837 /usr/libexec/docker/docker-runc-current
           ├─19304 /usr/bin/docker-containerd-shim-current a24a70ec9826457e9cccffdc4a1bcd39b7063515c7e57643c3dad1c1c1a0051e /var/run/docker/libcontainerd/a24a70ec9826457e9cccffdc4a1bcd39b7063515c7e57643c3dad1c1c1a0051e /usr/libexec/docker/docker-runc-current
           ├─19794 /usr/bin/docker-containerd-shim-current c1fc689ecb558e78a27dd16a550a71612a4e2393d1028d5a85ff23280f4bcd14 /var/run/docker/libcontainerd/c1fc689ecb558e78a27dd16a550a71612a4e2393d1028d5a85ff23280f4bcd14 /usr/libexec/docker/docker-runc-current
           ├─20069 /usr/bin/docker-containerd-shim-current 35cf7ea42c74da7dbdca80cec211c81f923a5515910e850212287d6a1c017707 /var/run/docker/libcontainerd/35cf7ea42c74da7dbdca80cec211c81f923a5515910e850212287d6a1c017707 /usr/libexec/docker/docker-runc-current
           ├─21476 /usr/bin/docker-containerd-shim-current 48ffb351782de52fe9d51ee3c3a264717fc2403f798f36224654b3dd76721117 /var/run/docker/libcontainerd/48ffb351782de52fe9d51ee3c3a264717fc2403f798f36224654b3dd76721117 /usr/libexec/docker/docker-runc-current
           ├─21881 /usr/bin/docker-containerd-shim-current 39168651ebe7d50c3b54aee14a550278379c9a1eaa47b329d3e0cfebe95b065b /var/run/docker/libcontainerd/39168651ebe7d50c3b54aee14a550278379c9a1eaa47b329d3e0cfebe95b065b /usr/libexec/docker/docker-runc-current
           ├─21986 /usr/bin/docker-containerd-shim-current 3c81bf59e1ae592e997c2a665e8ece2fcac9d1020a92675e168938dd456c060f /var/run/docker/libcontainerd/3c81bf59e1ae592e997c2a665e8ece2fcac9d1020a92675e168938dd456c060f /usr/libexec/docker/docker-runc-current
           ├─22213 /usr/bin/docker-containerd-shim-current 4dc2d97ed24b353ccb3519c9f6834b382fcea37ab93345246ca6879d506d037e /var/run/docker/libcontainerd/4dc2d97ed24b353ccb3519c9f6834b382fcea37ab93345246ca6879d506d037e /usr/libexec/docker/docker-runc-current
           ├─22309 /usr/bin/docker-containerd-shim-current 5b6e6f16968e9f1863cb4e54806a0e39a4aa85b606951f66477628664f24c518 /var/run/docker/libcontainerd/5b6e6f16968e9f1863cb4e54806a0e39a4aa85b606951f66477628664f24c518 /usr/libexec/docker/docker-runc-current
           ├─22461 /usr/bin/docker-containerd-shim-current 6ff6935bd78b916b4c3a17c2f91c0475e164aa6e53f09103cfad2df36963d90f /var/run/docker/libcontainerd/6ff6935bd78b916b4c3a17c2f91c0475e164aa6e53f09103cfad2df36963d90f /usr/libexec/docker/docker-runc-current
           ├─22504 /usr/bin/docker-containerd-shim-current 3ba155e1f3109d1e100f3fda972eac6f07aa62befa68e75abfc6b06049f1234b /var/run/docker/libcontainerd/3ba155e1f3109d1e100f3fda972eac6f07aa62befa68e75abfc6b06049f1234b /usr/libexec/docker/docker-runc-current
           ├─22549 /usr/bin/docker-containerd-shim-current b7d69ffba5bc76200cd6c9718ed240364562ad5ad0cb9165ccf08fca20d62d05 /var/run/docker/libcontainerd/b7d69ffba5bc76200cd6c9718ed240364562ad5ad0cb9165ccf08fca20d62d05 /usr/libexec/docker/docker-runc-current
           ├─22664 /usr/bin/docker-containerd-shim-current 6da13910777a3d1bbc1502581da1b071bf6a8df5df42464e2383b0ba76220358 /var/run/docker/libcontainerd/6da13910777a3d1bbc1502581da1b071bf6a8df5df42464e2383b0ba76220358 /usr/libexec/docker/docker-runc-current
           ├─22814 /usr/bin/docker-containerd-shim-current b858e139a67f474dffda1fe56cd4b9132c1bf6620f27347e8b20d5b9b340d772 /var/run/docker/libcontainerd/b858e139a67f474dffda1fe56cd4b9132c1bf6620f27347e8b20d5b9b340d772 /usr/libexec/docker/docker-runc-current
           ├─22851 /usr/bin/docker-containerd-shim-current 41db0c21214e3f5360e2611076469c39f3e367f7e8832c1a98f3bee32dff1527 /var/run/docker/libcontainerd/41db0c21214e3f5360e2611076469c39f3e367f7e8832c1a98f3bee32dff1527 /usr/libexec/docker/docker-runc-current
           ├─22971 /usr/bin/docker-containerd-shim-current 5456c365f163a92386a7fb734243a976fc03754a74fa2c125997c134e70da074 /var/run/docker/libcontainerd/5456c365f163a92386a7fb734243a976fc03754a74fa2c125997c134e70da074 /usr/libexec/docker/docker-runc-current
           ├─23044 /usr/bin/docker-containerd-shim-current 79def0f00704cd1e35864baa8158103b61236aa49dfbee0bf3d5f78a0a59951d /var/run/docker/libcontainerd/79def0f00704cd1e35864baa8158103b61236aa49dfbee0bf3d5f78a0a59951d /usr/libexec/docker/docker-runc-current
           ├─23155 /usr/bin/docker-containerd-shim-current 1151a97f0c0f761c9e36eaf62086e24ee6c7569e65b8575a70a04772cea53031 /var/run/docker/libcontainerd/1151a97f0c0f761c9e36eaf62086e24ee6c7569e65b8575a70a04772cea53031 /usr/libexec/docker/docker-runc-current
           ├─23583 /usr/bin/docker-containerd-shim-current a6ebeae4edfed654183bc237a7e13840e4580b416569f72c5c42336ddf592b7f /var/run/docker/libcontainerd/a6ebeae4edfed654183bc237a7e13840e4580b416569f72c5c42336ddf592b7f /usr/libexec/docker/docker-runc-current
           ├─23695 /usr/bin/docker-containerd-shim-current 677d14ac02d48c3c9c5322e72cc1036dfb9258fa18aeb46ed03196e01e6e6e04 /var/run/docker/libcontainerd/677d14ac02d48c3c9c5322e72cc1036dfb9258fa18aeb46ed03196e01e6e6e04 /usr/libexec/docker/docker-runc-current
           ├─23804 /usr/bin/docker-containerd-shim-current c03855a815d4dd242af6dc7b671cc514d9f3824c6f6bcf1fb923aab408c7c85d /var/run/docker/libcontainerd/c03855a815d4dd242af6dc7b671cc514d9f3824c6f6bcf1fb923aab408c7c85d /usr/libexec/docker/docker-runc-current
           ├─23980 /usr/bin/docker-containerd-shim-current 641ff6b128dfc5187f3363b11033e1430556acf0b394767fa2f6050753949860 /var/run/docker/libcontainerd/641ff6b128dfc5187f3363b11033e1430556acf0b394767fa2f6050753949860 /usr/libexec/docker/docker-runc-current
           ├─24146 /usr/bin/docker-containerd-shim-current a6d352b6d38244606cff427bf345da23560ad5c3f2870606ed06fd4c5ed0993d /var/run/docker/libcontainerd/a6d352b6d38244606cff427bf345da23560ad5c3f2870606ed06fd4c5ed0993d /usr/libexec/docker/docker-runc-current
           ├─24162 /usr/bin/docker-containerd-shim-current 72104619901cb96c1f32129b74f0f71baa45a9d0ec65d36286d9b1fdfab26b3a /var/run/docker/libcontainerd/72104619901cb96c1f32129b74f0f71baa45a9d0ec65d36286d9b1fdfab26b3a /usr/libexec/docker/docker-runc-current
           ├─24280 /usr/bin/docker-containerd-shim-current eac44776cb2487597d83f10552153e1e53ad90b8a7d17c77e3590f60cfad0c4f /var/run/docker/libcontainerd/eac44776cb2487597d83f10552153e1e53ad90b8a7d17c77e3590f60cfad0c4f /usr/libexec/docker/docker-runc-current
           ├─24531 /usr/bin/docker-containerd-shim-current 1f47cfc619ae607129e7b603647f7c1d9e5e97228ceee65995dd5eea695e1575 /var/run/docker/libcontainerd/1f47cfc619ae607129e7b603647f7c1d9e5e97228ceee65995dd5eea695e1575 /usr/libexec/docker/docker-runc-current
           ├─24546 /usr/bin/docker-containerd-shim-current b16e797bf573023c1e96eddd670cab22abbbe26c0f33e76c8a7f24fa07c23308 /var/run/docker/libcontainerd/b16e797bf573023c1e96eddd670cab22abbbe26c0f33e76c8a7f24fa07c23308 /usr/libexec/docker/docker-runc-current
           ├─24745 /usr/bin/docker-containerd-shim-current 80b22125134275532e71ff04b064c775b7d9ba083dae4563f297f663ca2ccc27 /var/run/docker/libcontainerd/80b22125134275532e71ff04b064c775b7d9ba083dae4563f297f663ca2ccc27 /usr/libexec/docker/docker-runc-current
           ├─24966 /usr/bin/docker-containerd-shim-current c789eba71803a46edd4ac447880a12700520cd0960681e55791bf17370c7406b /var/run/docker/libcontainerd/c789eba71803a46edd4ac447880a12700520cd0960681e55791bf17370c7406b /usr/libexec/docker/docker-runc-current
           ├─25146 /usr/bin/docker-containerd-shim-current eb3bd6007d23ee2079b9214c0043ba75927bca1e09d3d05eba425c2bb3056e73 /var/run/docker/libcontainerd/eb3bd6007d23ee2079b9214c0043ba75927bca1e09d3d05eba425c2bb3056e73 /usr/libexec/docker/docker-runc-current
           ├─25186 /usr/bin/docker-containerd-shim-current 53ed13905d10870623316f24d7f0285f3b57ca9b3584380260ccd161cf43845e /var/run/docker/libcontainerd/53ed13905d10870623316f24d7f0285f3b57ca9b3584380260ccd161cf43845e /usr/libexec/docker/docker-runc-current
           ├─25771 /usr/bin/docker-containerd-shim-current 8f726d18e7c263772abb7839e95bc46c07fc05cac54f197459caf54097f2ce60 /var/run/docker/libcontainerd/8f726d18e7c263772abb7839e95bc46c07fc05cac54f197459caf54097f2ce60 /usr/libexec/docker/docker-runc-current
           ├─31107 /usr/bin/docker-containerd-shim-current 48f7660fccba3b246398aba9a4c81122c48b54b8c43a010456b5bdf17f383c53 /var/run/docker/libcontainerd/48f7660fccba3b246398aba9a4c81122c48b54b8c43a010456b5bdf17f383c53 /usr/libexec/docker/docker-runc-current
           └─32505 /usr/bin/docker-containerd-shim-current e59aee723581b832c2786f96ee64067e501c8f67c3406b263e47ca24b0a9ce11 /var/run/docker/libcontainerd/e59aee723581b832c2786f96ee64067e501c8f67c3406b263e47ca24b0a9ce11 /usr/libexec/docker/docker-runc-current

Apr 17 18:49:52 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: time="2017-04-17T18:49:52.094398492+05:30" level=info msg="{Action=start, LoginUID=4294967295, PID=14716}"
Apr 17 18:49:52 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: 10.131.0.1 - - [17/Apr/2017:13:19:52 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"
Apr 17 18:49:52 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: 10.131.0.1 - - [17/Apr/2017:13:19:52 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"
Apr 17 18:50:02 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: 10.131.0.1 - - [17/Apr/2017:13:20:02 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"
Apr 17 18:50:02 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: 10.131.0.1 - - [17/Apr/2017:13:20:02 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"
Apr 17 18:50:07 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: time="2017-04-17T18:50:07.117299857+05:30" level=error msg="Error running exec in container: rpc error: code = 2 desc = shim error: context deadline exceeded"
Apr 17 18:50:07 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: time="2017-04-17T18:50:07.139602294+05:30" level=info msg="{Action=exec, LoginUID=4294967295, PID=14716}"
Apr 17 18:50:07 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: time="2017-04-17T18:50:07.141358223+05:30" level=info msg="{Action=start, LoginUID=4294967295, PID=14716}"
Apr 17 18:50:12 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: 10.131.0.1 - - [17/Apr/2017:13:20:12 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"
Apr 17 18:50:12 dhcp46-53.lab.eng.blr.redhat.com dockerd-current[5659]: 10.131.0.1 - - [17/Apr/2017:13:20:12 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"
-----------------------------------
Comment 2 Humble Chirammal 2017-07-24 18:08:08 UTC 
This was due to a docker bug which should be fixed with latest docker. I am moving this bug to ON_QA.
Comment 3 krishnaram Karthick 2017-09-14 07:29:45 UTC 
This bug shall be picked once we have a fix or workaround for https://bugzilla.redhat.com/show_bug.cgi?id=1473172
Comment 4 Humble Chirammal 2017-09-14 07:35:04 UTC 
(In reply to krishnaram Karthick from comment #3)
> This bug shall be picked once we have a fix or workaround for
> https://bugzilla.redhat.com/show_bug.cgi?id=1473172

Karthick, if you have more than one device in your node, you could give a try :). Atleast it gives some confirmation about the scale part of it.
Comment 5 krishnaram Karthick 2017-09-15 16:07:46 UTC 
300 PVCs got created without any issues. There was no hangs, but there was some delay to create PVCs as the number of PVCs increased. 300th PVC took close to 4 minutes.

Moving the bug to verified.
Comment 6 errata-xmlrpc 2017-10-11 07:07:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2879