### Description of problem ### During the attempt to create only one PVC using gluster-block the following message is displayed: ~~~ Failed to provision volume with StorageClass "glusterfs-storage-block": failed to create volume: heketi block volume creation failed: [heketi] failed to create volume: Server busy. Retry operation later. ~~~ When I checked the pod "heketi-storage-1-HASH" the following message is displayed: ~~~ [negroni] Started POST /blockvolumes [heketi] WARNING 2018/09/18 02:37:24 temporarily rejecting block volume request: pending block-hosting-volume found [negroni] Completed 429 Too Many Requests in 611.092µs [heketi] ERROR 2018/09/18 02:37:24 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations ~~~ This is a bit strange because I'm trying to create only one PVC and the server where this Pod is running has 14GB of free RAM. The "kubernetes.io/glusterfs" storage class works very well in the same cluster. ### Version-Release ### ocs 3.10 "independent mode" glusterfs 3.12.2 openshift v3.10.34 rhel 7.5 registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7 latest bf4115f0c8e8 10 days ago 311 MB registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7 latest 377527b85c0b 11 days ago 1.01 GB registry.access.redhat.com/rhgs3/rhgs-server-rhel7 latest 2603af579f4e 2 weeks ago 299 MB ### Info ### $ oc describe sc glusterfs-storage Name: glusterfs-storage IsDefaultClass: No Annotations: <none> Provisioner: kubernetes.io/glusterfs Parameters: resturl=http://heketi-storage.app-storage.svc:8080,restuser=admin,secretName=heketi-storage-admin-secret,secretNamespace=app-storage AllowVolumeExpansion: <unset> MountOptions: <none> ReclaimPolicy: Delete VolumeBindingMode: Immediate Events: <none> $ oc describe pvc storage-demo Name: storage-demo Namespace: app-storage StorageClass: glusterfs-storage-block Status: Pending Volume: Labels: <none> Annotations: control-plane.alpha.kubernetes.io/leader={"holderIdentity":"dc32c5bb-bae7-11e8-b957-0a580a830024","leaseDurationSeconds":15,"acquireTime":"2018-09-18T02:21:48Z","renewTime":"2018-09-18T02:40:56Z","lea... volume.beta.kubernetes.io/storage-class=glusterfs-storage-block volume.beta.kubernetes.io/storage-provisioner=gluster.org/glusterblock Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 18m (x12 over 19m) gluster.org/glusterblock dc32c5bb-bae7-11e8-b957-0a580a830024 Failed to provision volume with StorageClass "glusterfs-storage-block": failed to create volume: heketi block volume creation failed: [heketi] failed to create volume: Server busy. Retry operation later. Normal ExternalProvisioning 4m (x511 over 19m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "gluster.org/glusterblock" or manually created by system administrator Normal Provisioning 4m (x42 over 19m) gluster.org/glusterblock dc32c5bb-bae7-11e8-b957-0a580a830024 External provisioner is provisioning volume for claim "app-storage/storage-demo" #inventory openshift_storage_glusterfs_namespace=app-storage openshift_storage_glusterfs_storageclass=true openshift_storage_glusterfs_storageclass_default=false openshift_storage_glusterfs_block_deploy=true openshift_storage_glusterfs_block_host_vol_size=100 openshift_storage_glusterfs_block_storageclass=true openshift_storage_glusterfs_block_storageclass_default=false openshift_storage_glusterfs_is_native=false openshift_storage_glusterfs_heketi_is_native=true openshift_storage_glusterfs_heketi_executor=ssh openshift_storage_glusterfs_heketi_ssh_port=22 openshift_storage_glusterfs_heketi_ssh_user=root openshift_storage_glusterfs_heketi_ssh_sudo=false openshift_storage_glusterfs_heketi_ssh_keyfile="/root/.ssh/id_rsa" [glusterfs] ocs-0.mmagnani.lab glusterfs_ip=192.168.0.30 glusterfs_devices='[ "/dev/sdb" ]' ocs-1.mmagnani.lab glusterfs_ip=192.168.0.31 glusterfs_devices='[ "/dev/sdb" ]' ocs-2.mmagnani.lab glusterfs_ip=192.168.0.32 glusterfs_devices='[ "/dev/sdb" ]'
*** $ oc describe sc glusterfs-storage-block Name: glusterfs-storage-block IsDefaultClass: No Annotations: <none> Provisioner: gluster.org/glusterblock Parameters: chapauthenabled=true,hacount=3,restsecretname=heketi-storage-admin-secret-block,restsecretnamespace=app-storage,resturl=http://heketi-storage.app-storage.svc:8080,restuser=admin AllowVolumeExpansion: <unset> MountOptions: <none> ReclaimPolicy: Delete VolumeBindingMode: Immediate Events: <none>
Mauricio, The provisioner would get the "Server handling too many operations" error if heketi is performing 8 other operations. However, provisioner should retry and eventually succeed in creating the volume. Thanks for providing the image ids and I have verified that the volmanager and block-provisioner images are the latest OCS 3.10 images. Provide the following information for us to proceed: 1. tell us if the PV got created eventually, if yes then it is working as expected. 2. oc rsh into heketi pod and get the db dump. You can do so using the command "heketi-cli --user admin --secrete $HEKETI_ADMIN_KEY db dump > db_dump". Use oc rsync or any other command to get the file to a node from where you could attach it to the bug.
Hi Raghavendra, sorry for the delay in getting back to you. >>1. tell us if the PV got created eventually, if yes then it is working as expected. Currently it is impossible to create any PV using Gluster-block. >>2. oc rsh into heketi pod and get the db dump. You can do so using the command "heketi-cli --user admin --secrete $HEKETI_ADMIN_KEY db dump > db_dump". Use oc rsync or any other command to get the file to a node from where you could attach it to the bug. Sure! Ps: I tested with the openshift v3.10.45 version and the behavior is the same.
Created attachment 1486129 [details] heketi dump
Created attachment 1486130 [details] log heketi pod
Created attachment 1486131 [details] pv
Hey After seven hours is still in "Pending".
Created attachment 1486190 [details] pv7
Created attachment 1486191 [details] dump7
Created attachment 1486192 [details] log7
Mauricio, I experienced a very similar problem in an OpenShift Origin 3.10 cluster with an Independent Gluster deployment. I discovered that during the install, the ansible playbook for deploying Gluster is not starting the gluster-blockd service on the Gluster nodes. Check to see if gluster-blockd is running on your Gluster nodes: systemctl status gluster-blockd If it's not installed, you will need to install Gluster-block. Systemctl enable gluster-blockd systemctl start gluster-blockd Then scale Gluster-block-provisioner and Heketi pods down to zero, and then scale back up. Hopefully it will now be working.
Closing as suggested in comment 18.