Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem (please be detailed as possible and provide log snippests): Creating RBD PVC using thick provisioning enabled storage class is failing. Tried to create a PVC of size 5 GiB. The below errors were seen. Error logs from csi-provisioner container logs: I0308 10:16:00.065131 1 controller.go:1335] provision "default/pvc-thick-filesystem-rwo" class "ocs-storagecluster-ceph-rbd-thick": started I0308 10:16:00.065435 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-thick-filesystem-rwo", UID:"a26ae73b-b41d-4aad-88ea-69992ab35a51", APIVersion:"v1", ResourceVersion:"55957", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/pvc-thick-filesystem-rwo" I0308 10:16:00.070695 1 controller.go:731] CreateVolumeRequest name:"pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51" capacity_range:<required_bytes:5368709120 > volume_capabilities:<mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"clusterID" value:"openshift-storage" > parameters:<key:"csi.storage.k8s.io/pv/name" value:"pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51" > parameters:<key:"csi.storage.k8s.io/pvc/name" value:"pvc-thick-filesystem-rwo" > parameters:<key:"csi.storage.k8s.io/pvc/namespace" value:"default" > parameters:<key:"imageFeatures" value:"layering" > parameters:<key:"imageFormat" value:"2" > parameters:<key:"pool" value:"ocs-storagecluster-cephblockpool" > parameters:<key:"thickProvision" value:"true" > secrets:<key:"userID" value:"csi-rbd-provisioner" > secrets:<key:"userKey" value:"AQBl7EVg9whcBRAA8aEIae4kEFXCYcbbahKyGw==" > I0308 10:16:00.070811 1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume I0308 10:16:00.070821 1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51","csi.storage.k8s.io/pvc/name":"pvc-thick-filesystem-rwo","csi.storage.k8s.io/pvc/namespace":"default","imageFeatures":"layering","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool","thickProvision":"true"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} I0308 10:16:00.666683 1 connection.go:185] GRPC response: {} I0308 10:16:00.666754 1 connection.go:186] GRPC error: rpc error: code = Internal desc = failed to thick provision image: rbd: ret=-22, Invalid argument I0308 10:16:00.666795 1 controller.go:752] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = failed to thick provision image: rbd: ret=-22, Invalid argument I0308 10:16:00.666867 1 controller.go:1102] Final error received, removing PVC a26ae73b-b41d-4aad-88ea-69992ab35a51 from claims in progress W0308 10:16:00.666898 1 controller.go:961] Retrying syncing claim "a26ae73b-b41d-4aad-88ea-69992ab35a51", failure 9 E0308 10:16:00.666937 1 controller.go:984] error syncing claim "a26ae73b-b41d-4aad-88ea-69992ab35a51": failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd-thick": rpc error: code = Internal desc = failed to thick provision image: rbd: ret=-22, Invalid argument I0308 10:16:00.667488 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-thick-filesystem-rwo", UID:"a26ae73b-b41d-4aad-88ea-69992ab35a51", APIVersion:"v1", ResourceVersion:"55957", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd-thick": rpc error: code = Internal desc = failed to thick provision image: rbd: ret=-22, Invalid argument I0308 10:16:04.285851 1 leaderelection.go:273] successfully renewed lease openshift-storage/openshift-storage-rbd-csi-ceph-com I0308 10:16:09.301874 1 leaderelection.go:273] successfully renewed lease openshift-storage/openshift-storage-rbd-csi-ceph-com I0308 10:16:14.315737 1 leaderelection.go:273] successfully renewed lease openshift-storage/openshift-storage-rbd-csi-ceph-com I0308 10:16:17.657497 1 reflector.go:530] github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:82: Watch close - *v1.PersistentVolumeClaim total 14 items received I0308 10:16:19.330685 1 leaderelection.go:273] successfully renewed lease openshift-storage/openshift-storage-rbd-csi-ceph-com Error logs from csi-rbdplugin container: I0308 10:35:18.398290 1 utils.go:132] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 GRPC call: /csi.v1.Controller/CreateVolume I0308 10:35:18.398509 1 utils.go:133] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51","csi.storage.k8s.io/pvc/name":"pvc-thick-filesystem-rwo","csi.storage.k8s.io/pvc/namespace":"default","imageFeatures":"layering","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool","thickProvision":"true"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} I0308 10:35:18.398751 1 rbd_util.go:869] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 setting disableInUseChecks on rbd volume to: false I0308 10:35:18.400341 1 omap.go:84] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[] I0308 10:35:18.407524 1 omap.go:148] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51:fa120fe2-7ff9-11eb-af10-0a580a83001e]) I0308 10:35:18.410807 1 omap.go:148] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.fa120fe2-7ff9-11eb-af10-0a580a83001e"): map[csi.imagename:csi-vol-fa120fe2-7ff9-11eb-af10-0a580a83001e csi.volname:pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 csi.volume.owner:default]) I0308 10:35:18.410837 1 rbd_journal.go:436] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 generated Volume ID (0001-0011-openshift-storage-0000000000000001-fa120fe2-7ff9-11eb-af10-0a580a83001e) and image name (csi-vol-fa120fe2-7ff9-11eb-af10-0a580a83001e) for request name (pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51) I0308 10:35:18.410893 1 rbd_util.go:204] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 rbd: create ocs-storagecluster-cephblockpool/csi-vol-fa120fe2-7ff9-11eb-af10-0a580a83001e size 5120M (features: [layering]) using mon 172.30.250.69:6789,172.30.173.226:6789,172.30.185.202:6789 I0308 10:35:18.466027 1 rbd_util.go:415] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 rbd: delete csi-vol-fa120fe2-7ff9-11eb-af10-0a580a83001e using mon 172.30.250.69:6789,172.30.173.226:6789,172.30.185.202:6789, pool ocs-storagecluster-cephblockpool I0308 10:35:18.505031 1 rbd_util.go:386] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 executing [rbd task add trash remove ocs-storagecluster-cephblockpool/37908b26eb01 --id csi-rbd-provisioner --keyfile=/tmp/csi/keys/keyfile-942624165 -m 172.30.250.69:6789,172.30.173.226:6789,172.30.185.202:6789] for image (csi-vol-fa120fe2-7ff9-11eb-af10-0a580a83001e) using mon 172.30.250.69:6789,172.30.173.226:6789,172.30.185.202:6789, pool ocs-storagecluster-cephblockpool I0308 10:35:18.928546 1 cephcmds.go:59] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 command succeeded: ceph [rbd task add trash remove ocs-storagecluster-cephblockpool/37908b26eb01 --id csi-rbd-provisioner --keyfile=***stripped*** -m 172.30.250.69:6789,172.30.173.226:6789,172.30.185.202:6789] E0308 10:35:18.928697 1 controllerserver.go:491] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 failed to create volume: failed to thick provision image: rbd: ret=-22, Invalid argument I0308 10:35:18.937647 1 omap.go:118] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51] E0308 10:35:18.937753 1 utils.go:136] ID: 133 Req-ID: pvc-a26ae73b-b41d-4aad-88ea-69992ab35a51 GRPC error: rpc error: code = Internal desc = failed to thick provision image: rbd: ret=-22, Invalid argument I0308 10:35:48.097732 1 utils.go:132] ID: 134 GRPC call: /csi.v1.Identity/Probe The above errors occurred while creating a PVC of size 5 GiB. On further investigation, it is observed that PVC with requested capacity upto 1024 MiB is reaching Bound state. PVC provisioning with 1025 MiB size got failed. Storage class yaml: $ oc get sc ocs-storagecluster-ceph-rbd-thick -o yaml allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: description: Provides RWO Filesystem volumes, and RWO and RWX Block volumes with thick provisioning creationTimestamp: "2021-03-08T10:10:26Z" managedFields: - apiVersion: storage.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:allowVolumeExpansion: {} f:metadata: f:annotations: .: {} f:description: {} f:parameters: .: {} f:clusterID: {} f:csi.storage.k8s.io/controller-expand-secret-name: {} f:csi.storage.k8s.io/controller-expand-secret-namespace: {} f:csi.storage.k8s.io/fstype: {} f:csi.storage.k8s.io/node-stage-secret-name: {} f:csi.storage.k8s.io/node-stage-secret-namespace: {} f:csi.storage.k8s.io/provisioner-secret-name: {} f:csi.storage.k8s.io/provisioner-secret-namespace: {} f:imageFeatures: {} f:imageFormat: {} f:pool: {} f:thickProvision: {} f:provisioner: {} f:reclaimPolicy: {} f:volumeBindingMode: {} manager: Mozilla operation: Update time: "2021-03-08T10:10:26Z" name: ocs-storagecluster-ceph-rbd-thick resourceVersion: "55398" uid: 25632a6c-53d7-4b4e-b54e-ebe61a3e65bf parameters: clusterID: openshift-storage csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage imageFeatures: layering imageFormat: "2" pool: ocs-storagecluster-cephblockpool thickProvision: "true" provisioner: openshift-storage.rbd.csi.ceph.com reclaimPolicy: Delete volumeBindingMode: Immediate Logs will be shared shortly ============================================================================================================= Version of all relevant components (if applicable): ocs-operator.v4.8.0-276.ci OCP 4.8.0-0.nightly-2021-03-06-055252 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes, RBD PVC is not ready for use. Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: No, this is a new feature in OCS 4.8 Steps to Reproduce: 1. Create a storage class with thick provisioning enabled by giving the parameter thickProvision: "true" as shown in the yaml given in description. 2. Use the storage class to create a PVC of size greater than 1 GiB Actual results: PVC remain in Pending state Expected results: PVC should reach Bound state Additional info:
It seems that rbd_write_same() needs to be called with a maximum of 1024MB data. We will need to iterate through the size of the image and call the function multiple times.
(In reply to Niels de Vos from comment #2) > It seems that rbd_write_same() needs to be called with a maximum of 1024MB > data. We will need to iterate through the size of the image and call the > function multiple times. Neils, Does strip unit and count adjustment based on the size could do it in one shot? may be I am wrong.
must-gather logs - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-1936388/
These are the PVCs created using thick provisioning enabled storage class. $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-thick-filesystem-rwo Pending ocs-storagecluster-ceph-rbd-thick 112m pvc-thick-filesystem-rwo-1024mib Bound pvc-72847301-753a-41af-88fa-fc09e3b1a26a 1Gi RWO ocs-storagecluster-ceph-rbd-thick 67m pvc-thick-filesystem-rwo-1025mib Pending ocs-storagecluster-ceph-rbd-thick 69m pvc-thick-filesystem-rwo-1gib Bound pvc-86b5c03e-f3bf-4b90-af40-3c974939ab16 1Gi RWO ocs-storagecluster-ceph-rbd-thick 70m pvc-thick-filesystem-rwo-5000mib Pending ocs-storagecluster-ceph-rbd-thick 72m pvc-thick-filesystem-rwo-5120mib Pending ocs-storagecluster-ceph-rbd-thick 73m pvc-thick-filesystem-rwo-smallsize Bound pvc-b2db9150-e4ba-4439-a4ed-827295034124 100Mi RWO ocs-storagecluster-ceph-rbd-thick 80m Size of pending PVCs: pvc-thick-filesystem-rwo - 5GiB pvc-thick-filesystem-rwo-1025mib - 1025 MiB pvc-thick-filesystem-rwo-5000mib - 5000 MiB pvc-thick-filesystem-rwo-5120mib - 5120 MiB
Am upstream change has been posted: https://github.com/ceph/ceph-csi/pull/1911 A test-build of the container image can be found at (pull from) quay.io/nixpanic/cephcsi:wip_rbd_writesame-limit This change has been tested with various sizes, from 50 MB to 250 GB.
(In reply to Humble Chirammal from comment #3) > Neils, Does strip unit and count adjustment based on the size could do it in > one shot? may be I am wrong. No, the current stripe-size is the optimal size to write according to RBD experts :)
https://github.com/openshift/ceph-csi/pull/29 contains the backport for downstream
Test case tests/manage/pv_services/test_rbd_thick_provisioning.py passed in all recent runs. Test run : ocs-ci results for OCS4-8-Downstream-OCP4-8-AWS-IPI-1AZ-RHCOS-3M-3W-acceptance (BUILD ID: v4.8.0-399.ci RUN ID: 1621435609) Build URL : https://storage-jenkins-csb-ceph.cloud.paas.psi.redhat.com/job/ocs-ci/396/ This test creates 1 GiB and 5 GiB thick provisioned PVCs. Verified in version: OCS 4.8.0-399.ci OCP 4.8.0-0.nightly-2021-05-19-092807
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Container Storage 4.8.0 container images bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3003