Fedora Account System
Red Hat Associate
Red Hat Customer
*** Bug 1851374 has been marked as a duplicate of this bug. ***
The object store is not getting created because of the PG limit. The rook operator log shows: 2020-06-26 10:30:39.906986 I | exec: Running command: ceph osd pool create ocs-storagecluster-cephobjectstore.rgw.buckets.index 0 replicated ocs-storagecluster-cephobjectstore.rgw.buckets.index --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json --out-file /tmp/875991478 2020-06-26 10:30:40.952434 I | exec: Error ERANGE: pg_num 32 size 3 would mean 768 total pgs, which exceeds max 750 (mon_max_pg_per_osd 250 * num_in_osds 3) 2020-06-26 10:30:40.952568 E | op-object: failed to create or update object store ocs-storagecluster-cephobjectstore. failed to create pools: failed to create object pools: failed to create metadata pools: failed to create pool ocs-storagecluster-cephobjectstore.rgw.buckets.index for object store ocs-storagecluster-cephobjectstore This is due to a change in Ceph 14.2.8 when the default PG count was set for all pools to 32. The expectation is that the rgw metadata pools should only set 8 PGs by default, which will keep us well below the max PG count. However, these changes had not been backported to 4.4. We are just now hitting it since the base image is 14.2.8 or newer. We need to backport these two PRs to release-4.4: https://github.com/rook/rook/pull/5096 https://github.com/rook/rook/pull/5489
A third related backport was also needed to go along with #5489: https://github.com/rook/rook/pull/5177 The fix is verified with this change based on release-4.4. A replica pool, filesystem, and object store were all created successfully. Before fixing, I also verified that the issue reprod locally as I was not able to create both cephfs and rgw because of the PG count. A PR has been merged for the downstream release-4.4 rook branch. https://github.com/openshift/rook/pull/73
Installation for vSphere passed and RGW pod is running > csv $ oc get csv NAME DISPLAY VERSION REPLACES PHASE lib-bucket-provisioner.v1.0.0 lib-bucket-provisioner 1.0.0 Succeeded ocs-operator.v4.4.1-465.ci OpenShift Container Storage 4.4.1-465.ci Succeeded [vavuthu@localhost rem]$ > pods $ oc get pods -n openshift-storage NAME READY STATUS RESTARTS AGE csi-cephfsplugin-8k6m8 3/3 Running 0 56m csi-cephfsplugin-92tng 3/3 Running 0 56m csi-cephfsplugin-nrp29 3/3 Running 0 56m csi-cephfsplugin-provisioner-6fdd566b4d-2kgfc 5/5 Running 0 56m csi-cephfsplugin-provisioner-6fdd566b4d-kd4mq 5/5 Running 0 56m csi-rbdplugin-jv8bk 3/3 Running 0 56m csi-rbdplugin-provisioner-7d784b9d7c-bwlgr 5/5 Running 0 56m csi-rbdplugin-provisioner-7d784b9d7c-fwdcb 5/5 Running 0 56m csi-rbdplugin-pz9c7 3/3 Running 0 56m csi-rbdplugin-xq97h 3/3 Running 0 56m lib-bucket-provisioner-5b9cb4f848-zrvdw 1/1 Running 0 58m noobaa-core-0 1/1 Running 0 52m noobaa-db-0 1/1 Running 0 52m noobaa-endpoint-58ff4bdfd6-2ll9d 1/1 Running 0 50m noobaa-operator-6f9db4df44-jzrv4 1/1 Running 0 57m ocs-operator-7fdf85b7b4-bg6cl 1/1 Running 0 57m rook-ceph-crashcollector-compute-0-7fbb7b4b76-vxc22 1/1 Running 0 52m rook-ceph-crashcollector-compute-1-6d4656d5f5-v2dvs 1/1 Running 0 52m rook-ceph-crashcollector-compute-2-88cb7fddb-jkbl4 1/1 Running 0 52m rook-ceph-drain-canary-compute-0-6c7798b9b8-br59p 1/1 Running 0 52m rook-ceph-drain-canary-compute-1-7499bc7d6f-7pb22 1/1 Running 0 52m rook-ceph-drain-canary-compute-2-6b8dd759db-k9lzn 1/1 Running 0 52m rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-7794998dthwgm 1/1 Running 0 52m rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-5c6964ccxp5fk 1/1 Running 0 52m rook-ceph-mgr-a-7ddcffcb68-nj576 1/1 Running 0 53m rook-ceph-mon-a-7766b47887-txsjf 1/1 Running 0 55m rook-ceph-mon-b-669cc6c5d4-k5cgm 1/1 Running 0 54m rook-ceph-mon-c-689976cc99-gxz2l 1/1 Running 0 54m rook-ceph-operator-68557c549b-72h8w 1/1 Running 0 57m rook-ceph-osd-0-544cf5448b-cvc9p 1/1 Running 0 52m rook-ceph-osd-1-7ffcfdd9d5-pdqpq 1/1 Running 0 52m rook-ceph-osd-2-f95cffd6d-j2nbv 1/1 Running 0 52m rook-ceph-osd-prepare-ocs-deviceset-0-0-c58d7-9kfqd 0/1 Completed 0 53m rook-ceph-osd-prepare-ocs-deviceset-1-0-bx9h5-8cldc 0/1 Completed 0 53m rook-ceph-osd-prepare-ocs-deviceset-2-0-wxcj7-vksm4 0/1 Completed 0 53m rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-79b8bc7b9xk5 1/1 Running 0 51m rook-ceph-tools-b7789c95-9cphq 1/1 Running 0 52m > [vavuthu@localhost rem]$ oc get backingstore -A NAMESPACE NAME TYPE PHASE AGE openshift-storage noobaa-default-backing-store s3-compatible Ready 50m [vavuthu@localhost rem]$ > $ oc get bucketclass -A NAMESPACE NAME PLACEMENT PHASE AGE openshift-storage noobaa-default-bucket-class map[tiers:[map[backingStores:[noobaa-default-backing-store]]]] Ready 51m [vavuthu@localhost rem]$ Deploymnet job: https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/9251/ Tier1 is in progress: https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/9260/
VmWare tier1 execution running here: https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/9260/console And deployment already passed
(In reply to Travis Nielsen from comment #6) > This is due to a change in Ceph 14.2.8 when the default PG count was set for > all pools to 32. The expectation is that the rgw metadata pools should only > set 8 PGs by default, which will keep us well below the max PG count. > However, these changes had not been backported to 4.4. We are just now > hitting it since the base image is 14.2.8 or newer. If I get it right, this regression was introduced by the new ceph image via RHCS 4.1, and was not there in OCS 4.4.0 or earlier. If this is true, we don't need to put a doc text for this BZ. @Travis, can you confirm?
@Michael Correct, this is a result of moving to RHCS 4.1. Without that change, this issue would not have arisen in 4.4.1.
This issue was fixed in OCS 4.5 via these PRs and all of these PRs are already in release-4.5: https://github.com/rook/rook/pull/5098 (instead of https://github.com/rook/rook/pull/5096 which was specific to release 1.2) https://github.com/rook/rook/pull/5489 https://github.com/rook/rook/pull/5177 So, no new code change is required for OCS4.5. @michael do we still need to clone this BZ for OCS4.5?
Even though the patches are already in 4.5, maybe we should create a BZ for 4.5, also because one patch is slightly different, to explicitly verify the bugfix.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2830