Bug 1955546 - [GSS][External Mode] Deployment failure while configuring Noobaa due to Ceph objectstore user not ready
Summary: [GSS][External Mode] Deployment failure while configuring Noobaa due to Ceph ...
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: rook
Version: 4.6
Hardware: All
OS: Linux
high
high
Target Milestone: ---
: OCS 4.8.0
Assignee: Sébastien Han
QA Contact: Sidhant Agrawal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-30 11:02 UTC by Karun Josy
Modified: 2023-08-03 08:29 UTC (History)
8 users (show)

Fixed In Version: 4.8.0-416.ci
Doc Type: Bug Fix
Doc Text:
When the ODF Ceph version and RHCS external version differ, user creation (from CephObjectStoreUser CRS) commands are failing. This is due to a mismatch in the external RADOS Gateway server version and the `radosgw-admin` binary version on the Rook-Ceph Operator. Now, Rook-Ceph is relying on the RADOS Gateway Admin Operational API to perform user creation, so regardless of the version difference, operations will succeed.
Clone Of:
Environment:
Last Closed:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ceph go-ceph pull 496 0 None open rgwadmin: add support for RadosGW Admin Ops API 2021-05-26 12:11:11 UTC
Github red-hat-storage ocs-ci pull 2540 0 None closed OCS deployment with External RHCS cluster 2021-06-30 14:29:17 UTC
Github rook rook pull 7998 0 None open ceph: stop using radosgw-admin CLI for s3 user management 2021-05-27 15:33:44 UTC

Comment 7 Travis Nielsen 2021-05-04 18:37:22 UTC
This seems like a compatibility issue between radosgw-admin in the rook operator and the version of rgw in the external cluster. However, the external ceph version v14.2.11 is recent so I wouldn't expect an incompatibility.

2021-04-22T16:31:57.337314318Z 2021-04-22 16:31:57.337243 E | ceph-object-store-user-controller: failed to reconcile failed to create/update object store user "noobaa-ceph-objectstore-user": failed to create ceph object user "noobaa-ceph-objectstore-user". error code 1: failed to create s3 user: signal: segmentation fault (core dumped)

@Jiffin can you take a look?

Comment 8 David Guthrie 2021-05-04 20:44:11 UTC
Hi Travis,

The customer provided external data about the users being used by the root-ceph operator, I hope this helps.

there are 3 users.


[core@openshift-master-0 logs]$ oc get secret rook-ceph-external-cluster-details -o json |jq .data.external_cluster_details -r |base64 -d |jq  |grep userID
      "userID": "client.ocs"
      "userID": "csi-rbd-node"
      "userID": "csi-rbd-provisioner"
only 1 of those exists in ceph


[root@kzn-mon1 ~]# ceph auth get client.ocs
exported keyring for client.ocs
[client.ocs]
    key = AQCudoBgVilhJhAA+RLKV2WBTA4vBS8zjycr7w==
    caps mgr = "allow command config"
    caps mon = "allow r, allow command quorum_status, allow command version"
    caps osd = "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"
[root@kzn-mon1 ~]# ceph auth get csi-rbd-node
Error EINVAL: invalid entity_auth csi-rbd-node
[root@kzn-mon1 ~]# ceph auth get csi-rbd-provisioner
Error EINVAL: invalid entity_auth csi-rbd-provisioner

Comment 17 Sébastien Han 2021-05-12 13:19:40 UTC
Ashish, let's go ahead with that hotfix for now, but the plan is to fix this with 4.8.

Comment 18 Travis Nielsen 2021-05-17 17:52:11 UTC
Removing needsinfo since already being addressed by Seb

Comment 20 Sébastien Han 2021-05-27 15:33:49 UTC
Hello Jamey, this BZ is getting the attention it deserves. I marked it as a blocker for 4.8 and the engineering is committed to delivering this in OCS 4.8.
Thanks.

Comment 25 Sébastien Han 2021-06-22 08:10:12 UTC
Sidhant, did you create OB/OBC too? If it works then we can move to VERIFIED.
Thanks.


Note You need to log in before you can comment on or make changes to this bug.