Bug 1955546

Summary:	[GSS][External Mode] Deployment failure while configuring Noobaa due to Ceph objectstore user not ready
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	Karun Josy <kjosy>
Component:	rook	Assignee:	Sébastien Han <shan>
Status:	VERIFIED ---	QA Contact:	Sidhant Agrawal <sagrawal>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.6	CC:	dguthrie, jcrumple, jthottan, muagarwa, sagrawal, shan, tdesala, tnielsen
Target Milestone:	---
Target Release:	OCS 4.8.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	4.8.0-416.ci	Doc Type:	Bug Fix
Doc Text:	When the ODF Ceph version and RHCS external version differ, user creation (from CephObjectStoreUser CRS) commands are failing. This is due to a mismatch in the external RADOS Gateway server version and the `radosgw-admin` binary version on the Rook-Ceph Operator. Now, Rook-Ceph is relying on the RADOS Gateway Admin Operational API to perform user creation, so regardless of the version difference, operations will succeed.	Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 7 Travis Nielsen 2021-05-04 18:37:22 UTC

This seems like a compatibility issue between radosgw-admin in the rook operator and the version of rgw in the external cluster. However, the external ceph version v14.2.11 is recent so I wouldn't expect an incompatibility.

2021-04-22T16:31:57.337314318Z 2021-04-22 16:31:57.337243 E | ceph-object-store-user-controller: failed to reconcile failed to create/update object store user "noobaa-ceph-objectstore-user": failed to create ceph object user "noobaa-ceph-objectstore-user". error code 1: failed to create s3 user: signal: segmentation fault (core dumped)

@Jiffin can you take a look?

Comment 8 David Guthrie 2021-05-04 20:44:11 UTC

Hi Travis,

The customer provided external data about the users being used by the root-ceph operator, I hope this helps.

there are 3 users.


[core@openshift-master-0 logs]$ oc get secret rook-ceph-external-cluster-details -o json |jq .data.external_cluster_details -r |base64 -d |jq  |grep userID
      "userID": "client.ocs"
      "userID": "csi-rbd-node"
      "userID": "csi-rbd-provisioner"
only 1 of those exists in ceph


[root@kzn-mon1 ~]# ceph auth get client.ocs
exported keyring for client.ocs
[client.ocs]
    key = AQCudoBgVilhJhAA+RLKV2WBTA4vBS8zjycr7w==
    caps mgr = "allow command config"
    caps mon = "allow r, allow command quorum_status, allow command version"
    caps osd = "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"
[root@kzn-mon1 ~]# ceph auth get csi-rbd-node
Error EINVAL: invalid entity_auth csi-rbd-node
[root@kzn-mon1 ~]# ceph auth get csi-rbd-provisioner
Error EINVAL: invalid entity_auth csi-rbd-provisioner

Comment 17 Sébastien Han 2021-05-12 13:19:40 UTC

Ashish, let's go ahead with that hotfix for now, but the plan is to fix this with 4.8.

Comment 18 Travis Nielsen 2021-05-17 17:52:11 UTC

Removing needsinfo since already being addressed by Seb

Comment 20 Sébastien Han 2021-05-27 15:33:49 UTC

Hello Jamey, this BZ is getting the attention it deserves. I marked it as a blocker for 4.8 and the engineering is committed to delivering this in OCS 4.8.
Thanks.

Comment 25 Sébastien Han 2021-06-22 08:10:12 UTC

Sidhant, did you create OB/OBC too? If it works then we can move to VERIFIED.
Thanks.