Bug 2066906

Summary: ceph command failed with error: RADOS permission error (error connecting to the cluster) on consumer cluster
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: suchita <sgatfane>
Component: rookAssignee: Subham Rai <srai>
Status: CLOSED NOTABUG QA Contact: Neha Berry <nberry>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.10CC: ebenahar, madam, mbukatov, mmuench, muagarwa, nberry, ocs-bugs, odf-bz-bot, rcyriac, shan, srai, tnielsen
Target Milestone: ---Flags: shan: needinfo? (nberry)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-12 15:26:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description suchita 2022-03-22 18:11:34 UTC 
Description of problem (please be detailed as possible and provide log
snippests):

 In the consumer Cluster,  
rook ceph tool pod is running however it failed for ceph command with error: 
sh-4.4$ ceph -s
[errno 1] RADOS permission error (error connecting to the cluster)


rook-ceph-tools-86c9fb5d54-nrj4f                                  1/1     Running     0               4h27m


Version of all relevant components (if applicable):

$ oc get csv
NAME                                      DISPLAY                       VERSION           REPLACES                                  PHASE
mcg-operator.v4.10.0                      NooBaa Operator               4.10.0                                                      Succeeded
ocs-operator.v4.10.0                      OpenShift Container Storage   4.10.0                                                      Succeeded
ocs-osd-deployer.v2.0.0                   OCS OSD Deployer              2.0.0                                                       Succeeded
odf-csi-addons-operator.v4.10.0           CSI Addons                    4.10.0                                                      Succeeded
odf-operator.v4.10.0                      OpenShift Data Foundation     4.10.0                                                      Succeeded
ose-prometheus-operator.4.8.0             Prometheus Operator           4.8.0                                                       Succeeded
route-monitor-operator.v0.1.406-54ff884   Route Monitor Operator        0.1.406-54ff884   route-monitor-operator.v0.1.404-e29b74b   Succeeded


ocs-operator.v4.10.0 full version: "4.10.0-197"

=========ceph versions========
{
    "mon": {
        "ceph version 16.2.7-76.el8cp (f4d6ada772570ae8b05c62ad79e222fbd3f04188) pacific (stable)": 3
    },
    "mgr": {
        "ceph version 16.2.7-76.el8cp (f4d6ada772570ae8b05c62ad79e222fbd3f04188) pacific (stable)": 1
    },
    "osd": {
        "ceph version 16.2.7-76.el8cp (f4d6ada772570ae8b05c62ad79e222fbd3f04188) pacific (stable)": 3
    },
    "mds": {
        "ceph version 16.2.7-76.el8cp (f4d6ada772570ae8b05c62ad79e222fbd3f04188) pacific (stable)": 2
    },
    "overall": {
        "ceph version 16.2.7-76.el8cp (f4d6ada772570ae8b05c62ad79e222fbd3f04188) pacific (stable)": 9
    }
}





Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue be reproducible?
1/1

Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Create provider-consumer cluster using add-on (ref: https://docs.google.com/document/d/1ehNBscWgLGNYqnnZUp6RPnkR9ByYU69BgXvr_z2n5sE/edit#)
2.check for rook ceph tool pod and execute ceph command in pod


Actual results:
oc rsh pod/rook-ceph-tools-86c9fb5d54-nrj4f
sh-4.4$ ceph -s
[errno 1] RADOS permission error (error connecting to the cluster)

Expected results:
All ceph command should work as expected

Additional info:
workaround: 
Add [client.admin in consumer tool box: 
Copy [client.admin] from /etc/ceph/keyring from toolbox pod in provider and add it to /etc/ceph/keyring of consumer toolbox
Comment 4 Mudit Agarwal 2022-03-23 14:09:50 UTC 
Removing the blocker flag.
Comment 8 Travis Nielsen 2022-04-12 15:26:14 UTC 
Please reopen if more info is available or something else needed
Comment 9 Martin Bukatovic 2022-05-09 11:56:06 UTC 
(In reply to Travis Nielsen from comment #8)
> Please reopen if more info is available or something else needed

I created a separate RFE BZ to make it clear what is exactly expected here:

BZ 2083126