Bug 2021427

Summary: Must Gather, some ceph commands return error [exit code different from 0]
Product: [Red Hat Storage] Red Hat OpenShift Container Storage Reporter: Oded <oviner>
Component: must-gatherAssignee: Mudit Agarwal <muagarwa>
Status: CLOSED NEXTRELEASE QA Contact: Raz Tamir <ratamir>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.8CC: kramdoss, mmuench, muagarwa, ocs-bugs, sabose, sheggodu
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-06 11:03:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2014849    
Bug Blocks:    

Description Oded 2021-11-09 08:08:16 UTC
Description of problem (please be detailed as possible and provide log
snippests):
Some ceph commands return error [exit code different from 0]. 

Version of all relevant components (if applicable):
Provider:AWS
OCP Version:4.8.0-0.nightly-2021-11-06-151235
OCS Version:4.8.4
ceph version:
{
    "mon": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 3
    },
    "mgr": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 1
    },
    "osd": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 3
    },
    "mds": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 2
    },
    "overall": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 9
    }
}


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Collect mg
oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.8

2.Check content of ceph files [some ceph commands return error]
/ceph/logs/gather-ceph balancer dump-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20balancer%20dump-debug.log

/ceph/logs/gather-ceph balancer dump-json-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20balancer%20dump-json-debug.log

/ceph/logs/gather-ceph osd drain status-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20osd%20drain%20status-debug.log

/ceph/logs/gather-ceph osd drain status-json-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20osd%20drain%20status-json-debug.log

/ceph/logs/gather-ceph pool autoscale-status-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20pool%20autoscale-status-debug.log

/ceph/logs/gather-ceph pool autoscale-status-json-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20pool%20autoscale-status-json-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-12c39163-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27afd914-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27afd914-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-snap-schedule-list-ocs-storagecluster-cephblockpool-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-snap-schedule-list-ocs-storagecluster-cephblockpool-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27b961c1-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27b961c1-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27c2d1f3-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27c2d1f3-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-280ab9ac-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-280ab9ac-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-pool-status-ocs-storagecluster-cephblockpool-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-pool-status-ocs-storagecluster-cephblockpool-debug.log


Actual results:
Some ceph commands return error

Expected results:
All commands should be valid. And if not they should be removed from the MG command list


Additional info:

Comment 2 Mudit Agarwal 2021-11-30 11:37:07 UTC
I don't see any reason to fix it in z-stream, must-gather doesn't fail only these commands fail which is expected.
We should close this BZ it as Next release, fix is already there in 4.9.0