Bug 2021427 - Must Gather, some ceph commands return error [exit code different from 0]
Summary: Must Gather, some ceph commands return error [exit code different from 0]
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: must-gather
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Mudit Agarwal
QA Contact: Raz Tamir
URL:
Whiteboard:
Depends On: 2014849
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-09 08:08 UTC by Oded
Modified: 2021-12-06 11:03 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-06 11:03:40 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github red-hat-storage ocs-operator pull 1379 0 None Merged must-gather: remove invalid ceph commands 2021-11-29 11:22:53 UTC

Description Oded 2021-11-09 08:08:16 UTC
Description of problem (please be detailed as possible and provide log
snippests):
Some ceph commands return error [exit code different from 0]. 

Version of all relevant components (if applicable):
Provider:AWS
OCP Version:4.8.0-0.nightly-2021-11-06-151235
OCS Version:4.8.4
ceph version:
{
    "mon": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 3
    },
    "mgr": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 1
    },
    "osd": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 3
    },
    "mds": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 2
    },
    "overall": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 9
    }
}


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Collect mg
oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.8

2.Check content of ceph files [some ceph commands return error]
/ceph/logs/gather-ceph balancer dump-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20balancer%20dump-debug.log

/ceph/logs/gather-ceph balancer dump-json-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20balancer%20dump-json-debug.log

/ceph/logs/gather-ceph osd drain status-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20osd%20drain%20status-debug.log

/ceph/logs/gather-ceph osd drain status-json-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20osd%20drain%20status-json-debug.log

/ceph/logs/gather-ceph pool autoscale-status-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20pool%20autoscale-status-debug.log

/ceph/logs/gather-ceph pool autoscale-status-json-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20pool%20autoscale-status-json-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-12c39163-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27afd914-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27afd914-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-snap-schedule-list-ocs-storagecluster-cephblockpool-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-snap-schedule-list-ocs-storagecluster-cephblockpool-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27b961c1-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27b961c1-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27c2d1f3-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27c2d1f3-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-280ab9ac-40a6-11ec-80d7-0a580a83001c-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-280ab9ac-40a6-11ec-80d7-0a580a83001c-debug.log

/ceph/logs/gather-rbd-mirror-pool-status-ocs-storagecluster-cephblockpool-debug.log
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-pool-status-ocs-storagecluster-cephblockpool-debug.log


Actual results:
Some ceph commands return error

Expected results:
All commands should be valid. And if not they should be removed from the MG command list


Additional info:

Comment 2 Mudit Agarwal 2021-11-30 11:37:07 UTC
I don't see any reason to fix it in z-stream, must-gather doesn't fail only these commands fail which is expected.
We should close this BZ it as Next release, fix is already there in 4.9.0


Note You need to log in before you can comment on or make changes to this bug.