Description of problem (please be detailed as possible and provide log snippests): Some ceph commands return error [exit code different from 0]. Version of all relevant components (if applicable): Provider:AWS OCP Version:4.8.0-0.nightly-2021-11-06-151235 OCS Version:4.8.4 ceph version: { "mon": { "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 1 }, "osd": { "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 3 }, "mds": { "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 2 }, "overall": { "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 9 } } Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1.Collect mg oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.8 2.Check content of ceph files [some ceph commands return error] /ceph/logs/gather-ceph balancer dump-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20balancer%20dump-debug.log /ceph/logs/gather-ceph balancer dump-json-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20balancer%20dump-json-debug.log /ceph/logs/gather-ceph osd drain status-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20osd%20drain%20status-debug.log /ceph/logs/gather-ceph osd drain status-json-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20osd%20drain%20status-json-debug.log /ceph/logs/gather-ceph pool autoscale-status-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20pool%20autoscale-status-debug.log /ceph/logs/gather-ceph pool autoscale-status-json-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-ceph%20pool%20autoscale-status-json-debug.log /ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log /ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-12c39163-40a6-11ec-80d7-0a580a83001c-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-282990f6-40a6-11ec-80d7-0a580a83001c-debug.log /ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27afd914-40a6-11ec-80d7-0a580a83001c-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27afd914-40a6-11ec-80d7-0a580a83001c-debug.log /ceph/logs/gather-rbd-mirror-snap-schedule-list-ocs-storagecluster-cephblockpool-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-snap-schedule-list-ocs-storagecluster-cephblockpool-debug.log /ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27b961c1-40a6-11ec-80d7-0a580a83001c-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27b961c1-40a6-11ec-80d7-0a580a83001c-debug.log /ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27c2d1f3-40a6-11ec-80d7-0a580a83001c-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-27c2d1f3-40a6-11ec-80d7-0a580a83001c-debug.log /ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-280ab9ac-40a6-11ec-80d7-0a580a83001c-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-image-status-ocs-storagecluster-cephblockpool-csi-vol-280ab9ac-40a6-11ec-80d7-0a580a83001c-debug.log /ceph/logs/gather-rbd-mirror-pool-status-ocs-storagecluster-cephblockpool-debug.log http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr5037-b1771/jnk-pr5037-b1771_20211108T142241/logs/failed_testcase_ocs_logs_1636384919/test_must_gather%5bCEPH%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-0842b16fe1cc06319a1d580521acfa006a3a361ef9feac42136f9cc95a974e3d/ceph/logs/gather-rbd-mirror-pool-status-ocs-storagecluster-cephblockpool-debug.log Actual results: Some ceph commands return error Expected results: All commands should be valid. And if not they should be removed from the MG command list Additional info:
I don't see any reason to fix it in z-stream, must-gather doesn't fail only these commands fail which is expected. We should close this BZ it as Next release, fix is already there in 4.9.0