Description of problem (please be detailed as possible and provide log snippests): Currently, we dont have below commands in must-gather * rbd info * rbd snap ls * rbd snap info * ceph fs subvolume info (note this is available only 4.6 onwards * ceph fs subvolume snap ls * ceph fs subvolume snap info * ceph fs clone status It would be nice to gather these informations as part of must-gather
marking as blocker as these are new features of 4.6 and it would be helpful to get the required details in must-gather for better troubleshooting.
Created attachment 1724698 [details] oc_output from quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/namespaces/openshift-storage/oc_output
(In reply to Jilju Joy from comment #12) > Created attachment 1724698 [details] > oc_output from > quay-io-rhceph-dev-ocs-must-gather-sha256- > 0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/namespaces/ > openshift-storage/oc_output The output of commands suggested in comment #8 are present in the attachment. a) oc get volumesnapshot -A ; oc get volumesnapshot -A -o yaml Both the outputs are present in the same file quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/namespaces/openshift-storage/oc_output/volumesnapshot_-A b) oc get volumesnapshotclass ; oc get volumesnapshotclass -o yaml Both the outputs are present in the same file quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/namespaces/openshift-storage/oc_output/volumesnapshotclass c) oc get volumesnapshotcontent ; oc get volumesnapshotcontent -o yaml Both the outputs are present in the same file quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/namespaces/openshift-storage/oc_output/volumesnapshotcontent
Created attachment 1724699 [details] must_gather_commands from quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/ceph/must_gather_commands
Created attachment 1724700 [details] must_gather_commands_json_output from quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/ceph/must_gather_commands_json_output
(In reply to Madhu Rajanna from comment #0) > Description of problem (please be detailed as possible and provide log > snippests): > > Currently, we dont have below commands in must-gather > > * rbd info > * rbd snap ls The above two outputs are present in quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/ceph/must_gather_commands/rbd_snap_info.log Hi Pulkit, The file rbd_snap_info.log comes with '.log' extension. Is there any particular reason for that ? Other files in 'must_gather_commands' does not contain '.log' extension. Hi Madhu, Is it okay to have both the outputs in same file ? Please check the attachment. > * rbd snap info This is an invalid command. > * ceph fs subvolume info (note this is available only 4.6 onwards > * ceph fs subvolume snap ls > * ceph fs subvolume snap info > * ceph fs clone status Did not find the above four ceph fs command outputs in 'must_gather_commands' (quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/ceph/must_gather_commands) and 'must_gather_commands_json_output' (quay-io-rhceph-dev-ocs-must-gather-sha256-0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/ceph/must_gather_commands_json_output) directories. Moving the status back to assigned. Tested in version: ocs-operator.v4.6.0-144.ci > It would be nice to gather these informations as part of must-gather '
(In reply to Jilju Joy from comment #16) > > > > * rbd info > > * rbd snap ls > > The above two outputs are present in > quay-io-rhceph-dev-ocs-must-gather-sha256- > 0224479a5c476f28753648875e91c7f492477e14d2e3835e0865062362fac825/ceph/ > must_gather_commands/rbd_snap_info.log > > Hi Pulkit, > The file rbd_snap_info.log comes with '.log' extension. Is there any > particular reason for that ? Other files in 'must_gather_commands' does not > contain '.log' extension. > > Hi Madhu, > Is it okay to have both the outputs in same file ? Please check the > attachment. > >
Let me explain the fix here: 1. It is collecting snapshot information for following rbd commands in a single file named rbd_snap_info.log >> rbd ls -p { pool name } >> rbd info { image name } --pool { pool name } >> rbd snap ls { image name } --pool { pool name } { snap name } 2. It is collecting snapshot information for following cephFS commands in a single file named cephfs_snap_info.log >> ceph fs subvolume info $fs $subvol >> ceph fs subvolume snapshot ls $fs $subvol >> ceph fs subvolume snapshot info $fs $subvol $snap >> ceph fs clone status --> There is no such command These commands are different than the existing ceph commands because here we need to pass some parameter, that's why I thought to follow a different file structure for these. Instead of having the output in separate file, it will be in a single file (one for rbd and one for cephFS) and both the file would be present in must_gather_commands folder. >>> Did not find the above four ceph fs command outputs in 'must_gather_commands' As said earlier you will find this information in cephfs_snap_info.log but it is not present because the subvolumes don't have any snapshots, this file would be populated only if subvolumes has snapshot. I can change it to contain subvolume info even if snapshot is not present. >>> .log extension Have kept the extension as .log can change if there is any issue. ccing Ashish to take a look
One correction in the above comment, it will print subvolume info irresepctive of the snapshot, I think no cephFS subvolume was present in this setup. Jilju, can you please confirm if this setup had any cephFS subvolume. If not then this is expected.
(In reply to Mudit Agarwal from comment #20) > One correction in the above comment, it will print subvolume info > irresepctive of the snapshot, I think no cephFS subvolume was present in > this setup. > > Jilju, can you please confirm if this setup had any cephFS subvolume. If not > then this is expected. Yes, file ceph_fs_subvolume_ls_ocs-storagecluster-cephfilesystem_csi in must_gather_commands directory contains the list. Attachment contain this file.
Jilju, this BZ became dependent on Bug #1889441 which was fixed in 4.6.0-142.ci Can you please run the same test in that build also and share the output?
Collected must-gather logs and verified the content. Both cephfs and rbd volume snapshot details are present in 'must_gather_commands' directory. File cephfs_subvol_and_snap_info (when snapshots are present): http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jijoy-4nov/jijoy-4nov_20201104T111645/logs/failed_testcase_ocs_logs_1604582891/test_expansion_snapshot_clone_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-67f0978d8c422cc88268cbfc24223a71e8cc5ef9d6c152c6cb912ae0b50da8da/ceph/must_gather_commands/cephfs_subvol_and_snap_info File rbd_vol_and_snap_info (when snapshots are present): http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jijoy-4nov/jijoy-4nov_20201104T111645/logs/failed_testcase_ocs_logs_1604582891/test_expansion_snapshot_clone_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-67f0978d8c422cc88268cbfc24223a71e8cc5ef9d6c152c6cb912ae0b50da8da/ceph/must_gather_commands/rbd_vol_and_snap_info File cephfs_subvol_and_snap_info (when snapshots are not present): http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j009vu1cs33-t4bn/j009vu1cs33-t4bn_20201108T060925/logs/failed_testcase_ocs_logs_1604819966/test_recovery_from_volume_deletion_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-87129c2124ed57e69eff5e20f8e4438ee602a30e6868ecbe581acd7d3ef4070a/ceph/must_gather_commands/cephfs_subvol_and_snap_info File rbd_vol_and_snap_info (when snapshots are not present): http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j009vu1cs33-t4bn/j009vu1cs33-t4bn_20201108T060925/logs/failed_testcase_ocs_logs_1604819966/test_recovery_from_volume_deletion_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-87129c2124ed57e69eff5e20f8e4438ee602a30e6868ecbe581acd7d3ef4070a/ceph/must_gather_commands/rbd_vol_and_snap_info Verified the list and describe output of * volumesnapshot_-A * volumesnapshotclass * volumesnapshotcontent which is present in http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jijoy-4nov/jijoy-4nov_20201104T111645/logs/failed_testcase_ocs_logs_1604582891/test_expansion_snapshot_clone_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-67f0978d8c422cc88268cbfc24223a71e8cc5ef9d6c152c6cb912ae0b50da8da/namespaces/openshift-storage/oc_output/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Container Storage 4.6.0 security, bug fix, enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5605