Bug 1979155

Summary: [4.7 clone] must-gather-helper pod fails to come up on ODF Managed Services setup, hence no ceph collection succeeds
Product: [Red Hat Storage] Red Hat OpenShift Container Storage Reporter: Mudit Agarwal <muagarwa>
Component: must-gatherAssignee: Mudit Agarwal <muagarwa>
Status: CLOSED ERRATA QA Contact: Filip Balák <fbalak>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.7CC: ebenahar, muagarwa, nberry, ocs-bugs, olakra, omitrani, ratamir, sabose
Target Milestone: ---Keywords: AutomationBackLog, ZStream
Target Release: OCS 4.7.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v4.7.3-457.ci Doc Type: Bug Fix
Doc Text:
.`must-gather-helper` pod can now be created using the correct deployment yaml Previously, the `must-gather-helper` pod creation failed and skipped the ceph collections. This was caused by due to incorrect YAML deployment. This update fixes the selection of correct deployment YAML resulting in creating and running the `must-gather-helper` pod successfully.
 Story Points: ---
Clone Of: 1978663 Environment:
Last Closed: 2021-08-11 13:59:14 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1978663, 1979514    
Bug Blocks:    

Comment 10 Filip Balák 2021-08-05 09:33:35 UTC 
ocs must-gather now works correctly when multiple operators are installed in openshift-storage namespace.

$ oc get csv -n openshift-storage 
NAME                                          DISPLAY                                      VERSION        REPLACES   PHASE
anaconda-team-edition.v6.1.3                  Anaconda Team Edition                        6.1.3                     Succeeded
ansible-automation-platform-operator.v2.0.0   Ansible Automation Platform (early access)   2.0.0                     Succeeded
atomicorp-helm-operator.v0.0.1                Atomicorp Operator                           0.0.1                     Succeeded
ocs-operator.v4.7.3-243.ci                    OpenShift Container Storage                  4.7.3-243.ci              Succeeded

must-gather pod was created correctly and ceph data were collected:

$ oc adm must-gather --image=registry.redhat.io/ocs4/ocs-must-gather-rhel8:v4.7 --dest-dir=ocs-must-gather
(...)
[must-gather-vwczh] POD 2021-08-05T09:14:26.860643953Z creating helper pod
[must-gather-vwczh] POD 2021-08-05T09:14:27.883619966Z pod/must-gather-vwczh-helper created
[must-gather-vwczh] POD 2021-08-05T09:14:27.888425429Z debugging node ip-10-0-159-117.us-east-2.compute.internal 
[must-gather-vwczh] POD 2021-08-05T09:14:27.888765698Z debugging node ip-10-0-168-226.us-east-2.compute.internal 
[must-gather-vwczh] POD 2021-08-05T09:14:27.889061080Z debugging node ip-10-0-198-118.us-east-2.compute.internal 
[must-gather-vwczh] POD 2021-08-05T09:14:28.339277906Z pod/must-gather-vwczh-helper labeled
[must-gather-vwczh] POD 2021-08-05T09:14:28.361022913Z waiting for 101 102 104 105 to terminate
[must-gather-vwczh] POD 2021-08-05T09:14:28.480128534Z Starting pod/ip-10-0-198-118us-east-2computeinternal-debug ...
[must-gather-vwczh] POD 2021-08-05T09:14:28.480222570Z To use host binaries, run `chroot /host`
[must-gather-vwczh] POD 2021-08-05T09:14:28.567739414Z Starting pod/ip-10-0-159-117us-east-2computeinternal-debug ...
[must-gather-vwczh] POD 2021-08-05T09:14:28.567822179Z To use host binaries, run `chroot /host`
[must-gather-vwczh] POD 2021-08-05T09:14:28.583711640Z Starting pod/ip-10-0-168-226us-east-2computeinternal-debug ...
[must-gather-vwczh] POD 2021-08-05T09:14:28.583711640Z To use host binaries, run `chroot /host`
[must-gather-vwczh] POD 2021-08-05T09:15:28.573209818Z pod/must-gather-vwczh-helper condition met
(...)

$ ls -l ocs-must-gather/registry-redhat-io-ocs4-ocs-must-gather-rhel8-sha256-1949179411885858ec719ab052868c734b98b49787498a8297f1a4ace0283eae/ceph/
total 24
-rw-r--r--. 1 fbalak fbalak 3336 Aug  5 11:16 event-filter.html
drwxr-xr-x. 2 fbalak fbalak 4096 Aug  5 11:17 logs
drwxr-xr-x. 2 fbalak fbalak 4096 Aug  5 11:17 must_gather_commands
drwxr-xr-x. 2 fbalak fbalak 4096 Aug  5 11:17 must_gather_commands_json_output
drwxr-xr-x. 3 fbalak fbalak 4096 Aug  5 11:17 namespaces
-rw-r--r--. 1 fbalak fbalak  550 Aug  5 11:16 timestamp

$ cat ocs-must-gather/registry-redhat-io-ocs4-ocs-must-gather-rhel8-sha256-1949179411885858ec719ab052868c734b98b49787498a8297f1a4ace0283eae/ceph/namespaces/openshift-storage/ceph.rook.io/cephclusters/ocs-storagecluster-cephcluster.yaml 
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  creationTimestamp: "2021-08-05T07:06:46Z"
  finalizers:
  - cephcluster.ceph.rook.io
  generation: 1
  labels:
    app: ocs-storagecluster
(...)


--> VERIFIED
Comment 17 errata-xmlrpc 2021-08-11 13:59:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Container Storage 4.7.3 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3135