Bug 1942938

Summary: [release-4.7] must-gather improvements
Product: OpenShift Container Platform Reporter: Jan Chaloupka <jchaloup>
Component: ocAssignee: Jan Chaloupka <jchaloup>
Status: CLOSED ERRATA QA Contact: zhou ying <yinzhou>
Severity: high Docs Contact:
Priority: high    
Version: 4.7CC: aos-bugs, jokerman, maszulik, mfojtik, yinzhou
Target Milestone: ---   
Target Release: 4.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1942935
: 1942940 (view as bug list) Environment:
Last Closed: 2021-04-20 18:52:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1942935    
Bug Blocks: 1942940    

Description Jan Chaloupka 2021-03-25 11:06:26 UTC
+++ This bug was initially created as a clone of Bug #1942935 +++

- stop trying to gather metrics and other endpoints directly from pods
  https://github.com/openshift/oc/pull/763
- prevent inspect from panic-ing if pods are missing
  https://github.com/openshift/oc/pull/762
- indicate how many bytes the hidden secret key was
  https://github.com/openshift/oc/pull/752
- inspect clusteroperators as a backup to must-gather if it fails
  https://github.com/openshift/oc/pull/749
- Add summary to oc must-gather
  https://github.com/openshift/oc/pull/738

Comment 1 Maciej Szulik 2021-03-25 13:40:08 UTC
These improvements are significantly helping debugging clusters so I'm bumping priority for those to high.

Comment 2 zhou ying 2021-04-06 08:26:54 UTC
1) [root@localhost oc]# ./oc version 
Client Version: 4.7.99
Server Version: 4.5.37
Kubernetes Version: v1.18.3+cdb0358
[root@localhost oc]# ./oc adm must-gather 
[must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e29a7270bff4c9ca267b5a1976d2e4db2a1e86146aca6fc7145cba5b3fc09d8
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: 2c9a1b73-9cf7-4d08-b081-d55b6c02a0c3
ClusterVersion: Stable at "4.5.37"
ClusterOperators:
	All healthy and stable
.....


We could see the summary to oc must-gather. 



2) From the output of must-gather for pods :
[root@localhost openshift-etcd]# tree pods/
pods/
├── etcd-ip-10-0-169-230.us-west-1.compute.internal
│   ├── etcd
│   │   └── etcd
│   │       └── logs
│   │           ├── current.log
│   │           ├── previous.insecure.log
│   │           └── previous.log
│   ├── etcdctl
│   │   └── etcdctl
│   │       └── logs
│   │           ├── current.log
│   │           ├── previous.insecure.log
│   │           └── previous.log
│   ├── etcd-ensure-env-vars
│   │   └── etcd-ensure-env-vars
│   │       └── logs
│   │           ├── current.log
│   │           ├── previous.insecure.log
│   │           └── previous.log


we could make sure the function for : stop trying to gather metrics and other endpoints directly from pods, has works well . 


3)  Check the function for : indicate how many bytes the hidden secret key was:

[root@localhost ~]# ./oc get secret my-secret -o yaml 
apiVersion: v1
data:
  key1: c3VwZXJzZWNyZXQ=
  key2: dG9wc2VjcmV0
kind: Secret


[root@localhost secrets]# more my-secret.yaml 
---
apiVersion: v1
data:
  key1: MTEgYnl0ZXMgbG9uZw==
  key2: OSBieXRlcyBsb25n
kind: Secret

So this function works well . 

4) Check the function for :inspect clusteroperators as a backup to must-gather if it fails

 [root@localhost oc]# ./oc adm  must-gather --image='quay.io/openshift-release-dev/ocp-v4.0-art-dev:nonexist' 
[must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev:nonexist
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: 2c9a1b73-9cf7-4d08-b081-d55b6c02a0c3
ClusterVersion: Stable at "4.5.37"
ClusterOperators:
	All healthy and stable


[must-gather      ] OUT namespace/openshift-must-gather-9c7sm created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-sjltn created
[must-gather      ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev:nonexist created
[must-gather-7cggz] OUT gather did not start: unable to pull image: ImagePullBackOff: Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev:nonexist"
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-sjltn deleted
[must-gather      ] OUT namespace/openshift-must-gather-9c7sm deleted


When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: 2c9a1b73-9cf7-4d08-b081-d55b6c02a0c3
ClusterVersion: Stable at "4.5.37"
ClusterOperators:
	All healthy and stable


Gathering data for ns/openshift-config...
Gathering data for ns/openshift-config-managed...
.....

we could see when the must-gather failed , will start the inspect command .

Comment 6 errata-xmlrpc 2021-04-20 18:52:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.7 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1149