+++ This bug was initially created as a clone of Bug #1942935 +++ - stop trying to gather metrics and other endpoints directly from pods https://github.com/openshift/oc/pull/763 - prevent inspect from panic-ing if pods are missing https://github.com/openshift/oc/pull/762 - indicate how many bytes the hidden secret key was https://github.com/openshift/oc/pull/752 - inspect clusteroperators as a backup to must-gather if it fails https://github.com/openshift/oc/pull/749 - Add summary to oc must-gather https://github.com/openshift/oc/pull/738
These improvements are significantly helping debugging clusters so I'm bumping priority for those to high.
1) [root@localhost oc]# ./oc version Client Version: 4.7.99 Server Version: 4.5.37 Kubernetes Version: v1.18.3+cdb0358 [root@localhost oc]# ./oc adm must-gather [must-gather ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e29a7270bff4c9ca267b5a1976d2e4db2a1e86146aca6fc7145cba5b3fc09d8 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information. ClusterID: 2c9a1b73-9cf7-4d08-b081-d55b6c02a0c3 ClusterVersion: Stable at "4.5.37" ClusterOperators: All healthy and stable ..... We could see the summary to oc must-gather. 2) From the output of must-gather for pods : [root@localhost openshift-etcd]# tree pods/ pods/ ├── etcd-ip-10-0-169-230.us-west-1.compute.internal │ ├── etcd │ │ └── etcd │ │ └── logs │ │ ├── current.log │ │ ├── previous.insecure.log │ │ └── previous.log │ ├── etcdctl │ │ └── etcdctl │ │ └── logs │ │ ├── current.log │ │ ├── previous.insecure.log │ │ └── previous.log │ ├── etcd-ensure-env-vars │ │ └── etcd-ensure-env-vars │ │ └── logs │ │ ├── current.log │ │ ├── previous.insecure.log │ │ └── previous.log we could make sure the function for : stop trying to gather metrics and other endpoints directly from pods, has works well . 3) Check the function for : indicate how many bytes the hidden secret key was: [root@localhost ~]# ./oc get secret my-secret -o yaml apiVersion: v1 data: key1: c3VwZXJzZWNyZXQ= key2: dG9wc2VjcmV0 kind: Secret [root@localhost secrets]# more my-secret.yaml --- apiVersion: v1 data: key1: MTEgYnl0ZXMgbG9uZw== key2: OSBieXRlcyBsb25n kind: Secret So this function works well . 4) Check the function for :inspect clusteroperators as a backup to must-gather if it fails [root@localhost oc]# ./oc adm must-gather --image='quay.io/openshift-release-dev/ocp-v4.0-art-dev:nonexist' [must-gather ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev:nonexist When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information. ClusterID: 2c9a1b73-9cf7-4d08-b081-d55b6c02a0c3 ClusterVersion: Stable at "4.5.37" ClusterOperators: All healthy and stable [must-gather ] OUT namespace/openshift-must-gather-9c7sm created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-sjltn created [must-gather ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev:nonexist created [must-gather-7cggz] OUT gather did not start: unable to pull image: ImagePullBackOff: Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev:nonexist" [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-sjltn deleted [must-gather ] OUT namespace/openshift-must-gather-9c7sm deleted When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information. ClusterID: 2c9a1b73-9cf7-4d08-b081-d55b6c02a0c3 ClusterVersion: Stable at "4.5.37" ClusterOperators: All healthy and stable Gathering data for ns/openshift-config... Gathering data for ns/openshift-config-managed... ..... we could see when the must-gather failed , will start the inspect command .
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.7.7 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1149