Description of problem: Errors are observed while gathering data from must-gather utility. Based on : https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/ apiversion should be picked : apps/v1. Looks like it need some investigation Error 1 : ======== [must-gather-4bxjp] POD error: unable to recognize "/etc/node-gather-ds.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1" [must-gather-4bxjp] POD Error from server (NotFound): daemonsets.apps "node-gather-daemonset" not found [must-gather-4bxjp] POD error: unable to recognize "/etc/node-gather-ds.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1" Error 2: ======= [must-gather-4bxjp] POD error: there is no need to specify a resource type as a separate argument when passing arguments in resource/name form (e.g. 'oc get resource/<resource_name>' instead of 'oc get resource resource/<resource_name>' Version-Release number of selected component (if applicable): $ oc version Client Version: 4.4.0-0.nightly-2020-03-03-195752 Server Version: 4.4.0-0.nightly-2020-03-02-011520 Kubernetes Version: v1.17.1 How reproducible: always (while running automation tests) Steps to Reproduce: 1. Run tests : http://git.app.eng.bos.redhat.com/git/cnv-tests.git/tree/tests/must_gather/test_must_gather.py 2. 3. Actual results: older api version is being called Expected results: apiversion should be picked correctly. Additional info:
PR merged, and new image automatically pushed to quay.io/kubevirt/must-gather
Please set 'fixed in version' field
please add fixed in version
I ran the oc adm must-gather --dest-dir=./must-gather-talayan --image=registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.3.0-42 [must-gather ] OUT Using must-gather plugin-in image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.3.0-42 [must-gather ] OUT namespace/openshift-must-gather-2hwzs created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-4dcpb created [must-gather ] OUT pod for plug-in image registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.3.0-42 created [must-gather-lp67c] POD Gathering data for ns/openshift-cnv... [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Error from server (NotFound): namespaces "kubevirt-hyperconverged" not found [must-gather-lp67c] POD Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Gathering data for ns/openshift-marketplace... [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Error from server (NotFound): namespaces "cluster-network-addons" not found [must-gather-lp67c] POD Gathering data for ns/openshift-sdn... [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Error from server (NotFound): namespaces "sriov-network-operator" not found [must-gather-lp67c] POD Error from server (NotFound): namespaces "kubevirt-web-ui" not found [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Error from server (NotFound): namespaces "cdi" not found [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD Wrote inspect data to must-gather. [must-gather-lp67c] POD No resources found [must-gather-lp67c] POD No resources found [must-gather-lp67c] POD namespace/node-gather created [must-gather-lp67c] POD serviceaccount/node-gather created [must-gather-lp67c] POD securitycontextconstraints.security.openshift.io/privileged added to: ["system:serviceaccount:node-gather:node-gather"] [must-gather-lp67c] POD daemonset.apps/node-gather-daemonset created [must-gather-lp67c] OUT gather logs unavailable: unexpected EOF [must-gather-lp67c] OUT waiting for gather to complete [must-gather-lp67c] OUT gather never finished: timed out waiting for the condition [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-4dcpb deleted [must-gather ] OUT namespace/openshift-must-gather-2hwzs deleted error: gather never finished for pod must-gather-lp67c: timed out waiting for the condition Client Version: 4.4.0-rc.11 Server Version: 4.4.0-rc.8 Kubernetes Version: v1.17.1 it looks like this bug https://bugzilla.redhat.com/show_bug.cgi?id=1824007 however i don't see any of the errors mentioned in the bug description.
Does must-gather still generate all the correct files in the output directory?
no
(In reply to Tareq Alayan from comment #10) > no also adding oc describe ds node-gather-daemonset -nnode-gather Name: node-gather-daemonset Selector: name=node-gather-daemonset Node-Selector: <none> Labels: <none> Annotations: deprecated.daemonset.template.generation: 1 Desired Number of Nodes Scheduled: 3 Current Number of Nodes Scheduled: 3 Number of Nodes Scheduled with Up-to-date Pods: 3 Number of Nodes Scheduled with Available Pods: 3 Number of Nodes Misscheduled: 0 Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: name=node-gather-daemonset Service Account: node-gather Containers: node-probe: Image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.3.0-42 Port: <none> Host Port: <none> Command: /bin/bash -c echo ok > /tmp/healthy && sleep INF Limits: cpu: 100m memory: 50Mi Requests: cpu: 100m memory: 50Mi Readiness: exec [cat /tmp/healthy] delay=5s timeout=1s period=5s #success=1 #failure=3 Environment: <none> Mounts: /host/dev from dev (rw) /host/etc from etc (rw) /host/opt from opt (rw) /host/proc from proc (rw) /host/sys from sys (rw) /host/var from var (rw) Volumes: sys: Type: HostPath (bare host directory volume) Path: /sys HostPathType: Directory proc: Type: HostPath (bare host directory volume) Path: /proc HostPathType: Directory dev: Type: HostPath (bare host directory volume) Path: /dev HostPathType: Directory etc: Type: HostPath (bare host directory volume) Path: /etc HostPathType: Directory opt: Type: HostPath (bare host directory volume) Path: /opt HostPathType: Directory var: Type: HostPath (bare host directory volume) Path: /var HostPathType: Directory Events: <none> shouldn't this ds dies after must-gather is done or killed? I mean there are no output and this ds is still up. let me know if u need access to env
I tried again on a different cluster (this time i didn't get any timeout) But i see error message about non existing ns: [must-gather-9hxpl] POD Gathering data for ns/openshift-cnv... [must-gather-9hxpl] POD Wrote inspect data to must-gather. [must-gather-9hxpl] POD Error from server (NotFound): namespaces "kubevirt-hyperconverged" not found [must-gather-9hxpl] POD Error from server (NotFound): namespaces "cluster-network-addons" not found [must-gather-9hxpl] POD Error from server (NotFound): namespaces "sriov-network-operator" not found [must-gather-9hxpl] POD Error from server (NotFound): namespaces "kubevirt-web-ui" not found [must-gather-9hxpl] POD Error from server (NotFound): namespaces "cdi" not found where this is coming from? I don't see any POD error: unable to recognize "/etc/node-gather-ds.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1" [must-gather-4bxjp] POD Error from server (NotFound): daemonsets.apps "node-gather-daemonset" not found [must-gather-4bxjp] POD error: unable to recognize "/etc/node-gather-ds.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1" So i can't reproduce the issue reported but i saw the non found namespaces and sometinmes it fails on https://bugzilla.redhat.com/show_bug.cgi?id=1824007
I will move this bug to verified. I opened a new bug https://bugzilla.redhat.com/show_bug.cgi?id=1831061 for the non existing ns issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:2011
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days