Bug 1816778

Summary:	[must gather] Needs Investigation for APIversion related errors
Product:	Container Native Virtualization (CNV)	Reporter:	Geetika Kapoor <gkapoor>
Component:	Providers	Assignee:	Avram Levitter <alevitte>
Status:	CLOSED ERRATA	QA Contact:	Tareq Alayan <talayan>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.3.0	CC:	alevitte, cnv-qe-bugs, fdeutsch, ncredi, pelauter, talayan
Target Milestone:	---	Keywords:	Regression
Target Release:	2.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	cnv-must-gather-container-v2.3.0-42	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-05-04 19:10:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Geetika Kapoor 2020-03-24 17:40:28 UTC

Description of problem:

Errors are observed while gathering data from must-gather utility.

Based on : https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/
apiversion should be picked : apps/v1.

Looks like it need some investigation

Error 1 :
========

[must-gather-4bxjp] POD error: unable to recognize "/etc/node-gather-ds.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
[must-gather-4bxjp] POD Error from server (NotFound): daemonsets.apps "node-gather-daemonset" not found
[must-gather-4bxjp] POD error: unable to recognize "/etc/node-gather-ds.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1"

Error 2:
=======
[must-gather-4bxjp] POD error: there is no need to specify a resource type as a separate argument when passing arguments in resource/name form (e.g. 'oc get resource/<resource_name>' instead of 'oc get resource resource/<resource_name>'



Version-Release number of selected component (if applicable):
$ oc version
Client Version: 4.4.0-0.nightly-2020-03-03-195752
Server Version: 4.4.0-0.nightly-2020-03-02-011520
Kubernetes Version: v1.17.1


How reproducible:
always (while running automation tests)

Steps to Reproduce:
1. Run tests : http://git.app.eng.bos.redhat.com/git/cnv-tests.git/tree/tests/must_gather/test_must_gather.py
2.
3.

Actual results:

older api version is being called

Expected results:

apiversion should be picked correctly.

Additional info:

Comment 3 Avram Levitter 2020-04-01 08:16:53 UTC

PR merged, and new image automatically pushed to quay.io/kubevirt/must-gather

Comment 6 Nelly Credi 2020-04-13 06:57:13 UTC

Please set 'fixed in version' field

Comment 7 Nelly Credi 2020-04-20 07:52:48 UTC

please add fixed in version

Comment 8 Tareq Alayan 2020-04-27 08:17:14 UTC

I ran the 
oc adm must-gather --dest-dir=./must-gather-talayan --image=registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.3.0-42
[must-gather      ] OUT Using must-gather plugin-in image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.3.0-42
[must-gather      ] OUT namespace/openshift-must-gather-2hwzs created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-4dcpb created
[must-gather      ] OUT pod for plug-in image registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.3.0-42 created
[must-gather-lp67c] POD Gathering data for ns/openshift-cnv...
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Error from server (NotFound): namespaces "kubevirt-hyperconverged" not found
[must-gather-lp67c] POD Gathering data for ns/openshift-operator-lifecycle-manager...
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Gathering data for ns/openshift-marketplace...
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Error from server (NotFound): namespaces "cluster-network-addons" not found
[must-gather-lp67c] POD Gathering data for ns/openshift-sdn...
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Error from server (NotFound): namespaces "sriov-network-operator" not found
[must-gather-lp67c] POD Error from server (NotFound): namespaces "kubevirt-web-ui" not found
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Error from server (NotFound): namespaces "cdi" not found
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD Wrote inspect data to must-gather.
[must-gather-lp67c] POD No resources found
[must-gather-lp67c] POD No resources found
[must-gather-lp67c] POD namespace/node-gather created
[must-gather-lp67c] POD serviceaccount/node-gather created
[must-gather-lp67c] POD securitycontextconstraints.security.openshift.io/privileged added to: ["system:serviceaccount:node-gather:node-gather"]
[must-gather-lp67c] POD daemonset.apps/node-gather-daemonset created
[must-gather-lp67c] OUT gather logs unavailable: unexpected EOF
[must-gather-lp67c] OUT waiting for gather to complete
[must-gather-lp67c] OUT gather never finished: timed out waiting for the condition
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-4dcpb deleted
[must-gather      ] OUT namespace/openshift-must-gather-2hwzs deleted
error: gather never finished for pod must-gather-lp67c: timed out waiting for the condition

Client Version: 4.4.0-rc.11
Server Version: 4.4.0-rc.8
Kubernetes Version: v1.17.1


it looks like this bug https://bugzilla.redhat.com/show_bug.cgi?id=1824007


however i don't see any of the errors mentioned in the bug description.

Comment 9 Avram Levitter 2020-04-27 09:57:33 UTC

Does must-gather still generate all the correct files in the output directory?

Comment 10 Tareq Alayan 2020-04-27 12:12:44 UTC

no

Comment 11 Tareq Alayan 2020-04-27 12:47:09 UTC

(In reply to Tareq Alayan from comment #10)
> no

also adding oc describe ds node-gather-daemonset -nnode-gather

Name:           node-gather-daemonset
Selector:       name=node-gather-daemonset
Node-Selector:  <none>
Labels:         <none>
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 3
Number of Nodes Misscheduled: 0
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           name=node-gather-daemonset
  Service Account:  node-gather
  Containers:
   node-probe:
    Image:      registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.3.0-42
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/bash
      -c
      echo ok > /tmp/healthy && sleep INF
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Readiness:    exec [cat /tmp/healthy] delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /host/dev from dev (rw)
      /host/etc from etc (rw)
      /host/opt from opt (rw)
      /host/proc from proc (rw)
      /host/sys from sys (rw)
      /host/var from var (rw)
  Volumes:
   sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  Directory
   proc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  Directory
   dev:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
   etc:
    Type:          HostPath (bare host directory volume)
    Path:          /etc
    HostPathType:  Directory
   opt:
    Type:          HostPath (bare host directory volume)
    Path:          /opt
    HostPathType:  Directory
   var:
    Type:          HostPath (bare host directory volume)
    Path:          /var
    HostPathType:  Directory
Events:            <none>


shouldn't this ds dies after must-gather is done or killed? 
I mean there are no output and this ds is still up.

let me know if u need access to env

Comment 12 Tareq Alayan 2020-05-04 14:08:40 UTC

I tried again on a different cluster (this time i didn't get any timeout)

But i see error message about non existing ns: 

[must-gather-9hxpl] POD Gathering data for ns/openshift-cnv...
[must-gather-9hxpl] POD Wrote inspect data to must-gather.
[must-gather-9hxpl] POD Error from server (NotFound): namespaces "kubevirt-hyperconverged" not found

[must-gather-9hxpl] POD Error from server (NotFound): namespaces "cluster-network-addons" not found

[must-gather-9hxpl] POD Error from server (NotFound): namespaces "sriov-network-operator" not found
[must-gather-9hxpl] POD Error from server (NotFound): namespaces "kubevirt-web-ui" not found

[must-gather-9hxpl] POD Error from server (NotFound): namespaces "cdi" not found


where this is coming from?

I don't see any 
POD error: unable to recognize "/etc/node-gather-ds.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
[must-gather-4bxjp] POD Error from server (NotFound): daemonsets.apps "node-gather-daemonset" not found
[must-gather-4bxjp] POD error: unable to recognize "/etc/node-gather-ds.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1"



So i can't reproduce the issue reported but i saw the non found namespaces and sometinmes it fails on https://bugzilla.redhat.com/show_bug.cgi?id=1824007

Comment 13 Tareq Alayan 2020-05-04 14:36:10 UTC

I will move this bug to verified.
I opened a new bug https://bugzilla.redhat.com/show_bug.cgi?id=1831061 for the non existing ns issue.

Comment 15 errata-xmlrpc 2020-05-04 19:10:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:2011

Comment 16 Red Hat Bugzilla 2023-09-14 05:54:39 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days