1953796 – must-gather: dev_vfio data is not collected

Bug 1953796 - must-gather: dev_vfio data is not collected

Summary: must-gather: dev_vfio data is not collected

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Container Native Virtualization (CNV)
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Petr Horáček
QA Contact:	Meni Yakove
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-04-27 00:28 UTC by Debarati Basu-Nag
Modified:	2021-07-27 14:31 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-07-27 14:30:44 UTC
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2021:2920	0	None	None	None	2021-07-27 14:31:34 UTC

Description Debarati Basu-Nag 2021-04-27 00:28:06 UTC

Description of problem:
dev_vfio data is not collected from the nodes against oc client version
{noformat}
[cnv-qe-jenkins@iuo-dbasunag-2-dkjx6-executor nodes]$ oc version
Client Version: 4.8.0-202104211714.p0.git.a5f16de-a5f16de
Server Version: 4.8.0-0.nightly-2021-04-09-101800
Kubernetes Version: v1.21.0-rc.0+6d27558
[cnv-qe-jenkins@iuo-dbasunag-2-dkjx6-executor nodes]$ 
{noformat}

Version-Release number of selected component (if applicable):
OC Client Version: 4.8.0-202104211714.p0.git.a5f16de-a5f16de

How reproducible:
Run the following command:
oc adm must-gather --image=registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v4.8.0

Steps to Reproduce:
1. Run "oc adm must-gather --image=registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v4.8.0"
2. Check that dev_vfio file was not collected 

Actual results:
{noformat}
[cnv-qe-jenkins@iuo-dbasunag-2-dkjx6-executor ~]$ cd must-gather.local.4578152400825858150/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-84f7df08407451f5c877cbf92c2d096661955375fb69d1021c2a886cfb1f3a4b/nodes/
[cnv-qe-jenkins@iuo-dbasunag-2-dkjx6-executor nodes]$ ls -ltR
.:
total 0
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 125 Apr 27 00:00 iuo-dbasunag-2-dkjx6-worker-0-7vz7c
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 125 Apr 27 00:00 iuo-dbasunag-2-dkjx6-worker-0-5cpsc
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 125 Apr 27 00:00 iuo-dbasunag-2-dkjx6-worker-0-28ctc
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 113 Apr 27 00:00 iuo-dbasunag-2-dkjx6-master-2
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 113 Apr 27 00:00 iuo-dbasunag-2-dkjx6-master-1
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 113 Apr 27 00:00 iuo-dbasunag-2-dkjx6-master-0

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c:
total 5920
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 5719514 Apr 26 23:57 iuo-dbasunag-2-dkjx6-worker-0-7vz7c_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins  336452 Apr 26 23:57 iuo-dbasunag-2-dkjx6-worker-0-7vz7c_logs_NetworkManager

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc:
total 5420
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 5236137 Apr 26 23:57 iuo-dbasunag-2-dkjx6-worker-0-5cpsc_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins  310834 Apr 26 23:57 iuo-dbasunag-2-dkjx6-worker-0-5cpsc_logs_NetworkManager

./iuo-dbasunag-2-dkjx6-worker-0-28ctc:
total 5088
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 4871876 Apr 26 23:57 iuo-dbasunag-2-dkjx6-worker-0-28ctc_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins  332594 Apr 26 23:57 iuo-dbasunag-2-dkjx6-worker-0-28ctc_logs_NetworkManager

./iuo-dbasunag-2-dkjx6-master-2:
total 4484
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 4500931 Apr 26 23:57 iuo-dbasunag-2-dkjx6-master-2_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   89119 Apr 26 23:57 iuo-dbasunag-2-dkjx6-master-2_logs_NetworkManager

./iuo-dbasunag-2-dkjx6-master-1:
total 3488
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 3490204 Apr 26 23:57 iuo-dbasunag-2-dkjx6-master-1_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   77504 Apr 26 23:57 iuo-dbasunag-2-dkjx6-master-1_logs_NetworkManager

./iuo-dbasunag-2-dkjx6-master-0:
total 3080
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 3073909 Apr 26 23:57 iuo-dbasunag-2-dkjx6-master-0_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   76967 Apr 26 23:57 iuo-dbasunag-2-dkjx6-master-0_logs_NetworkManager
[cnv-qe-jenkins@iuo-dbasunag-2-dkjx6-executor nodes]$ oc version
Client Version: 4.8.0-202104211714.p0.git.a5f16de-a5f16de
Server Version: 4.8.0-0.nightly-2021-04-09-101800
Kubernetes Version: v1.21.0-rc.0+6d27558
[cnv-qe-jenkins@iuo-dbasunag-2-dkjx6-executor nodes]$
{noformat}

Expected results: This works with oc Client Version: v4.2.0-alpha.0-657-g51011e4 
{noformat}
debaratibasu-nag@Debaratis-MacBook-Pro ~ % cd must-gather.local.5339472069230139744/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-84f7df08407451f5c877cbf92c2d096661955375fb69d1021c2a886cfb1f3a4b/nodes
debaratibasu-nag@Debaratis-MacBook-Pro nodes % ls
iuo-dbasunag-2-dkjx6-master-0		iuo-dbasunag-2-dkjx6-master-2		iuo-dbasunag-2-dkjx6-worker-0-5cpsc
iuo-dbasunag-2-dkjx6-master-1		iuo-dbasunag-2-dkjx6-worker-0-28ctc	iuo-dbasunag-2-dkjx6-worker-0-7vz7c
debaratibasu-nag@Debaratis-MacBook-Pro nodes % ls -ltR
total 0
drwxr-xr-x  19 debaratibasu-nag  staff  608 Apr 26 20:08 iuo-dbasunag-2-dkjx6-worker-0-7vz7c
drwxr-xr-x  19 debaratibasu-nag  staff  608 Apr 26 20:08 iuo-dbasunag-2-dkjx6-worker-0-5cpsc
drwxr-xr-x  19 debaratibasu-nag  staff  608 Apr 26 20:08 iuo-dbasunag-2-dkjx6-worker-0-28ctc
drwxr-xr-x   4 debaratibasu-nag  staff  128 Apr 26 20:08 iuo-dbasunag-2-dkjx6-master-2
drwxr-xr-x   4 debaratibasu-nag  staff  128 Apr 26 20:08 iuo-dbasunag-2-dkjx6-master-1
drwxr-xr-x   4 debaratibasu-nag  staff  128 Apr 26 20:08 iuo-dbasunag-2-dkjx6-master-0

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c:
total 13144
drwxr-xr-x  4 debaratibasu-nag  staff      128 Apr 26 20:08 etc
-rw-r--r--  1 debaratibasu-nag  staff  6155408 Apr 26 20:03 iuo-dbasunag-2-dkjx6-worker-0-7vz7c_logs_kubelet
-rw-r--r--  1 debaratibasu-nag  staff   356577 Apr 26 20:03 iuo-dbasunag-2-dkjx6-worker-0-7vz7c_logs_NetworkManager
-rw-r--r--  1 debaratibasu-nag  staff    12756 Apr 26 20:03 lspci
-rw-r--r--  1 debaratibasu-nag  staff      388 Apr 26 20:03 proc_cmdline
-rw-r--r--  1 debaratibasu-nag  staff      156 Apr 26 20:03 dev_vfio
-rw-r--r--  1 debaratibasu-nag  staff    75939 Apr 26 20:03 dmesg
-rw-r--r--  1 debaratibasu-nag  staff     1326 Apr 26 20:03 var-lib-cni-bin
-rw-r--r--  1 debaratibasu-nag  staff      524 Apr 26 20:03 nft-ip-mangle
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 sys_sriov_numvfs
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 sys_sriov_totalvfs
-rw-r--r--  1 debaratibasu-nag  staff     2375 Apr 26 20:03 nft-ip-filter
-rw-r--r--  1 debaratibasu-nag  staff      356 Apr 26 20:03 nft-ip6-nat
-rw-r--r--  1 debaratibasu-nag  staff    75999 Apr 26 20:03 nft-ip-nat
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 bridge
-rw-r--r--  1 debaratibasu-nag  staff    11013 Apr 26 20:03 ip.txt
-rw-r--r--  1 debaratibasu-nag  staff        3 Apr 26 20:03 vlan

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c/etc:
total 0
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 kubernetes
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 cni

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c/etc/kubernetes:
total 0
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 cni

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c/etc/kubernetes/cni:
total 0
drwxr-xr-x  5 debaratibasu-nag  staff  160 Apr 26 20:08 net.d

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c/etc/kubernetes/cni/net.d:
total 8
drwxr-xr-x  4 debaratibasu-nag  staff  128 Apr 26 20:08 whereabouts.d
drwxr-xr-x  3 debaratibasu-nag  staff   96 Apr 26 20:08 multus.d
-rw-r--r--  1 debaratibasu-nag  staff  399 Apr 26 20:03 00-multus.conf

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c/etc/kubernetes/cni/net.d/whereabouts.d:
total 32
-rw-r--r--  1 debaratibasu-nag  staff    162 Apr 26 20:03 whereabouts.conf
-rw-r--r--  1 debaratibasu-nag  staff  11028 Apr 26 20:03 whereabouts.kubeconfig

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c/etc/kubernetes/cni/net.d/multus.d:
total 24
-rw-r--r--  1 debaratibasu-nag  staff  10978 Apr 26 20:03 multus.kubeconfig

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c/etc/cni:
total 0
drwxr-xr-x  5 debaratibasu-nag  staff  160 Apr 26 20:08 net.d

./iuo-dbasunag-2-dkjx6-worker-0-7vz7c/etc/cni/net.d:
total 24
-rw-r--r--  1 debaratibasu-nag  staff  438 Apr 26 20:03 100-crio-bridge.conf
-rw-r--r--  1 debaratibasu-nag  staff   54 Apr 26 20:03 200-loopback.conf
-rw-r--r--  1 debaratibasu-nag  staff  639 Apr 26 20:03 87-podman-bridge.conflist

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc:
total 11968
drwxr-xr-x  4 debaratibasu-nag  staff      128 Apr 26 20:08 etc
-rw-r--r--  1 debaratibasu-nag  staff  5571870 Apr 26 20:03 iuo-dbasunag-2-dkjx6-worker-0-5cpsc_logs_kubelet
-rw-r--r--  1 debaratibasu-nag  staff   339502 Apr 26 20:03 iuo-dbasunag-2-dkjx6-worker-0-5cpsc_logs_NetworkManager
-rw-r--r--  1 debaratibasu-nag  staff    12756 Apr 26 20:03 lspci
-rw-r--r--  1 debaratibasu-nag  staff    70976 Apr 26 20:03 dmesg
-rw-r--r--  1 debaratibasu-nag  staff      388 Apr 26 20:03 proc_cmdline
-rw-r--r--  1 debaratibasu-nag  staff      156 Apr 26 20:03 dev_vfio
-rw-r--r--  1 debaratibasu-nag  staff     1326 Apr 26 20:03 var-lib-cni-bin
-rw-r--r--  1 debaratibasu-nag  staff      524 Apr 26 20:03 nft-ip-mangle
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 sys_sriov_numvfs
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 sys_sriov_totalvfs
-rw-r--r--  1 debaratibasu-nag  staff     2365 Apr 26 20:03 nft-ip-filter
-rw-r--r--  1 debaratibasu-nag  staff      356 Apr 26 20:03 nft-ip6-nat
-rw-r--r--  1 debaratibasu-nag  staff    75921 Apr 26 20:03 nft-ip-nat
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 bridge
-rw-r--r--  1 debaratibasu-nag  staff    13389 Apr 26 20:03 ip.txt
-rw-r--r--  1 debaratibasu-nag  staff        3 Apr 26 20:03 vlan

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc/etc:
total 0
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 kubernetes
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 cni

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc/etc/kubernetes:
total 0
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 cni

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc/etc/kubernetes/cni:
total 0
drwxr-xr-x  5 debaratibasu-nag  staff  160 Apr 26 20:08 net.d

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc/etc/kubernetes/cni/net.d:
total 8
drwxr-xr-x  4 debaratibasu-nag  staff  128 Apr 26 20:08 whereabouts.d
drwxr-xr-x  3 debaratibasu-nag  staff   96 Apr 26 20:08 multus.d
-rw-r--r--  1 debaratibasu-nag  staff  399 Apr 26 20:03 00-multus.conf

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc/etc/kubernetes/cni/net.d/whereabouts.d:
total 32
-rw-r--r--  1 debaratibasu-nag  staff    162 Apr 26 20:03 whereabouts.conf
-rw-r--r--  1 debaratibasu-nag  staff  11028 Apr 26 20:03 whereabouts.kubeconfig

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc/etc/kubernetes/cni/net.d/multus.d:
total 24
-rw-r--r--  1 debaratibasu-nag  staff  10978 Apr 26 20:03 multus.kubeconfig

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc/etc/cni:
total 0
drwxr-xr-x  5 debaratibasu-nag  staff  160 Apr 26 20:08 net.d

./iuo-dbasunag-2-dkjx6-worker-0-5cpsc/etc/cni/net.d:
total 24
-rw-r--r--  1 debaratibasu-nag  staff  438 Apr 26 20:03 100-crio-bridge.conf
-rw-r--r--  1 debaratibasu-nag  staff   54 Apr 26 20:03 200-loopback.conf
-rw-r--r--  1 debaratibasu-nag  staff  639 Apr 26 20:03 87-podman-bridge.conflist

./iuo-dbasunag-2-dkjx6-worker-0-28ctc:
total 11184
drwxr-xr-x  4 debaratibasu-nag  staff      128 Apr 26 20:08 etc
-rw-r--r--  1 debaratibasu-nag  staff   343357 Apr 26 20:03 iuo-dbasunag-2-dkjx6-worker-0-28ctc_logs_NetworkManager
-rw-r--r--  1 debaratibasu-nag  staff  5172409 Apr 26 20:03 iuo-dbasunag-2-dkjx6-worker-0-28ctc_logs_kubelet
-rw-r--r--  1 debaratibasu-nag  staff    12756 Apr 26 20:03 lspci
-rw-r--r--  1 debaratibasu-nag  staff      388 Apr 26 20:03 proc_cmdline
-rw-r--r--  1 debaratibasu-nag  staff      156 Apr 26 20:03 dev_vfio
-rw-r--r--  1 debaratibasu-nag  staff    70607 Apr 26 20:03 dmesg
-rw-r--r--  1 debaratibasu-nag  staff     1326 Apr 26 20:03 var-lib-cni-bin
-rw-r--r--  1 debaratibasu-nag  staff      524 Apr 26 20:03 nft-ip-mangle
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 sys_sriov_numvfs
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 sys_sriov_totalvfs
-rw-r--r--  1 debaratibasu-nag  staff     2367 Apr 26 20:03 nft-ip-filter
-rw-r--r--  1 debaratibasu-nag  staff      356 Apr 26 20:03 nft-ip6-nat
-rw-r--r--  1 debaratibasu-nag  staff    75979 Apr 26 20:03 nft-ip-nat
-rw-r--r--  1 debaratibasu-nag  staff        0 Apr 26 20:03 bridge
-rw-r--r--  1 debaratibasu-nag  staff        3 Apr 26 20:03 vlan
-rw-r--r--  1 debaratibasu-nag  staff    12071 Apr 26 20:03 ip.txt

./iuo-dbasunag-2-dkjx6-worker-0-28ctc/etc:
total 0
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 kubernetes
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 cni

./iuo-dbasunag-2-dkjx6-worker-0-28ctc/etc/kubernetes:
total 0
drwxr-xr-x  3 debaratibasu-nag  staff  96 Apr 26 20:08 cni

./iuo-dbasunag-2-dkjx6-worker-0-28ctc/etc/kubernetes/cni:
total 0
drwxr-xr-x  5 debaratibasu-nag  staff  160 Apr 26 20:08 net.d

./iuo-dbasunag-2-dkjx6-worker-0-28ctc/etc/kubernetes/cni/net.d:
total 8
drwxr-xr-x  4 debaratibasu-nag  staff  128 Apr 26 20:08 whereabouts.d
drwxr-xr-x  3 debaratibasu-nag  staff   96 Apr 26 20:08 multus.d
-rw-r--r--  1 debaratibasu-nag  staff  399 Apr 26 20:03 00-multus.conf

./iuo-dbasunag-2-dkjx6-worker-0-28ctc/etc/kubernetes/cni/net.d/whereabouts.d:
total 32
-rw-r--r--  1 debaratibasu-nag  staff    162 Apr 26 20:03 whereabouts.conf
-rw-r--r--  1 debaratibasu-nag  staff  11028 Apr 26 20:03 whereabouts.kubeconfig

./iuo-dbasunag-2-dkjx6-worker-0-28ctc/etc/kubernetes/cni/net.d/multus.d:
total 24
-rw-r--r--  1 debaratibasu-nag  staff  10978 Apr 26 20:03 multus.kubeconfig

./iuo-dbasunag-2-dkjx6-worker-0-28ctc/etc/cni:
total 0
drwxr-xr-x  5 debaratibasu-nag  staff  160 Apr 26 20:08 net.d

./iuo-dbasunag-2-dkjx6-worker-0-28ctc/etc/cni/net.d:
total 24
-rw-r--r--  1 debaratibasu-nag  staff  438 Apr 26 20:03 100-crio-bridge.conf
-rw-r--r--  1 debaratibasu-nag  staff   54 Apr 26 20:03 200-loopback.conf
-rw-r--r--  1 debaratibasu-nag  staff  639 Apr 26 20:03 87-podman-bridge.conflist

./iuo-dbasunag-2-dkjx6-master-2:
total 9040
-rw-r--r--  1 debaratibasu-nag  staff  4534442 Apr 26 20:03 iuo-dbasunag-2-dkjx6-master-2_logs_kubelet
-rw-r--r--  1 debaratibasu-nag  staff    89656 Apr 26 20:03 iuo-dbasunag-2-dkjx6-master-2_logs_NetworkManager

./iuo-dbasunag-2-dkjx6-master-1:
total 7008
-rw-r--r--  1 debaratibasu-nag  staff  3510131 Apr 26 20:03 iuo-dbasunag-2-dkjx6-master-1_logs_kubelet
-rw-r--r--  1 debaratibasu-nag  staff    77504 Apr 26 20:03 iuo-dbasunag-2-dkjx6-master-1_logs_NetworkManager

./iuo-dbasunag-2-dkjx6-master-0:
total 6200
-rw-r--r--  1 debaratibasu-nag  staff    76967 Apr 26 20:03 iuo-dbasunag-2-dkjx6-master-0_logs_NetworkManager
-rw-r--r--  1 debaratibasu-nag  staff  3092500 Apr 26 20:03 iuo-dbasunag-2-dkjx6-master-0_logs_kubelet
debaratibasu-nag@Debaratis-MacBook-Pro nodes % oc version
Client Version: v4.2.0-alpha.0-657-g51011e4
Server Version: 4.8.0-0.nightly-2021-04-09-101800
Kubernetes Version: v1.21.0-rc.0+6d27558
debaratibasu-nag@Debaratis-MacBook-Pro nodes % 
{noformat}

Additional info:

Comment 1 oshoval 2021-05-13 11:34:06 UTC

Hi,

There is a known issue with the version you used, it was outdated.
Can you try please the latest CNV must gather version ?

It was released after this ticket was opened
see for example https://bugzilla.redhat.com/show_bug.cgi?id=1952036

Thanks

Comment 2 Debarati Basu-Nag 2021-05-13 17:27:15 UTC

With latest 4.8.0 I have verified that this issue is no longer present

{noformat}
(cnv-tests) [cnv-qe-jenkins@iuo-dbn-480-smrvv-executor cnv-tests]$ ls -ltR must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/
must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/:
total 12
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 4096 May 13 17:22 iuo-dbn-480-smrvv-worker-0-pp5wp
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 4096 May 13 17:22 iuo-dbn-480-smrvv-worker-0-lqs6x
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 4096 May 13 17:22 iuo-dbn-480-smrvv-worker-0-b472v
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins  107 May 13 17:22 iuo-dbn-480-smrvv-master-2
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins  107 May 13 17:22 iuo-dbn-480-smrvv-master-1
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins  107 May 13 17:22 iuo-dbn-480-smrvv-master-0

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-pp5wp:
total 20404
drwxr-xr-x. 4 cnv-qe-jenkins cnv-qe-jenkins       35 May 13 17:22 etc
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 19757374 May 13 17:16 iuo-dbn-480-smrvv-worker-0-pp5wp_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   881173 May 13 17:16 iuo-dbn-480-smrvv-worker-0-pp5wp_logs_NetworkManager
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    12756 May 13 17:16 lspci
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      381 May 13 17:16 proc_cmdline
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   101059 May 13 17:16 dmesg
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      156 May 13 17:16 dev_vfio
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins     1326 May 13 17:16 var-lib-cni-bin
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 opt-cni-bin
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 sys_sriov_totalvfs
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 sys_sriov_numvfs
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      460 May 13 17:16 nft-ip6-mangle
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      461 May 13 17:16 nft-ip6-filter
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      524 May 13 17:16 nft-ip-mangle
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins     2551 May 13 17:16 nft-ip-filter
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      662 May 13 17:16 nft-ip6-nat
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    79886 May 13 17:16 nft-ip-nat
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        3 May 13 17:16 vlan
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 bridge
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    11609 May 13 17:16 ip.txt

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-pp5wp/etc:
total 0
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 19 May 13 17:22 cni
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 17 May 13 17:22 kubernetes

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-pp5wp/etc/cni:
total 0
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 6 May 13 17:22 net.d

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-pp5wp/etc/cni/net.d:
total 0

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-pp5wp/etc/kubernetes:
total 0
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 19 May 13 17:22 cni

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-pp5wp/etc/kubernetes/cni:
total 0
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 6 May 13 17:22 net.d

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-pp5wp/etc/kubernetes/cni/net.d:
total 0

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-lqs6x:
total 23888
drwxr-xr-x. 4 cnv-qe-jenkins cnv-qe-jenkins       35 May 13 17:22 etc
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 23400935 May 13 17:16 iuo-dbn-480-smrvv-worker-0-lqs6x_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   823660 May 13 17:16 iuo-dbn-480-smrvv-worker-0-lqs6x_logs_NetworkManager
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    12756 May 13 17:16 lspci
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      381 May 13 17:16 proc_cmdline
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    75828 May 13 17:16 dmesg
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      156 May 13 17:16 dev_vfio
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins     1326 May 13 17:16 var-lib-cni-bin
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 opt-cni-bin
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 sys_sriov_totalvfs
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 sys_sriov_numvfs
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      460 May 13 17:16 nft-ip6-mangle
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      465 May 13 17:16 nft-ip6-filter
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      524 May 13 17:16 nft-ip-mangle
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins     2553 May 13 17:16 nft-ip-filter
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      664 May 13 17:16 nft-ip6-nat
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    79903 May 13 17:16 nft-ip-nat
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        3 May 13 17:16 vlan
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 bridge
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    12647 May 13 17:16 ip.txt

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-lqs6x/etc:
total 0
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 19 May 13 17:22 cni
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 17 May 13 17:22 kubernetes

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-lqs6x/etc/cni:
total 0
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 6 May 13 17:22 net.d

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-lqs6x/etc/cni/net.d:
total 0

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-lqs6x/etc/kubernetes:
total 0
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 19 May 13 17:22 cni

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-lqs6x/etc/kubernetes/cni:
total 0
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 6 May 13 17:22 net.d

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-lqs6x/etc/kubernetes/cni/net.d:
total 0

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-b472v:
total 22544
drwxr-xr-x. 4 cnv-qe-jenkins cnv-qe-jenkins       35 May 13 17:22 etc
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 22009236 May 13 17:16 iuo-dbn-480-smrvv-worker-0-b472v_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   842738 May 13 17:16 iuo-dbn-480-smrvv-worker-0-b472v_logs_NetworkManager
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    12756 May 13 17:16 lspci
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      381 May 13 17:16 proc_cmdline
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    75120 May 13 17:16 dmesg
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      156 May 13 17:16 dev_vfio
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins     1326 May 13 17:16 var-lib-cni-bin
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 opt-cni-bin
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 sys_sriov_totalvfs
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 sys_sriov_numvfs
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      460 May 13 17:16 nft-ip6-mangle
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      465 May 13 17:16 nft-ip6-filter
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      524 May 13 17:16 nft-ip-mangle
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins     2542 May 13 17:16 nft-ip-filter
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins      664 May 13 17:16 nft-ip6-nat
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    79793 May 13 17:16 nft-ip-nat
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        3 May 13 17:16 vlan
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins        0 May 13 17:16 bridge
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins    13385 May 13 17:16 ip.txt

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-b472v/etc:
total 0
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 19 May 13 17:22 cni
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 17 May 13 17:22 kubernetes

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-b472v/etc/cni:
total 0
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 6 May 13 17:22 net.d

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-b472v/etc/cni/net.d:
total 0

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-b472v/etc/kubernetes:
total 0
drwxr-xr-x. 3 cnv-qe-jenkins cnv-qe-jenkins 19 May 13 17:22 cni

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-b472v/etc/kubernetes/cni:
total 0
drwxr-xr-x. 2 cnv-qe-jenkins cnv-qe-jenkins 6 May 13 17:22 net.d

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-worker-0-b472v/etc/kubernetes/cni/net.d:
total 0

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-master-2:
total 5376
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 5427991 May 13 17:16 iuo-dbn-480-smrvv-master-2_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   69976 May 13 17:16 iuo-dbn-480-smrvv-master-2_logs_NetworkManager

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-master-1:
total 6096
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 6154001 May 13 17:16 iuo-dbn-480-smrvv-master-1_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   82119 May 13 17:16 iuo-dbn-480-smrvv-master-1_logs_NetworkManager

must-gather.local.4953088058232816798/registry-proxy-engineering-redhat-com-rh-osbs-container-native-virtualization-cnv-must-gather-rhel8-sha256-bf486421c49a28cce3b82576af46cf3e863286d329be1cc5e5eaeaae96b007bc/nodes/iuo-dbn-480-smrvv-master-0:
total 5572
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins 5618519 May 13 17:16 iuo-dbn-480-smrvv-master-0_logs_kubelet
-rw-r--r--. 1 cnv-qe-jenkins cnv-qe-jenkins   81922 May 13 17:16 iuo-dbn-480-smrvv-master-0_logs_NetworkManager
(cnv-tests) [cnv-qe-jenkins@iuo-dbn-480-smrvv-executor cnv-tests]$ 
{noformat}

Comment 5 errata-xmlrpc 2021-07-27 14:30:44 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2920

Note You need to log in before you can comment on or make changes to this bug.