Bug 2216447

Summary:	must-gather: Multiple empty files under vms/<vm-name> if the VM was live migrated
Product:	Container Native Virtualization (CNV)	Reporter:	nijin ashok <nashok>
Component:	Logging	Assignee:	Dharmit Shah <dshah>
Status:	CLOSED ERRATA	QA Contact:	Ahmad <ahafe>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.13.0	CC:	dbasunag, kmajcher, sradco, stirabos
Target Milestone:	---
Target Release:	4.14.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	hco-bundle-registry-container-v4.14.0.rhel9-1274	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-01 14:51:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description nijin ashok 2023-06-21 12:19:01 UTC

Description of problem:

If the VM was live migrated, there will be virt-launcher pods with the status "completed".

~~~
# oc get pod
NAME                                         READY   STATUS      RESTARTS   AGE
virt-launcher-rhel8-i181f1yxkfqqrv83-75lxb   1/1     Running     0          55s
virt-launcher-rhel8-i181f1yxkfqqrv83-bn728   0/1     Completed   0          86s
virt-launcher-rhel8-i181f1yxkfqqrv83-lmlqv   0/1     Completed   0          29h
virt-launcher-rhel8-i181f1yxkfqqrv83-lzcwx   0/1     Completed   0          22h
virt-launcher-rhel8-i181f1yxkfqqrv83-v5lvb   0/1     Completed   0          73s
~~~

must-gather is trying to do `oc exec` on all these completed pods which will fail. This will result in multiple empty files in must gather.

~~~
nijin-cnv]# ls -l vms/rhel8-i181f1yxkfqqrv83/
total 44
drwxr-xr-x. 2 root root    6 Jun 21 17:39 log
-rw-------. 1 root root 5206 Jun 21 17:34 nijin-cnv_rhel8-i181f1yxkfqqrv83.log
-rw-r--r--. 1 root root 1133 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-75lxb.bridge.txt
-rw-r--r--. 1 root root 8330 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-75lxb.dumpxml.xml
-rw-r--r--. 1 root root 1287 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-75lxb.ip.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-75lxb.ruletables.txt
-rw-r--r--. 1 root root  263 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-bn728.bridge.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-bn728.dumpxml.xml
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-bn728.ip.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-bn728.ruletables.txt
-rw-r--r--. 1 root root  263 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-lmlqv.bridge.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-lmlqv.dumpxml.xml
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-lmlqv.ip.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-lmlqv.ruletables.txt
-rw-r--r--. 1 root root  263 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-lzcwx.bridge.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-lzcwx.dumpxml.xml
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-lzcwx.ip.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-lzcwx.ruletables.txt
-rw-r--r--. 1 root root  263 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-v5lvb.bridge.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-v5lvb.dumpxml.xml
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-v5lvb.ip.txt
-rw-r--r--. 1 root root    0 Jun 21 17:39 virt-launcher-rhel8-i181f1yxkfqqrv83-v5lvb.ruletables.txt
~~~ 



Version-Release number of selected component (if applicable):

OpenShift Virtualization   4.13.1

How reproducible:

100%

Steps to Reproduce:

1. Migrate VMs multiple times so that we will have virt-launcher pods with "completed" status.

2. Collect a must-gather and list the files under namespace/<namespace-name>/vms/<vm-name>

There will be multiple empty files under it.

Actual results:

must-gather: Multiple empty files under vms/<vm-name> if the VM was live migrated.

Expected results:

it should only do `oc exec` on running virt-launcher pods.

Additional info:

Comment 1 Ahmad 2023-07-20 14:43:32 UTC

QA:

Tested fix version CNV-v4.14.0.rhel9-1279

created a VM and executed  live migration multiple times:

[cloud-user@ocp-psi-executor log]$ oc get virtualmachineinstancemigrations
NAME                                      PHASE       VMI
fedora-middle-jay-migration-7e1go         Succeeded   fedora-middle-jay
fedora-middle-jay-migration-8unuc         Succeeded   fedora-middle-jay
fedora-middle-jay-migration-duk2u         Succeeded   fedora-middle-jay
fedora-middle-jay-migration-wcw52         Succeeded   fedora-middle-jay
rhel9-orthodox-dormouse-migration-kcyil   Succeeded   rhel9-orthodox-dormouse



[cloud-user@ocp-psi-executor fedora-middle-jay]$ oc get pods
NAME                                          READY   STATUS      RESTARTS   AGE
virt-launcher-fedora-middle-jay-5z6hs         0/1     Completed   0          4h11m
virt-launcher-fedora-middle-jay-f4485         0/1     Completed   0          4h47m
virt-launcher-fedora-middle-jay-rnr67         0/1     Completed   0          4h20m
virt-launcher-fedora-middle-jay-tm5h2         1/1     Running     0          163m
virt-launcher-fedora-middle-jay-wj6xj         0/1     Completed   0          4h19m





must_gather results under namespace/<namespace-name>/vms/<vm-name>:

[cloud-user@ocp-psi-executor fedora-middle-jay]$ ls -l
total 32
-rw-------. 1 cloud-user cloud-user  6812 Jul 20 07:58 default_fedora-middle-jay.log
drwxr-sr-x. 2 cloud-user cloud-user     6 Jul 20 10:41 log
-rw-r--r--. 1 cloud-user cloud-user  1202 Jul 20 10:41 virt-launcher-fedora-middle-jay-tm5h2.bridge.txt
-rw-r--r--. 1 cloud-user cloud-user 10022 Jul 20 10:41 virt-launcher-fedora-middle-jay-tm5h2.dumpxml.xml
-rw-r--r--. 1 cloud-user cloud-user  1286 Jul 20 10:41 virt-launcher-fedora-middle-jay-tm5h2.ip.txt
-rw-r--r--. 1 cloud-user cloud-user  1655 Jul 20 10:41 virt-launcher-fedora-middle-jay-tm5h2.ruletables.txt




no empty files and logs are only about running virt-launcher pods

Comment 8 errata-xmlrpc 2023-08-01 14:51:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Virtualization 4.12.5 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:4421