Created attachment 1781743 [details]
vm-fedora.yaml

Description of problem:
The NET_ADMIN capability was removed from the virt-launcher pod, which is the back-end port of each CNV VM. As a result - running iptables command on a virt-launcher pod fails (with "Permission Denied" error).
Running gathering the virt-launcher's iptables info is now possible from the virt-handler pod (a brief how-to in the "Additional info" section). must-gather should be able to collect its iptables data the same way.


Version-Release number of selected component (if applicable):
Openshift Client: 4.8.0-fc.0
Openshift version: 4.8.0-fc.0


How reproducible:
Always


Steps to Reproduce:
1. On an OCP+CNV 4.8 cluster - create a VM (VM manifest yaml is attached):
$ oc apply -f vm-fedora.yaml 
virtualmachine.kubevirt.io/vm-fedora created

2. Start the VM:
$ virtctl start vm-fedora
VM vm-fedora was scheduled to start

3. Wait for the VM to run:
$ oc get vmi -w
NAME        AGE   PHASE        IP    NODENAME
vm-fedora   0s    Scheduling         
vm-fedora   5s    Scheduled          worker-0-2
vm-fedora   5s    Scheduled          worker-0-2
vm-fedora   7s    Running      10.129.2.101   worker-0-2

4. Run must-gather:
$ oc adm must-gather --image=registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v4.8.0 --dest-dir=<path to destination dir>


Actual results:
<BUG>
The output doesn't contain any iptables data.


Expected results:
iptables output files named iptables.txt exist in the output directory.


Additional info:
With NET_ADMIN capability removed from the virt-launcher pod, running ipables manually fails with "Permission denied", even when run as root user
$ oc get pod
NAME                            READY   STATUS    RESTARTS   AGE
virt-launcher-vm-fedora-7xmrv   2/2     Running   0          36m

$ oc exec -it virt-launcher-vm-fedora-7xmrv -- bash
Defaulted container "compute" out of: compute, volumecontainerdisk, container-disk-binary (init), volumecontainerdisk-init (init)

[root@vm-fedora /]# iptables -t nat -L -v -n
iptables v1.8.4 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)


To get the virt-launcher's iptables:
1. Search for the virt-handler pod that runs on the same node where the VM runs (worker-0-2 in this example).
$ c get pods -n openshift-cnv -owide | grep virt-handler
virt-handler-4r2pg                                    1/1     Running   0          4d5h   10.129.2.37       worker-0-2   <none>           <none>
virt-handler-59pw2                                    1/1     Running   0          4d5h   10.131.0.37       worker-0-0   <none>           <none>
virt-handler-kxd6j                                    1/1     Running   0          4d5h   10.128.2.45       worker-0-1   <none>           <none>

2. Connect to the virt-handler pod's bash:
$ oc exec -it -n openshift-cnv virt-handler-4r2pg -- bash
Defaulted container "virt-handler" out of: virt-handler, virt-launcher (init)

3. Find the virt-launcher's process:
$ ps -ef | grep /usr/bin/virt-launcher
root     4047884 4047871  0 15:06 ?        00:00:00 /usr/bin/virt-launcher --qemu-timeout 5m --name vm-fedora --uid ac4055ab-4809-4fcc-a2a5-7b625d61ccf4 --namespace openshift-virtualization-os-images --kubevirt-share-dir /var/run/kubevirt --ephemeral-disk-dir /var/run/kubevirt-ephemeral-disks --container-disk-dir /var/run/kubevirt/container-disks --grace-period-seconds 15 --hook-sidecars 0 --less-pvc-space-toleration 10 --ovmf-path /usr/share/OVMF
root     4047939 4047884  0 15:06 ?        00:00:02 /usr/bin/virt-launcher --qemu-timeout 5m --name vm-fedora --uid ac4055ab-4809-4fcc-a2a5-7b625d61ccf4 --namespace openshift-virtualization-os-images --kubevirt-share-dir /var/run/kubevirt --ephemeral-disk-dir /var/run/kubevirt-ephemeral-disks --container-disk-dir /var/run/kubevirt/container-disks --grace-period-seconds 15 --hook-sidecars 0 --less-pvc-space-toleration 10 --ovmf-path /usr/share/OVMF --no-fork true

4. Switch to the virt-launcher network namespace:
$ nsenter -t 4047884 -n

5. Verify you are in the correct namespace by verifying the k6t-eth0 interface is  there:
$ ip link show dev k6t-eth0
5: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 4e:c0:90:24:7e:d3 brd ff:ff:ff:ff:ff:ff

6. Now you can run iptables, e.g.:
$ iptables -t nat -L -v -n