Description of problem:
If the primary interface (and network) is not explicitly specified in the VM spec, then it is not added after hot-plugging a bridged interface. As a result, after rebooting the VM, then primary interface is entirely removed from the VM.


Version-Release number of selected component (if applicable):
CNV 4.14.0 (brew.registry.redhat.io/rh-osbs/iib:532547)
cluster-network-addons-operator-rhel9:v4.14.0-16


How reproducible:
Always


Steps to Reproduce:
1.
Create a VM using the attached vm-fedora.yaml (note that no network or interface are specified in the spec).

2.
Create a node bridge interface on all workers using the attached nncp-linux-bridge.yaml
(set the port name to the name ot an available secondary interface on the nodes).

3.
Create a NetworkAttachmentDefinition using the attached bridge-nad.yaml, which utilizes the bridge created in section 2.

4.
Hot-plug an interface based on the NetworkAttachmentDefinition you created.
$ virtctl addinterface vm-fedora --name hp2 --network-attachment-definition-name hp-br-nad

5. Migrate the VM to complete the hot-plug action:
$ cat << EOF | oc apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
  name: migration-job
spec:
  vmiName: vm-fedora
EOF

6. After migration is done and hot-plug is completed, check the VM spec - only the secondary interface/network is there:
$ oc get vm vm-fedora -o yaml
...
    spec:
      architecture: amd64
      domain:
        devices:
...
          interfaces:
          - bridge: {}
            macAddress: 02:51:aa:00:00:00
            name: hp2
...
      networks:
      - multus:
          networkName: hp-br-nad
        name: hp2
...
Both interfaces still appear in the guest VM:
$ virtctl console vm-fedora
Successfully connected to vm-fedora console. The escape sequence is ^]
[fedora@vm-fedora ~]$ 
[fedora@vm-fedora ~]$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 7850 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:4f:40:51 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8900 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:51:aa:00:00:00 brd ff:ff:ff:ff:ff:ff
    altname enp2s0


7. Hard-reboot the VM:
$ virtctl restart vm-fedora

8. After reboot is completed - login to the guest VM again, and check its interfaces:
$ virtctl console vm-fedora
Successfully connected to vm-fedora console. The escape sequence is ^]

[fedora@vm-fedora ~]$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8900 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:51:aa:00:00:00 brd ff:ff:ff:ff:ff:ff
    altname enp1s0


Actual results:
<BUG>
Only one interface appears, and although it's named eth0, you can see that its MAC is the MAC address of the secondary (hot-plugged) interface before the reboot.


Expected results:
VM primary interface must remain. It might mean that the primary interface should be added to the VM spec when a new interface is hot-plugged.


Additional info: