Bug 1907988 - VM loses dynamic IP address of its default interface after migration
Summary: VM loses dynamic IP address of its default interface after migration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Networking
Version: 2.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.6.0
Assignee: Alona Kaplan
QA Contact: Yossi Segev
URL:
Whiteboard:
: 1909493 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-15 16:00 UTC by Yossi Segev
Modified: 2021-03-10 11:23 UTC (History)
5 users (show)

Fixed In Version: virt-launcher-container-v2.6.0-104
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-10 11:22:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
vm-fedora.yaml (892 bytes, text/plain)
2020-12-15 16:48 UTC, Yossi Segev
no flags Details
vm-mig.yaml (126 bytes, text/plain)
2020-12-15 16:49 UTC, Yossi Segev
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt kubevirt pull 4804 0 None closed virt-launcher: [masquerade] Stop filtering dhcp reuqests by vm MAC 2021-02-16 11:51:51 UTC
Github kubevirt kubevirt pull 4846 0 None closed [release-0.37] virt-launcher: [masquerade] Stop filtering dhcp reuqests by vm MAC 2021-02-16 11:51:52 UTC
Github kubevirt kubevirt pull 4847 0 None closed [release-0.36] virt-launcher: [masquerade] Stop filtering dhcp reuqests by vm MAC 2021-02-16 11:51:51 UTC
Red Hat Product Errata RHSA-2021:0799 0 None None None 2021-03-10 11:23:19 UTC

Description Yossi Segev 2020-12-15 16:00:10 UTC
Description of problem:
A VM is assigned a dynamic IP (if not explicitly configured with a static one). After migrating and rebooting the VM, it's primary interface doesn't get an IP address.


Version-Release number of selected component (if applicable):
CNV v2.6.0


How reproducible:
Always


Steps to Reproduce:
1. Create a VM using the attached Vm yaml:
$ oc apply -f vm-fedora.yaml

2. Start the VMI:
$ virtctl start vm-fedora

3. Wait for the VM to be in "Running" state
$ oc get vmi -w
NAME        AGE   PHASE    IP            NODENAME
vm-fedora   0s                           
vm-fedora   0s    Pending                 
vm-fedora   1s    Scheduling                 
vm-fedora   1s    Scheduling                 
vm-fedora   5s    Scheduled                  ysegev4726-kbmwp-worker-0-gl4s5
vm-fedora   5s    Scheduled                  ysegev4726-kbmwp-worker-0-gl4s5
vm-fedora   8s    Running      10.129.3.135   ysegev4726-kbmwp-worker-0-gl4s5

4. Login to the VM using console:
$ virtctl console vm-fedora

5. When prompted:
Username: fedora
Password: fedora

6. Verify the primary interface was assigned an IP address:
[fedora@vm-fedora ~]$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:00:d2:f7:28 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 10.0.2.2/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
       valid_lft 86313473sec preferred_lft 86313473sec
    inet6 fe80::ff:fed2:f728/64 scope link 
       valid_lft forever preferred_lft forever

7. Return to the cluster (using ctrl+], or open another shell), and run the attached migration spec:
$ oc apply -f vm-mig.yaml 
virtualmachineinstancemigration.kubevirt.io/vm-fedora created

8. After the migration is completed (VMI is running on a different node) - return to the VM console ("virtctl console vm-fedora").

9. Make sure the primary interface still has its IP address:
[fedora@vm-fedora ~]$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:00:d2:f7:28 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 10.0.2.2/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
       valid_lft 86313298sec preferred_lft 86313298sec
    inet6 fe80::ff:fed2:f728/64 scope link 
       valid_lft forever preferred_lft forever

10. Reboot the VM:
[fedora@vm-fedora ~]$ sudo shutdown -r now

11. After the VM finishes startup - enter username and password, and check the primary interface IP address:
[fedora@vm-fedora ~]$ ip addr show dev eth0


Actual results:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:00:d2:f7:28 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet6 fe80::ff:fed2:f728/64 scope link 
       valid_lft forever preferred_lft forever

<BUG> No IPv4 address.


Expected results:
The interface should have a dynamic IP address, just like before the migration and restart.


Additional info:
1. The bug doesn't happen without migration (i.e. when just restarting the VM, without migrating it to a different node - the VM comes back up with the IP address on its primary interface).
2. The bug doesn't happen when setting a static IP to the primary interface.
3. The bug is only seen on the VM's default masquerade interface (note that the scenario here doesn't include any bridge creation or multus configuration).

Comment 1 Yossi Segev 2020-12-15 16:48:37 UTC
Created attachment 1739399 [details]
vm-fedora.yaml

Comment 2 Yossi Segev 2020-12-15 16:49:17 UTC
Created attachment 1739400 [details]
vm-mig.yaml

Comment 3 chhu 2020-12-29 07:48:12 UTC
I hit this bug on rhel6.10 guest with e1000 interface after restart VM(without migration operation).
-----------------------------------------------------------------------------------------------------
Create a rhel6.10 guest with e1000 interface and sata disk from dv, start it:
# oc get vmi
NAME                 AGE     PHASE     IP             NODENAME
asb-vm-dv-nfs-rhel   2m43s   Running   10.128.3.105   intel-e52650-16-3.englab.nay.redhat.com

# ping 10.128.3.105
PING 10.128.3.105 (10.128.3.105) 56(84) bytes of data.
64 bytes from 10.128.3.105: icmp_seq=1 ttl=64 time=0.602 ms
64 bytes from 10.128.3.105: icmp_seq=2 ttl=64 time=0.043 ms

# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:00:00:12:E7:0F  
          inet addr:10.0.2.2  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::ff:fe12:e70f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1042 (1.0 KiB)  TX bytes:1502 (1.4 KiB)

sh-4.4# virsh dumpxml openshift-cnv_asb-vm-dv-nfs-rhel| grep interface -A 8
    <interface type='ethernet'>
      <mac address='02:00:00:12:e7:0f'/>
      <target dev='tap0' managed='no'/>
      <model type='e1000'/>
      <mtu size='1450'/>
      <boot order='2'/>
      <alias name='ua-default'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>

Restart the VM:
# oc get vmi
NAME                 AGE   PHASE     IP             NODENAME
asb-vm-dv-nfs-rhel   9s    Running   10.128.3.106   intel-e52650-16-3.englab.nay.redhat.com

# ping 10.128.3.106
PING 10.128.3.106 (10.128.3.106) 56(84) bytes of data.
64 bytes from 10.128.3.106: icmp_seq=1 ttl=64 time=0.613 ms
64 bytes from 10.128.3.106: icmp_seq=2 ttl=64 time=0.039 ms

[root@localhost ~]# lspci|grep Ethernet
02:01.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
[root@localhost ~]# ifconfig -a
eth1      Link encap:Ethernet  HWaddr 02:00:00:84:D5:BF  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

sh-4.4# virsh dumpxml openshift-cnv_asb-vm-dv-nfs-rhel|grep interface -A 8
    <interface type='ethernet'>
      <mac address='02:00:00:84:d5:bf'/>
      <target dev='tap0' managed='no'/>
      <model type='e1000'/>
      <mtu size='1450'/>
      <boot order='2'/>
      <alias name='ua-default'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>

After restart the VM, the mac address changed, ip address changed, and the interface has no ip address in VM.

Comment 4 Alona Kaplan 2021-01-03 07:36:32 UTC
*** Bug 1909493 has been marked as a duplicate of this bug. ***

Comment 5 Petr Horáček 2021-01-07 10:56:36 UTC
chhu, your issue does not seem to be connected to this BZ. I took the liberty to open another BZ for you: https://bugzilla.redhat.com/show_bug.cgi?id=1913674 I will need you to provide some missing info there.

Comment 6 Yossi Segev 2021-01-24 15:25:21 UTC
Verified on:
OCP version: 4.7.0-fc.3
CNV version: 2.6.0
virt-launcher version: virt-launcher-container-v2.6.0-105 (registry.redhat.io/container-native-virtualization/virt-launcher@sha256:0fd8c961ee02ce119e437f98ac2c8859139ab046a23cc18363aabd91a9cb8e56)

Verified by repeating the exact scenario in the bug description, and now IP address still exists on the primary interface after migration and reboot.

Comment 9 errata-xmlrpc 2021-03-10 11:22:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 2.6.0 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0799


Note You need to log in before you can comment on or make changes to this bug.