Description of problem: L2 can not get ip address in nested Hyper-V on KVM with ADVANCED-VIRT-8.4.0-RHEL-8-20210205.n.0: qemu-kvm-5.2.0-5.module+el8.4.0+9775+0937c167.x86_64 Can not reproduce with qemu-kvm-5.1.0-19.module+el8.3.1+9795+4ce2a535.x86_64 (ADVANCED-VIRT-8.3.1-RHEL-8-20210205.n.0) Can not reproduce with qemu-kvm-4.2.0-44.module+el8.4.0+9776+c5744f20.x86_64 (slow train in AppStream: RHEL-8.4.0-20210205.n.0) Version-Release number of selected components (if applicable): qemu-kvm-5.2.0-5.module+el8.4.0+9775+0937c167.x86_64 (ADVANCED-VIRT-8.4.0-RHEL-8-20210205.n.0) RHEL Version: L0: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz RHEL8.4 with kernel-4.18.0-281.el8.x86_64 qemu-kvm-5.2.0-5.module+el8.4.0+9775+0937c167.x86_64(AV-8.4.0-RHEL-8-20210205.n.0) L1: Nested Hyper-V 2019/2016 L2: RHEL8.4 How reproducible: 100% Steps to Reproduce: 1. Start L0 with enabled nested # cat /sys/module/kvm_intel/parameters/nested 1 2. Start L1 Windows server 2019 with enabled Hyper-V role: /usr/libexec/qemu-kvm -name intel-2019-bios -m 16G -smp 8 \ -rtc base=localtime,driftfix=none \ -boot order=cd,menu=on \ -monitor stdio -M q35 -vga std -vnc :1 \ -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0,vhost=on,queues=4 \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x3 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1 \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:52:11:36:1f:80,bus=pci.2,mq=on,vectors=10 \ -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/images/2019-bios.qcow2,node-name=system_file \ -blockdev driver=qcow2,node-name=drive_system_disk,file=system_file \ -object iothread,id=thread0 \ -device virtio-blk-pci,iothread=thread0,drive=drive_system_disk,id=system_disk,bootindex=0,bus=pci.1 \ -usb -device usb-tablet \ -cpu host,hv_passthrough 3. Check L1 can get ip address 4. Start L2 rhel8 guest and check ip address Actual results: L2 can not get ip address # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:15:5d:e1:46:03 brd ff:ff:ff:ff:ff:ff inet6 fe80::215:5dff:fee1:4603/64 scope link valid_lft forever preferred_lft forever Expected results: L2 can get ip Additional info: N/A
(In reply to HuijingHei from comment #0) ... > -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1 > \ > -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:52:11:36:1f:80,bus=pci.2,mq=on, > vectors=10 \ > -blockdev Does the issue reproduce with some other NIC, e.g. e1000e? Or is the failure specific to virtio (and maybe virtio-win)?
(In reply to Vitaly Kuznetsov from comment #2) > (In reply to HuijingHei from comment #0) > ... > > -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1 > > \ > > -device > > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:52:11:36:1f:80,bus=pci.2,mq=on, > > vectors=10 \ > > -blockdev > > Does the issue reproduce with some other NIC, e.g. e1000e? Or is the failure > specific to virtio (and maybe virtio-win)? Thanks Vitaly! Start L1 (with AV qemu-kvm-5.2.0) with e1000e and rtl8139 NIC, create new switch, can not reproduce the problem. Then create new virtio switch, L2 also can get ip, the issue is gone Check in L1 and find the root cause: the interface changed when start L1 with AV qemu-kvm-5.2.0, but the switch still connected to the old interface. If change the switch to connect to new interface, there is no problem. Is this by design? ============================================================ 1) Start L1 with qemu-kvm-4.2.0-44, check the interface > get-netadapter Name InterfaceDescription ---- -------------------- Ethernet Red Hat VirtIO Ethernet Adapter vEthernet (New Virtual Hyper-V Virtual Ethernet Adapter Switch) 2) Start L1 with AV qemu-kvm-5.2.0-5, check the interface > get-netadapter Name InterfaceDescription ---- -------------------- Ethernet 3 Red Hat VirtIO Ethernet Adapter #2 vEthernet (New Virtual Hyper-V Virtual Ethernet Adapter Switch)
(In reply to HuijingHei from comment #3) > (In reply to Vitaly Kuznetsov from comment #2) > > (In reply to HuijingHei from comment #0) > > ... > > > -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1 > > > \ > > > -device > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:52:11:36:1f:80,bus=pci.2,mq=on, > > > vectors=10 \ > > > -blockdev > > > > Does the issue reproduce with some other NIC, e.g. e1000e? Or is the failure > > specific to virtio (and maybe virtio-win)? > > Thanks Vitaly! > > Start L1 (with AV qemu-kvm-5.2.0) with e1000e and rtl8139 NIC, create new > switch, can not reproduce the problem. Then create new virtio switch, L2 > also can get ip, the issue is gone > > Check in L1 and find the root cause: the interface changed when start L1 > with AV qemu-kvm-5.2.0, but the switch still connected to the old interface. > If change the switch to connect to new interface, there is no problem. Is > this by design? > I'm not exactly sure how Windows matches NICs (hope someone from Meirav's team know), apparently it's not just MAC. It is more or less OK when things change with machine type change, it wouldn't be OK within the same machine type. E.g. you can try using "-M pc-q35-rhel8.3.0" instead of '-M q35' and things are supposed to stay constant.
(In reply to Vitaly Kuznetsov from comment #4) > > > > Check in L1 and find the root cause: the interface changed when start L1 > > with AV qemu-kvm-5.2.0, but the switch still connected to the old interface. > > If change the switch to connect to new interface, there is no problem. Is > > this by design? > > > > I'm not exactly sure how Windows matches NICs (hope someone from Meirav's > team know), > apparently it's not just MAC. It is more or less OK when things change with > machine > type change, it wouldn't be OK within the same machine type. E.g. you can > try using > > "-M pc-q35-rhel8.3.0" instead of '-M q35' and things are supposed to stay > constant. Hi Vitaly, Test L1 using '-M pc-q35-rhel8.2.0' with qemu-kvm 4.2.0-44(not support pc-q35-rhel8.3.0) and 5.2.0, check L1 net interface is different, do you mean they should be the same? Thanks! ====================================================================== With qemu-kvm 5.2.0: > Get-NetAdapter | Format-List -Property "Name", "InterfaceDescription" Name : Ethernet 3 InterfaceDescription : Red Hat VirtIO Ethernet Adapter #2 With qemu-kvm 4.2.0-44: > Get-NetAdapter | Format-List -Property "Name", "InterfaceDescription" Name : Ethernet InterfaceDescription : Red Hat VirtIO Ethernet Adapter
Ok, so the problem here is that with QEMU upgrade Windows starts seeing Virio NIC as a different one as as this new NIC is not configured, networking doesn't work. It would be great if someone from virtio-win team could take a look, I honestly have no idea how Windows distinguishes NIC, apparently it's more than just MAC address.
(In reply to HuijingHei from comment #6) > Hi Vitaly, > > Test L1 using '-M pc-q35-rhel8.2.0' with qemu-kvm 4.2.0-44(not support > pc-q35-rhel8.3.0) and 5.2.0, check L1 net interface is different, do you > mean they should be the same? Thanks! > Yes, I believe that with the same machine type and the exact same configuration Windows should not see your nic as a new one.
(In reply to Vitaly Kuznetsov from comment #9) > (In reply to HuijingHei from comment #6) > > Hi Vitaly, > > > > Test L1 using '-M pc-q35-rhel8.2.0' with qemu-kvm 4.2.0-44(not support > > pc-q35-rhel8.3.0) and 5.2.0, check L1 net interface is different, do you > > mean they should be the same? Thanks! > > > > Yes, I believe that with the same machine type and the exact same > configuration > Windows should not see your nic as a new one. Thanks Vitaly for your confirmation!
(In reply to HuijingHei from comment #6) > (In reply to Vitaly Kuznetsov from comment #4) > > > > > > Check in L1 and find the root cause: the interface changed when start L1 > > > with AV qemu-kvm-5.2.0, but the switch still connected to the old interface. > > > If change the switch to connect to new interface, there is no problem. Is > > > this by design? > > > > > > > I'm not exactly sure how Windows matches NICs (hope someone from Meirav's > > team know), > > apparently it's not just MAC. It is more or less OK when things change with > > machine > > type change, it wouldn't be OK within the same machine type. E.g. you can > > try using > > > > "-M pc-q35-rhel8.3.0" instead of '-M q35' and things are supposed to stay > > constant. > > Hi Vitaly, > > Test L1 using '-M pc-q35-rhel8.2.0' with qemu-kvm 4.2.0-44(not support > pc-q35-rhel8.3.0) and 5.2.0, check L1 net interface is different, do you > mean they should be the same? Thanks! > > ====================================================================== > With qemu-kvm 5.2.0: > > Get-NetAdapter | Format-List -Property "Name", "InterfaceDescription" > > Name : Ethernet 3 > InterfaceDescription : Red Hat VirtIO Ethernet Adapter #2 > > > With qemu-kvm 4.2.0-44: > > Get-NetAdapter | Format-List -Property "Name", "InterfaceDescription" > > Name : Ethernet > InterfaceDescription : Red Hat VirtIO Ethernet Adapter Hi, If for some reason the NIC changed its location on the PCI bugs, from a Windows point of view it will be a new NIC. Can you please check the device address in the device manager? Run devmgmt.msc -> Click on "Network adapters" -> Click on the "Red Hat VirtIO Ethernet Adapter" (on each one if there are several)-> On the first property page you should see "Location" text label. You can also try to explicitly specify the address on the bus for virtio-net. In any case, if for some reason the address is changing (according to what Device Manager will show), that might mean that the qemu is not preserving HW settings correctly for "pc-q35-rhel8.2.0" machine accross the versions. Best regards, Yan.
(In reply to Yan Vugenfirer from comment #12) > (In reply to HuijingHei from comment #6) > > (In reply to Vitaly Kuznetsov from comment #4) > > > > > > > > Check in L1 and find the root cause: the interface changed when start L1 > > > > with AV qemu-kvm-5.2.0, but the switch still connected to the old interface. > > > > If change the switch to connect to new interface, there is no problem. Is > > > > this by design? > > > > > > > > > > I'm not exactly sure how Windows matches NICs (hope someone from Meirav's > > > team know), > > > apparently it's not just MAC. It is more or less OK when things change with > > > machine > > > type change, it wouldn't be OK within the same machine type. E.g. you can > > > try using > > > > > > "-M pc-q35-rhel8.3.0" instead of '-M q35' and things are supposed to stay > > > constant. > > > > Hi Vitaly, > > > > Test L1 using '-M pc-q35-rhel8.2.0' with qemu-kvm 4.2.0-44(not support > > pc-q35-rhel8.3.0) and 5.2.0, check L1 net interface is different, do you > > mean they should be the same? Thanks! > > > > ====================================================================== > > With qemu-kvm 5.2.0: > > > Get-NetAdapter | Format-List -Property "Name", "InterfaceDescription" > > > > Name : Ethernet 3 > > InterfaceDescription : Red Hat VirtIO Ethernet Adapter #2 > > > > > > With qemu-kvm 4.2.0-44: > > > Get-NetAdapter | Format-List -Property "Name", "InterfaceDescription" > > > > Name : Ethernet > > InterfaceDescription : Red Hat VirtIO Ethernet Adapter > > Hi, > > > If for some reason the NIC changed its location on the PCI bugs, from a > Windows point of view it will be a new NIC. > Can you please check the device address in the device manager? Run > devmgmt.msc -> Click on "Network adapters" -> Click on the "Red Hat VirtIO > Ethernet Adapter" (on each one if there are several)-> On the first property > page you should see "Location" text label. > > You can also try to explicitly specify the address on the bus for virtio-net. > > In any case, if for some reason the address is changing (according to what > Device Manager will show), that might mean that the qemu is not preserving > HW settings correctly for "pc-q35-rhel8.2.0" machine accross the versions. Hi Yan, Checked vm( with "pc-q35-rhel8.2.0") the device address is the same "PCI bus 2, device 0, function 0", with qemu-kvm 4.2.0-44 and 5.2.0, but get different interface name. Thanks! BR Huijing Hei
1. The bug is not related to Hyper-V. According to comment #6 Windows (L1 guest) treats the device as a new device. 2. The issue is not virtio-net specific. The same happens with QXL device (after booting the quest on QEMU 5.2 upstream, the settings of VGA are reset), and probably with other devices. 3. The bug looks like a critical bug that can hinder the upgrade from QEMU 4.2 to QEMU 5.2 as the devices' settings will be restored to default and it even might cause Windows VMs to reset Windows activation status.
Hi Lijin. Did your team test the migration of Windows VMs from RHEL8.3 to RHEL8.4? Thanks, Yan.
Not sure if this issue is for stastic ip, maybe relate to bz1934158