Description of problem: After live migrating win2016 guest with failover vf from BCM57810 and 82599ES network cards separately on src and dst hosts, ping dst host will fail in windows guest and the network info of Device manager is src netowrk card info, not the dst network card info. The guest is hung when rebooting the guest and scan the hardware changes via Device Manager. Version-Release number of selected component (if applicable): # rpm -q qemu-kvm qemu-kvm-7.0.0-9.el9.x86_64 [root@dell-per730-28 home]# uname -r 5.14.0-133.el9.x86_64 source nic info: NetXtreme II BCM57810 10 Gigabit Ethernet destination nic info: 82599ES 10-Gigabit SFI/SFP+ Network Connection How reproducible: 100% Steps to Reproduce: 1.create vf on both src host and dst host echo 1 > /sys/bus/pci/devices/0000\:1a\:00.1/sriov_numvfs 2. create failover-vf and failover-bridge network on both src and dst host # virsh net-dumpxml failover-bridge <network connections='1'> <name>failover-bridge</name> <uuid>1943a508-b0b7-4274-be5a-6f0143d10f40</uuid> <forward mode='bridge'/> <bridge name='br0'/> </network> # virsh net-dumpxml failover-vf <network connections='1'> <name>failover-vf</name> <uuid>4319b666-8f4b-410a-886f-17b6df772224</uuid> <forward mode='hostdev' managed='yes'> <address type='pci' domain='0x0000' bus='0x1a' slot='0x08' function='0x2'/> </forward> </network> 3. boot win2016 guest with failover vf on src host <interface type='network'> <mac address='52:54:00:aa:1c:ef'/> <source network='failover-bridge'/> <model type='virtio'/> <teaming type='persistent'/> <alias name='ua-test'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:aa:1c:ef'/> <source network='failover-vf'/> <teaming type='transient' persistent='ua-test'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </interface> 4. live migrating the guest 5. after migration, check the network in guest 6. reboot the guest or scan the hardware changes via Device manager in guest Actual results: after step 3, network info are correct, ping can succeed. after step 5, ping dst host ip succeeds, ping src host ip fails. The nic info in Device Manager is BCM57810, not 82599ES. See attachment. ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : WIN-6O8PO0LB52A Primary Dns Suffix . . . . . . . : Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : lab.eng.pek2.redhat.com Ethernet adapter Ethernet Instance 0 3: Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #3 Physical Address. . . . . . . . . : 52-54-00-00-16-16 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IPv6 Address. . . . . . . . . . . : 2620:52:0:49d2:b48b:fb85:cc13:d246(Preferred) Link-local IPv6 Address . . . . . : fe80::b48b:fb85:cc13:d246%7(Preferred) IPv4 Address. . . . . . . . . . . : 10.73.211.186(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.254.0 Lease Obtained. . . . . . . . . . : Wednesday, July 27, 2022 2:42:50 AM Lease Expires . . . . . . . . . . : Thursday, July 28, 2022 2:42:46 AM Default Gateway . . . . . . . . . : fe80::52c7:903:533b:88e1%7 10.73.211.254 DHCP Server . . . . . . . . . . . : 10.73.2.108 DHCPv6 IAID . . . . . . . . . . . : 122835968 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2A-6F-99-05-9A-7A-64-D7-19-B5 DNS Servers . . . . . . . . . . . : 10.73.2.107 10.73.2.108 10.66.127.10 NetBIOS over Tcpip. . . . . . . . : Enabled Ethernet adapter Ethernet Instance 0 4: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #4 Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::9528:5d3e:a846:ee90%2(Preferred) IPv4 Address. . . . . . . . . . . : 192.168.43.102(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . . . . : Wednesday, July 27, 2022 2:42:42 AM Lease Expires . . . . . . . . . . : Wednesday, July 27, 2022 2:52:42 AM Default Gateway . . . . . . . . . : 192.168.43.2 DHCP Server . . . . . . . . . . . : 192.168.43.6 DHCPv6 IAID . . . . . . . . . . . : 38949888 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2A-6F-99-05-9A-7A-64-D7-19-B5 DNS Servers . . . . . . . . . . . : 192.168.43.2 NetBIOS over Tcpip. . . . . . . . : Enabled Tunnel adapter isatap.lab.eng.pek2.redhat.com: Media State . . . . . . . . . . . : Media disconnected Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com Description . . . . . . . . . . . : Microsoft ISATAP Adapter Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes Tunnel adapter isatap.{387B8C0A-C757-46F0-A319-4E4EE4163740}: Media State . . . . . . . . . . . : Media disconnected Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2 Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes C:\Windows\system32>ping 192.168.43.6 Pinging 192.168.43.6 with 32 bytes of data: Reply from 192.168.43.6: bytes=32 time<1ms TTL=64 Reply from 192.168.43.6: bytes=32 time<1ms TTL=64 Reply from 192.168.43.6: bytes=32 time<1ms TTL=64 Reply from 192.168.43.6: bytes=32 time<1ms TTL=64 Ping statistics for 192.168.43.6: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), C:\Windows\system32>ping 192.168.43.101 Pinging 192.168.43.101 with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Ping statistics for 192.168.43.101: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), C:\Windows\system32> after step6, guest is hung, see attachment. Expected results: Network info is 82599ES after migration. And ping both src and dst host ip can succeed. Additional info: After exchanging src and dst nic, network of guest works well after migration, no the issue.
I think this must be debugged on windows side first to understand what happens.
Yuri, I think this BZ is for you too (like BZ2090712)
Failover vf migration is only supported in RHV and it is technical preview. It is not supported in OSP and CNV. So set the priority to medium. If anything wrong, please correct me.
Should be fixed in https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=53700760
It works well with following package versions and "-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off". qemu-kvm-7.2.0-14.el9_2.x86_64 virtio-win driver: 100.93.104.23900 C:\Windows\system32>ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : WIN-U33AVC7B2CM Primary Dns Suffix . . . . . . . : Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : lab.eng.pek2.redhat.com Ethernet adapter Ethernet 3: Media State . . . . . . . . . . . : Media disconnected Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #3 Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Ethernet adapter Ethernet 2: Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2 Physical Address. . . . . . . . . : 52-54-00-01-16-16 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IPv6 Address. . . . . . . . . . . : 2620:52:0:49d2:f5df:637a:2f63:4a2f(Preferred) Link-local IPv6 Address . . . . . : fe80::f5df:637a:2f63:4a2f%6(Preferred) IPv4 Address. . . . . . . . . . . : 10.73.211.168(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.254.0 Lease Obtained. . . . . . . . . . : Friday, July 21, 2023 4:37:45 AM Lease Expires . . . . . . . . . . : Friday, July 21, 2023 11:36:30 PM Default Gateway . . . . . . . . . : fe80::52c7:903:533b:88e1%6 10.73.211.254 DHCP Server . . . . . . . . . . . : 10.73.2.108 DHCPv6 IAID . . . . . . . . . . . : 122835968 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11 DNS Servers . . . . . . . . . . . : 10.72.17.5 10.68.5.26 NetBIOS over Tcpip. . . . . . . . : Enabled Ethernet adapter Ethernet 7: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Intel(R) 82599 Virtual Function Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::393e:afea:c449:cccd%11(Preferred) IPv4 Address. . . . . . . . . . . : 192.168.53.200(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . . . . : Friday, July 21, 2023 11:39:03 AM Lease Expires . . . . . . . . . . : Friday, July 21, 2023 11:49:03 AM Default Gateway . . . . . . . . . : 192.168.53.2 DHCP Server . . . . . . . . . . . : 192.168.53.6 DHCPv6 IAID . . . . . . . . . . . : 324162560 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11 DNS Servers . . . . . . . . . . . : 192.168.53.2 NetBIOS over Tcpip. . . . . . . . : Enabled Tunnel adapter isatap.{DD6493D0-1320-4D34-8038-6679E1D16381}: Media State . . . . . . . . . . . : Media disconnected Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Microsoft ISATAP Adapter Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes Tunnel adapter Teredo Tunneling Pseudo-Interface: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes IPv6 Address. . . . . . . . . . . : 2001:0:348b:fb58:143c:3a72:f5b6:2c57(Preferred) Link-local IPv6 Address . . . . . : fe80::143c:3a72:f5b6:2c57%3(Preferred) Default Gateway . . . . . . . . . : DHCPv6 IAID . . . . . . . . . . . : 134217728 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11 NetBIOS over Tcpip. . . . . . . . : Disabled Tunnel adapter isatap.lab.eng.pek2.redhat.com: Media State . . . . . . . . . . . : Media disconnected Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2 Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes C:\Windows\system32>ping 192.168.53.6 Pinging 192.168.53.6 with 32 bytes of data: Reply from 192.168.53.6: bytes=32 time<1ms TTL=64 Reply from 192.168.53.6: bytes=32 time<1ms TTL=64 Reply from 192.168.53.6: bytes=32 time<1ms TTL=64 Reply from 192.168.53.6: bytes=32 time<1ms TTL=64 Ping statistics for 192.168.53.6: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms C:\Windows\system32>ping 192.168.53.101 Pinging 192.168.53.101 with 32 bytes of data: Reply from 192.168.53.101: bytes=32 time<1ms TTL=64 Reply from 192.168.53.101: bytes=32 time<1ms TTL=64 Reply from 192.168.53.101: bytes=32 time<1ms TTL=64 Reply from 192.168.53.101: bytes=32 time<1ms TTL=64 Ping statistics for 192.168.53.101: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms C:\Windows\system32>
According to comment 5, first pre-verified the bug. Here is the new bug with 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'. Bug 2224964 - after windows guest failover vf migration with non-intel adapters, the network can't work, unless adding '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'