Bug 2090712
| Summary: | the network in win2016/win2012r2 guest can't work after failover vf migraion between QL41000 and SFC9220 network cards | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Yanhui Ma <yama> |
| Component: | virtio-win | Assignee: | ybendito |
| virtio-win sub component: | virtio-win-prewhql | QA Contact: | Virtualization Bugs <virt-bugs> |
| Status: | CLOSED MIGRATED | Docs Contact: | Daniel Vozenilek <davozeni> |
| Severity: | medium | ||
| Priority: | medium | CC: | chayang, coli, jherrman, jinzhao, juzhang, lvivier, qizhu, virt-maint, yalzhang, yanghliu, ybendito, ymankad, yvugenfi |
| Version: | 9.1 | Keywords: | MigratedToJIRA, Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Windows | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Known Issue | |
| Doc Text: |
.Broadcom network adapters work incorrectly on Windows VMs after a live migration
Currently, network adapters from the Broadcom family of devices, such as Broadcom, Qlogic, or Marvell, cannot be hot-unplugged during live migration of Windows virtual machines (VMs). As a consequence, the adapters work incorrectly after the migration is complete.
This problem affects only those adapters that are attached to Windows VMs using Single-root I/O virtualization (SR-IOV).
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-06-28 08:24:58 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
For win2022 guest, after step 5, the results are the same. But when rebooting the guest or scan the hardware changes via Device manager, the network will works well. Move to sst_virtualization_windows pool as the problem occurs only with windows guest. (In reply to Laurent Vivier from comment #4) > Move to sst_virtualization_windows pool as the problem occurs only with > windows guest. Thank you Laurent! BTW do you know who works on windows guest related failover vf migration? (In reply to Yanhui Ma from comment #5) > (In reply to Laurent Vivier from comment #4) > > Move to sst_virtualization_windows pool as the problem occurs only with > > windows guest. > > Thank you Laurent! BTW do you know who works on windows guest related > failover vf migration? It seems the hotplug of the card is not detected (or done?) Perhaps Yuri can help? Hi I have opened a bug about QL41112 + Win2022 vm before. Please check if it has the same root cause with this bug. Bug 2028349 - [Win2022_vm][qede] The "Device Manager" shows QL41112 VF is working properly but no relevant network configuration values can be displayed in “ipconfig" cmdlet The issue also can be reproduced with win2016 guest between source host BCM57810 nic and target host 82599ES nic. Ping will fail. After migration, the output of ipconfig /all is still BCM57810 info. And guest can't be rebooted. If exchanging source and target host, then no issue. Same as BZ#2091528 The way failover works now, protocol driver installation that is used to facilitate the binding in Windows guest needs to know exact PNP ID of the card to bind to. And this is a list that is part of the installation. We need to add additional NICs to the list on the first stage and open new BZ to work on generic mechanism to identify the card that should binded to virtio-net device Failover vf migration is only supported in RHV and it is technical preview. It is not supported in OSP and CNV. So set the priority to medium. If anything wrong, please correct me. (In reply to Yanhui Ma from comment #23) > Failover vf migration is only supported in RHV and it is technical preview. > It is not supported in OSP and CNV. So set the priority to medium. > If anything wrong, please correct me. I agree with the assessment. We have an additional failover bug set to high; I suggest moving it to medium too: BZ#2091528 - the network in win2016/win2022 guest can't work after failover vf migraion between MT2892 network cards (In reply to Yvugenfi from comment #24) > (In reply to Yanhui Ma from comment #23) > > Failover vf migration is only supported in RHV and it is technical preview. > > It is not supported in OSP and CNV. So set the priority to medium. > > If anything wrong, please correct me. > > I agree with the assessment. We have an additional failover bug set to high; > I suggest moving it to medium too: > BZ#2091528 - the network in win2016/win2022 guest can't work after failover > vf migraion between MT2892 network cards Done. Hi Yuri, The internal target release is rhel9.3.0, just want to confirm with you, will you plan to fix it in rhel9.3.0? If no, could you please remove the ITR? If yes, I will set a ITM for the bug. Thanks in advance. (In reply to Yanhui Ma from comment #26) > Hi Yuri, > > The internal target release is rhel9.3.0, just want to confirm with you, > will you plan to fix it in rhel9.3.0? > If no, could you please remove the ITR? If yes, I will set a ITM for the > bug. Thanks in advance. Upstream PR already sent: https://github.com/virtio-win/kvm-guest-drivers-windows/pull/933 We plan to merge downstream. It works with following package versions and '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
qemu-kvm-7.2.0-14.el9_2.x86_64
virtio-win driver:
100.93.104.23900
C:\Windows\system32>ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : WIN-U33AVC7B2CM
Primary Dns Suffix . . . . . . . :
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : lab.eng.pek2.redhat.com
Ethernet adapter Ethernet 3:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #3
Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Ethernet adapter Ethernet 2:
Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com
Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2
Physical Address. . . . . . . . . : 52-54-00-01-16-16
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IPv6 Address. . . . . . . . . . . : 2620:52:0:49d2:f5df:637a:2f63:4a2f(Preferred)
Link-local IPv6 Address . . . . . : fe80::f5df:637a:2f63:4a2f%6(Preferred)
IPv4 Address. . . . . . . . . . . : 10.73.211.168(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.254.0
Lease Obtained. . . . . . . . . . : Thursday, July 20, 2023 4:20:37 AM
Lease Expires . . . . . . . . . . : Thursday, July 20, 2023 11:11:43 PM
Default Gateway . . . . . . . . . : fe80::52c7:903:533b:88e1%6
10.73.211.254
DHCP Server . . . . . . . . . . . : 10.73.2.108
DHCPv6 IAID . . . . . . . . . . . : 122835968
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11
DNS Servers . . . . . . . . . . . : 10.72.17.5
10.68.5.26
NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter Ethernet 5:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Solarflare Flareon Ultra 8000 Series 10G Adapter
Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IPv6 Address. . . . . . . . . . . : 2001::ccf8:685e:b71f:697(Preferred)
Link-local IPv6 Address . . . . . : fe80::ccf8:685e:b71f:697%2(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.200.113(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : Thursday, July 20, 2023 11:13:24 AM
Lease Expires . . . . . . . . . . : Friday, July 21, 2023 11:13:23 AM
Default Gateway . . . . . . . . . : fe80::b2f9:63ff:feb8:6d37%2
192.168.200.254
DHCP Server . . . . . . . . . . . : 192.168.200.254
DHCPv6 IAID . . . . . . . . . . . : 273830912
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11
DNS Servers . . . . . . . . . . . : 192.168.200.253
NetBIOS over Tcpip. . . . . . . . : Enabled
Tunnel adapter isatap.{11A01082-D0D7-4A10-B26B-DDBD2369FB48}:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Microsoft ISATAP Adapter
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Tunnel adapter Teredo Tunneling Pseudo-Interface:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv6 Address. . . . . . . . . . . : 2001:0:348b:fb58:20db:163c:f5b6:2c57(Preferred)
Link-local IPv6 Address . . . . . : fe80::20db:163c:f5b6:2c57%3(Preferred)
Default Gateway . . . . . . . . . :
DHCPv6 IAID . . . . . . . . . . . : 134217728
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11
NetBIOS over Tcpip. . . . . . . . : Disabled
Tunnel adapter isatap.lab.eng.pek2.redhat.com:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com
Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
C:\Windows\system32>ping 192.168.200.103
Pinging 192.168.200.103 with 32 bytes of data:
Reply from 192.168.200.103: bytes=32 time<1ms TTL=64
Reply from 192.168.200.103: bytes=32 time<1ms TTL=64
Reply from 192.168.200.103: bytes=32 time<1ms TTL=64
Reply from 192.168.200.103: bytes=32 time<1ms TTL=64
Ping statistics for 192.168.200.103:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
C:\Windows\system32>ping 192.168.200.65
Pinging 192.168.200.65 with 32 bytes of data:
Reply from 192.168.200.65: bytes=32 time<1ms TTL=64
Reply from 192.168.200.65: bytes=32 time<1ms TTL=64
Reply from 192.168.200.65: bytes=32 time<1ms TTL=64
Reply from 192.168.200.65: bytes=32 time<1ms TTL=64
Ping statistics for 192.168.200.65:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
According to comment 28, first pre-verified the bug. Here is the new bug with 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'. Bug 2224964 - after windows guest failover vf migration with non-intel adapters, the network can't work, unless adding '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' |
Description of problem: After live migrating win2016/win2012r2 guest with failover vf between QL41000 and SFC9220 network cards separately on src and dst hosts, ping will fail in windows guest and the network info of Device manager is src netowrk card info, not the dst network card info. Version-Release number of selected component (if applicable): # rpm -q qemu-kvm qemu-kvm-7.0.0-3.el9.x86_64 # uname -r 5.14.0-92.el9.x86_64 src host nic info FastLinQ QL41000 Series 10/25/40/50GbE Controller dst host nic info: SFC9220 10/40G Ethernet Controller How reproducible: 100% Steps to Reproduce: 1.create vf on both src host and dst host echo 1 > /sys/bus/pci/devices/0000\:1a\:00.1/sriov_numvfs 2. create failover-vf and failover-bridge network on both src and dst host # virsh net-dumpxml failover-bridge <network connections='1'> <name>failover-bridge</name> <uuid>1943a508-b0b7-4274-be5a-6f0143d10f40</uuid> <forward mode='bridge'/> <bridge name='br0'/> </network> # virsh net-dumpxml failover-vf <network connections='1'> <name>failover-vf</name> <uuid>4319b666-8f4b-410a-886f-17b6df772224</uuid> <forward mode='hostdev' managed='yes'> <address type='pci' domain='0x0000' bus='0x1a' slot='0x08' function='0x2'/> </forward> </network> 3. boot win216/win2012r2 guest with failover vf on src host <interface type='network'> <mac address='52:54:00:aa:1c:ef'/> <source network='failover-bridge'/> <model type='virtio'/> <teaming type='persistent'/> <alias name='ua-test'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:aa:1c:ef'/> <source network='failover-vf'/> <teaming type='transient' persistent='ua-test'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </interface> 4. live migrating the guest 5. after migration, check the network in guest 6. reboot the guest or scan the hardware changes via Device manager in guest Actual results: After step5, after migration, ping will fail and network info of vf is QLogic FastLinQ QL41112-DE 10GbE Adapter(src nic info), not the Solarflare Flareon Ultra 8000 Series 10G Adapter(dst nic info). ping 192.168.200.254 Pinging 192.168.200.254 with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Ping statistics for 192.168.200.254: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), C:\Windows\system32>ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : WIN-GB4G3MJFSG1 Primary Dns Suffix . . . . . . . : Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : lab.eng.pek2.redhat.com Ethernet adapter Ethernet Instance 0 3: Media State . . . . . . . . . . . : Media disconnected Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #3 Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Ethernet adapter Ethernet Instance 0 2: Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2 Physical Address. . . . . . . . . : 52-54-00-01-16-16 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IPv6 Address. . . . . . . . . . . : 2620:52:0:49d2:70c4:4922:e7f2:78b(Preferred) Link-local IPv6 Address . . . . . : fe80::70c4:4922:e7f2:78b%4(Preferred) IPv4 Address. . . . . . . . . . . : 10.73.211.161(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.254.0 Lease Obtained. . . . . . . . . . : Thursday, May 26, 2022 6:35:05 PM Lease Expires . . . . . . . . . . : Friday, May 27, 2022 6:35:04 AM Default Gateway . . . . . . . . . : fe80::52c7:903:533b:88e1%4 10.73.211.254 DHCP Server . . . . . . . . . . . : 10.73.2.108 DHCPv6 IAID . . . . . . . . . . . : 106058752 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2A-1D-60-4C-9A-1B-C6-8F-35-F3 DNS Servers . . . . . . . . . . . : 10.73.2.107 10.73.2.108 10.66.127.10 NetBIOS over Tcpip. . . . . . . . : Enabled Ethernet adapter Ethernet Instance 0 5: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : QLogic FastLinQ QL41112-DE 10GbE Adapter (VBD Client SR-IOV VF) Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::8dd9:4cf7:421:cdd8%15(Preferred) IPv4 Address. . . . . . . . . . . : 192.168.200.26(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . . . . : Thursday, May 26, 2022 6:35:33 PM Lease Expires . . . . . . . . . . : Friday, May 27, 2022 6:35:32 PM Default Gateway . . . . . . . . . : 192.168.200.254 DHCP Server . . . . . . . . . . . : 192.168.200.254 DHCPv6 IAID . . . . . . . . . . . : 273830912 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2A-1D-60-4C-9A-1B-C6-8F-35-F3 DNS Servers . . . . . . . . . . . : 192.168.200.253 NetBIOS over Tcpip. . . . . . . . : Enabled Tunnel adapter Teredo Tunneling Pseudo-Interface: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes IPv6 Address. . . . . . . . . . . : 2001:0:34f1:8072:383d:20ca:f5b6:2c5e(Preferred) Link-local IPv6 Address . . . . . : fe80::383d:20ca:f5b6:2c5e%2(Preferred) Default Gateway . . . . . . . . . : DHCPv6 IAID . . . . . . . . . . . : 134217728 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2A-1D-60-4C-9A-1B-C6-8F-35-F3 NetBIOS over Tcpip. . . . . . . . : Disabled Tunnel adapter Reusable ISATAP Interface {9EF5D6D7-56AD-44E7-AF94-F90258691408}: Media State . . . . . . . . . . . : Media disconnected Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2 Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes Tunnel adapter isatap.lab.eng.pek2.redhat.com: Media State . . . . . . . . . . . : Media disconnected Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com Description . . . . . . . . . . . : Microsoft ISATAP Adapter #4 Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes After step 6, the guest can't be rebooted, and the scan for hardware changes will not quit. Pls see attachments. Expected results: After migration, the network should works well in windows guest. Additional info: There is not the issue for RHEL guest.