2090712 – the network in win2016/win2012r2 guest can't work after failover vf migraion between QL41000 and SFC9220 network cards

This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2090712 - the network in win2016/win2012r2 guest can't work after failover vf migraion between QL41000 and SFC9220 network cards

Summary: the network in win2016/win2012r2 guest can't work after failover vf migraion ...

Keywords:
Status:	CLOSED MIGRATED
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	virtio-win
Sub Component:
Version:	9.1
Hardware:	x86_64
OS:	Windows
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	ybendito
QA Contact:	Virtualization Bugs
Docs Contact:	Daniel Vozenilek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-05-26 12:21 UTC by Yanhui Ma
Modified:	2023-09-26 15:27 UTC (History)
CC List:	13 users (show)
Fixed In Version:
Doc Type:	Known Issue
Doc Text:	.Broadcom network adapters work incorrectly on Windows VMs after a live migration Currently, network adapters from the Broadcom family of devices, such as Broadcom, Qlogic, or Marvell, cannot be hot-unplugged during live migration of Windows virtual machines (VMs). As a consequence, the adapters work incorrectly after the migration is complete. This problem affects only those adapters that are attached to Windows VMs using Single-root I/O virtualization (SR-IOV).
Clone Of:
Environment:
Last Closed:	2023-06-28 08:24:58 UTC
Type:	---
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHEL-910	0	None	None	None	2023-09-26 15:27:44 UTC
Red Hat Issue Tracker	RHELPLAN-123470	0	None	None	None	2023-09-26 15:27:44 UTC

Description Yanhui Ma 2022-05-26 12:21:00 UTC

Description of problem:

After live migrating win2016/win2012r2 guest with failover vf between QL41000 and SFC9220 network cards separately on src and dst hosts, ping will fail in windows guest and the network info of Device manager is src netowrk card info, not the dst network card info.

Version-Release number of selected component (if applicable):

# rpm -q qemu-kvm
qemu-kvm-7.0.0-3.el9.x86_64
# uname -r
5.14.0-92.el9.x86_64

src host nic info
FastLinQ QL41000 Series 10/25/40/50GbE Controller

dst host nic info:
SFC9220 10/40G Ethernet Controller


How reproducible:

100%

Steps to Reproduce:
1.create vf on both src host and dst host

echo 1 > /sys/bus/pci/devices/0000\:1a\:00.1/sriov_numvfs

2. create failover-vf and failover-bridge network on both src and dst host

# virsh net-dumpxml failover-bridge 
<network connections='1'>
  <name>failover-bridge</name>
  <uuid>1943a508-b0b7-4274-be5a-6f0143d10f40</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

# virsh net-dumpxml failover-vf
<network connections='1'>
  <name>failover-vf</name>
  <uuid>4319b666-8f4b-410a-886f-17b6df772224</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x1a' slot='0x08' function='0x2'/>
  </forward>
</network>


3. boot win216/win2012r2 guest with failover vf on src host
    <interface type='network'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source network='failover-bridge'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='ua-test'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source network='failover-vf'/>
      <teaming type='transient' persistent='ua-test'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </interface>

4. live migrating the guest
5. after migration, check the network in guest
6. reboot the guest or scan the hardware changes via Device manager in guest

Actual results:
After step5, after migration, ping will fail and network info of vf is 
QLogic FastLinQ QL41112-DE 10GbE Adapter(src nic info), not the Solarflare Flareon Ultra 8000 Series 10G Adapter(dst nic info).


ping 192.168.200.254

Pinging 192.168.200.254 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 192.168.200.254:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Windows\system32>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : WIN-GB4G3MJFSG1
   Primary Dns Suffix  . . . . . . . : 
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : lab.eng.pek2.redhat.com

Ethernet adapter Ethernet Instance 0 3:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #3
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes

Ethernet adapter Ethernet Instance 0 2:

   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2
   Physical Address. . . . . . . . . : 52-54-00-01-16-16
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:49d2:70c4:4922:e7f2:78b(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::70c4:4922:e7f2:78b%4(Preferred) 
   IPv4 Address. . . . . . . . . . . : 10.73.211.161(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.254.0
   Lease Obtained. . . . . . . . . . : Thursday, May 26, 2022 6:35:05 PM
   Lease Expires . . . . . . . . . . : Friday, May 27, 2022 6:35:04 AM
   Default Gateway . . . . . . . . . : fe80::52c7:903:533b:88e1%4
                                       10.73.211.254
   DHCP Server . . . . . . . . . . . : 10.73.2.108
   DHCPv6 IAID . . . . . . . . . . . : 106058752
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2A-1D-60-4C-9A-1B-C6-8F-35-F3
   DNS Servers . . . . . . . . . . . : 10.73.2.107
                                       10.73.2.108
                                       10.66.127.10
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet Instance 0 5:

   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : QLogic FastLinQ QL41112-DE 10GbE Adapter (VBD Client SR-IOV VF)
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::8dd9:4cf7:421:cdd8%15(Preferred) 
   IPv4 Address. . . . . . . . . . . : 192.168.200.26(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : Thursday, May 26, 2022 6:35:33 PM
   Lease Expires . . . . . . . . . . : Friday, May 27, 2022 6:35:32 PM
   Default Gateway . . . . . . . . . : 192.168.200.254
   DHCP Server . . . . . . . . . . . : 192.168.200.254
   DHCPv6 IAID . . . . . . . . . . . : 273830912
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2A-1D-60-4C-9A-1B-C6-8F-35-F3
   DNS Servers . . . . . . . . . . . : 192.168.200.253
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2001:0:34f1:8072:383d:20ca:f5b6:2c5e(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::383d:20ca:f5b6:2c5e%2(Preferred) 
   Default Gateway . . . . . . . . . : 
   DHCPv6 IAID . . . . . . . . . . . : 134217728
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2A-1D-60-4C-9A-1B-C6-8F-35-F3
   NetBIOS over Tcpip. . . . . . . . : Disabled

Tunnel adapter Reusable ISATAP Interface {9EF5D6D7-56AD-44E7-AF94-F90258691408}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.lab.eng.pek2.redhat.com:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #4
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes


After step 6, the guest can't be rebooted, and the scan for hardware changes will not quit. Pls see attachments.

Expected results:
After migration, the network should works well in windows guest.

Additional info:

There is not the issue for RHEL guest.

Comment 3 Yanhui Ma 2022-05-26 14:01:07 UTC

For win2022 guest, after step 5, the results are the same. But when rebooting  the guest or scan the hardware changes via Device manager, the network will works well.

Comment 4 Laurent Vivier 2022-05-31 07:25:44 UTC

Move to sst_virtualization_windows pool as the problem occurs only with windows guest.

Comment 5 Yanhui Ma 2022-05-31 11:12:02 UTC

(In reply to Laurent Vivier from comment #4)
> Move to sst_virtualization_windows pool as the problem occurs only with
> windows guest.

Thank you Laurent! BTW do you know who works on windows guest related failover vf migration?

Comment 6 Laurent Vivier 2022-05-31 15:12:35 UTC

(In reply to Yanhui Ma from comment #5)
> (In reply to Laurent Vivier from comment #4)
> > Move to sst_virtualization_windows pool as the problem occurs only with
> > windows guest.
> 
> Thank you Laurent! BTW do you know who works on windows guest related
> failover vf migration?

It seems the hotplug of the card is not detected (or done?)

Perhaps Yuri can help?

Comment 7 Yanghang Liu 2022-06-02 10:26:35 UTC

Hi

I have opened a bug about  QL41112 + Win2022 vm before.  Please check if it has the same root cause with this bug.

Bug 2028349 - [Win2022_vm][qede] The "Device Manager" shows QL41112 VF is working properly but no relevant network configuration values can be displayed in “ipconfig" cmdlet

Comment 8 Yanhui Ma 2022-06-20 07:35:15 UTC

The issue also can be reproduced with win2016 guest between source host BCM57810 nic and target host 82599ES nic.
Ping will fail. After migration, the output of ipconfig /all is still BCM57810 info. And guest can't be rebooted.

If exchanging source and target host, then no issue.

Comment 9 Yvugenfi@redhat.com 2022-06-23 09:37:22 UTC

Same as BZ#2091528

The way failover works now, protocol driver installation that is used to facilitate the binding in Windows guest needs to know exact PNP ID of the card to bind to. And this is a list that is part of the installation. 
We need to add additional NICs to the list on the first stage and open new BZ to work on generic mechanism to identify the card that should binded to virtio-net device

Comment 23 Yanhui Ma 2023-03-24 04:36:07 UTC

Failover vf migration is only supported in RHV and it is technical preview. It is not supported in OSP and CNV. So set the priority to medium.
If anything wrong, please correct me.

Comment 24 Yvugenfi@redhat.com 2023-03-26 08:47:29 UTC

(In reply to Yanhui Ma from comment #23)
> Failover vf migration is only supported in RHV and it is technical preview.
> It is not supported in OSP and CNV. So set the priority to medium.
> If anything wrong, please correct me.

I agree with the assessment. We have an additional failover bug set to high; I suggest moving it to medium too:
BZ#2091528 - the network in win2016/win2022 guest can't work after failover vf migraion between MT2892 network cards

Comment 25 Yanhui Ma 2023-03-27 08:30:09 UTC

(In reply to Yvugenfi from comment #24)
> (In reply to Yanhui Ma from comment #23)
> > Failover vf migration is only supported in RHV and it is technical preview.
> > It is not supported in OSP and CNV. So set the priority to medium.
> > If anything wrong, please correct me.
> 
> I agree with the assessment. We have an additional failover bug set to high;
> I suggest moving it to medium too:
> BZ#2091528 - the network in win2016/win2022 guest can't work after failover
> vf migraion between MT2892 network cards

Done.

Comment 26 Yanhui Ma 2023-06-05 02:26:43 UTC

Hi Yuri,

The internal target release is rhel9.3.0, just want to confirm with you, will you plan to fix it in rhel9.3.0?
If no, could you please remove the ITR? If yes, I will set a ITM for the bug. Thanks in advance.

Comment 27 Yvugenfi@redhat.com 2023-06-28 08:24:27 UTC

(In reply to Yanhui Ma from comment #26)
> Hi Yuri,
> 
> The internal target release is rhel9.3.0, just want to confirm with you,
> will you plan to fix it in rhel9.3.0?
> If no, could you please remove the ITR? If yes, I will set a ITM for the
> bug. Thanks in advance.

Upstream PR already sent: https://github.com/virtio-win/kvm-guest-drivers-windows/pull/933
We plan to merge downstream.

Comment 28 Yanhui Ma 2023-07-20 04:26:36 UTC

It works with following package versions and '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'

qemu-kvm-7.2.0-14.el9_2.x86_64
virtio-win driver:
100.93.104.23900


C:\Windows\system32>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : WIN-U33AVC7B2CM
   Primary Dns Suffix  . . . . . . . : 
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : lab.eng.pek2.redhat.com

Ethernet adapter Ethernet 3:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #3
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes

Ethernet adapter Ethernet 2:

   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2
   Physical Address. . . . . . . . . : 52-54-00-01-16-16
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:49d2:f5df:637a:2f63:4a2f(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::f5df:637a:2f63:4a2f%6(Preferred) 
   IPv4 Address. . . . . . . . . . . : 10.73.211.168(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.254.0
   Lease Obtained. . . . . . . . . . : Thursday, July 20, 2023 4:20:37 AM
   Lease Expires . . . . . . . . . . : Thursday, July 20, 2023 11:11:43 PM
   Default Gateway . . . . . . . . . : fe80::52c7:903:533b:88e1%6
                                       10.73.211.254
   DHCP Server . . . . . . . . . . . : 10.73.2.108
   DHCPv6 IAID . . . . . . . . . . . : 122835968
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11
   DNS Servers . . . . . . . . . . . : 10.72.17.5
                                       10.68.5.26
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet 5:

   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Solarflare Flareon Ultra 8000 Series 10G Adapter
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2001::ccf8:685e:b71f:697(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::ccf8:685e:b71f:697%2(Preferred) 
   IPv4 Address. . . . . . . . . . . : 192.168.200.113(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : Thursday, July 20, 2023 11:13:24 AM
   Lease Expires . . . . . . . . . . : Friday, July 21, 2023 11:13:23 AM
   Default Gateway . . . . . . . . . : fe80::b2f9:63ff:feb8:6d37%2
                                       192.168.200.254
   DHCP Server . . . . . . . . . . . : 192.168.200.254
   DHCPv6 IAID . . . . . . . . . . . : 273830912
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11
   DNS Servers . . . . . . . . . . . : 192.168.200.253
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{11A01082-D0D7-4A10-B26B-DDBD2369FB48}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2001:0:348b:fb58:20db:163c:f5b6:2c57(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::20db:163c:f5b6:2c57%3(Preferred) 
   Default Gateway . . . . . . . . . : 
   DHCPv6 IAID . . . . . . . . . . . : 134217728
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-47-5F-53-9A-E9-2D-4B-32-11
   NetBIOS over Tcpip. . . . . . . . : Disabled

Tunnel adapter isatap.lab.eng.pek2.redhat.com:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

C:\Windows\system32>ping 192.168.200.103

Pinging 192.168.200.103 with 32 bytes of data:
Reply from 192.168.200.103: bytes=32 time<1ms TTL=64
Reply from 192.168.200.103: bytes=32 time<1ms TTL=64
Reply from 192.168.200.103: bytes=32 time<1ms TTL=64
Reply from 192.168.200.103: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.200.103:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Windows\system32>ping 192.168.200.65

Pinging 192.168.200.65 with 32 bytes of data:
Reply from 192.168.200.65: bytes=32 time<1ms TTL=64
Reply from 192.168.200.65: bytes=32 time<1ms TTL=64
Reply from 192.168.200.65: bytes=32 time<1ms TTL=64
Reply from 192.168.200.65: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.200.65:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

Comment 29 Yanhui Ma 2023-07-24 05:36:18 UTC

According to comment 28, first pre-verified the bug. Here is the new bug with 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'.
Bug 2224964 - after windows guest failover vf migration with non-intel adapters, the network can't work, unless adding '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'

Note You need to log in before you can comment on or make changes to this bug.