Bug 1912088

Summary: [netkvm][failover] netkvm adapter should have valid IP address after installation of VF bonding protocol
Product: Red Hat Enterprise Linux 8 Reporter: ybendito
Component: virtio-winAssignee: ybendito
virtio-win sub component: virtio-win-prewhql QA Contact: Yanhui Ma <yama>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: lijin, mdean, vrozenfe, yalzhang, yanghliu, ybendito, yvugenfi
Version: ---Keywords: Triaged
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Windows   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-16 09:59:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1950227    

Description ybendito 2021-01-03 10:31:35 UTC 
Description of problem:

Failover setup for Windows includes 2 adapters with the same MAC address (VF and netkvm). Typically only one of adapters has valid IP address before protocol installation. If the VF has obtained an IP address and netkvm did not, after installation of the protocol the netkvm adapter still does not have valid IP address and does not function correctly until the system is rebooted.

Version-Release number of selected component (if applicable):

virtio-win-prewhql-0.1-191

How reproducible:

Steps to Reproduce:
1. Bring up the failover setup with VF and netkvm before protocol installation. Ensure the VF has valid IP address and netkvm does not. Wait several minutes then install the protocol.
2. The installation of the protocol unbinds the VF from all the protocols except of VIOPROT and the VF does not an IP address anymore. The netkvm does not have an IP address also. Ping does not work.


Actual results:


Expected results:


Additional info:
Comment 1 Yanghang Liu 2021-01-25 11:49:54 UTC 
Hi Yuri,

There is a question that I need to confirm with you:

I just started a Win10 vm with a failover VF and a failover virtio net device[1].

At this time, the failover VF gets the IP address 169.254.X.X and the failover virtio net device gets a valid IP address.[2]  <--- The ip address that I obversed in the win10 vm is expected. Am I right?

But it seems that the ping does not work well in the win10 vm:

1.ping local host successfully
# ping 10.73.33.242

2.Fail to ping another host which is in the same network
# ping 10.73.33.244
Reply from 10.73.33.240: Destination host unreachable.

3.Fail to ping the external network
# ping redhat.com
Ping request could not find host redhat.com. Please check the name and try again.


Is the ping problem I observed is expected? Could I open a BZ for this ping problem?



The detailed info is as following:
[1] The device xml and qemu cmd line
    <interface type='bridge'>
      <mac address='f4:8e:38:c3:84:84'/>
      <source bridge='switch'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='f4:8e:38:c3:84:84'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x06' slot='0x10' function='0x0'/>
      </source>
      <teaming type='transient' persistent='net0'/>
    </interface>

-netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 \
-device virtio-net-pci,failover=on,netdev=hostnet0,id=net0,mac=f4:8e:38:c3:10:10,bus=pci.1,addr=0x0 \
-device vfio-pci,host=0000:06:10.0,id=hostdev0,bus=pci.4,addr=0x0,failover_pair_id=net0 \


[2] The ip address info in the win10 vm
C:\Windows\system32>ipconfig -all
...
Ethernet adapter Ethernet 7:

   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Intel(R) 82599 Virtual Function
   Physical Address. . . . . . . . . : F4-8E-38-C3-10-10
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:536:2561:835:c520(Preferred) 
   Temporary IPv6 Address. . . . . . : 2620:52:0:4920:99c4:91e0:9ea2:4249(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::536:2561:835:c520%6(Preferred) 
   Autoconfiguration IPv4 Address. . : 169.254.197.32(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%6
   DHCPv6 IAID . . . . . . . . . . . : 116690488
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-27-53-E9-9A-9A-F0-FC-06-AB-14
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet 8:

   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #5
   Physical Address. . . . . . . . . : F4-8E-38-C3-10-10
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:1579:c4a8:e140:f438(Preferred) 
   Temporary IPv6 Address. . . . . . : 2620:52:0:4920:f534:4f00:6f60:daf1(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::1579:c4a8:e140:f438%11(Preferred) 
   IPv4 Address. . . . . . . . . . . : 10.73.33.240(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.254.0
   Lease Obtained. . . . . . . . . . : Monday, January 25, 2021 8:35:45 AM
   Lease Expires . . . . . . . . . . : Monday, January 25, 2021 8:35:44 PM
   Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%11
                                       10.73.33.254
   DHCP Server . . . . . . . . . . . : 10.73.2.108
   DHCPv6 IAID . . . . . . . . . . . : 234131000
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-27-53-E9-9A-9A-F0-FC-06-AB-14
   DNS Servers . . . . . . . . . . . : 10.73.2.107
                                       10.73.2.108
                                       10.66.127.10
   NetBIOS over Tcpip. . . . . . . . : Enabled
Comment 2 ybendito 2021-01-25 13:34:00 UTC 
(In reply to Yanghang Liu from comment #1)
> Hi Yuri,
> 
> There is a question that I need to confirm with you:
> 
> I just started a Win10 vm with a failover VF and a failover virtio net
> device[1].
> 
> At this time, the failover VF gets the IP address 169.254.X.X and the
> failover virtio net device gets a valid IP address.[2]  <--- The ip address
> that I obversed in the win10 vm is expected. Am I right?
> 

This is not a proper state for this BZ.
This BZ is dedicated to specific problem exactly as defined in the description.
The VF starts first and gets an IP address, the virtio-net does not.
Upon protocol installation the virtio-net still stays without IP address.
So all the rest is not related to this BZ.
Comment 3 ybendito 2021-01-25 13:42:59 UTC 
There are 2 ways to get to the setup required for this BZ:
1. Start the VM with VF driver installed but without netkvm driver installed.
2. Even if both adapters have drivers installed: disable both adapters, then enable VF, then enable netkvm. Then wait, then install the protocol.
Comment 4 Yanghang Liu 2021-01-28 14:28:27 UTC 
Thanks for the explanation
Comment 5 Yanghang Liu 2021-03-12 09:30:35 UTC 
I can reproduce this problem in the following test env:

Test env:
4.18.0-295.el8.x86_64
qemu-kvm-5.2.0-11.module+el8.4.0+10268+62bcbbed.x86_64
virtio-win-prewhql-0.1-196




Test step:
(1) start a Win2012 vm with a failover 82599ES vf and a virtio net device

The domain xml:
    <interface type='bridge'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source network='failover-bridge' portid='c36c08da-f519-4c5f-a626-72dfda0e56e1' bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='ua-test'/>
    </interface>

    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:aa:1c:ef'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x06' slot='0x10' function='0x0'/>
      </source>
      <teaming type='transient' persistent='ua-test'/>
      <alias name='hostdev0'/>
    </interface>

The qemu cmd line:
-netdev tap,fd=39,id=hostua-test,vhost=on,vhostfd=40
-device virtio-net-pci,failover=on,netdev=hostua-test,id=ua-test,mac=52:54:00:aa:1c:ef,bus=pci.1,addr=0x0
-device vfio-pci,host=0000:06:10.0,id=hostdev0,bus=pci.4,addr=0x0,failover_pair_id=ua-test

(2) make sure the vf has a valid ip address and the virtio net device has an invalid ip address

C:\Windows\system32>ipconfig -all

Ethernet adapter Ethernet 2:

   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:8124:5566:ee1e:e2cf(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::8124:5566:ee1e:e2cf%23(Preferred) 
   Autoconfiguration IPv4 Address. . : 169.254.226.207(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%23
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet 6:

   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Intel(R) 82599 Virtual Function
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:39a2:7f6d:2723:1582(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::39a2:7f6d:2723:1582%19(Preferred) 
   IPv4 Address. . . . . . . . . . . : 10.73.33.220(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.254.0
   Lease Obtained. . . . . . . . . . : Friday, March 12, 2021 9:24:20 AM
   Lease Expires . . . . . . . . . . : Friday, March 12, 2021 9:24:19 PM
   Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%19
                                       10.73.33.254
   DHCP Server . . . . . . . . . . . : 10.73.2.108
   DNS Servers . . . . . . . . . . . : 10.73.2.107
                                       10.73.2.108
                                       10.66.127.10
   NetBIOS over Tcpip. . . . . . . . : Enabled




(3) install the  VIOPROT protocol

C:\Users\Administrator\Desktop\virtio-win-prewhql-0.1-196\Win8\amd64>netcfg -v -l vioprot.inf -c p -i VIOPROT
Trying to install VIOPROT ...

... vioprot.inf was copied to C:\Windows\INF\oem23.inf.

... done.


(4) check the network info again

The virtio net device has an invalid ip address 169.254.X.X at this time.

C:\Windows\system32>ipconfig -all

Ethernet adapter Ethernet 2:

   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:8124:5566:ee1e:e2cf(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::8124:5566:ee1e:e2cf%23(Preferred) 
   Autoconfiguration IPv4 Address. . : 169.254.226.207(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%23
   DNS Servers . . . . . . . . . . . : 10.73.2.107
                                       10.73.2.108
                                       10.66.127.10
   NetBIOS over Tcpip. . . . . . . . : Enabled
Comment 6 ybendito 2021-04-05 06:49:25 UTC 
Merged upstream https://github.com/virtio-win/kvm-guest-drivers-windows/pull/563
Comment 8 Yanghang Liu 2021-04-21 10:43:03 UTC 
Test host env:
qemu-kvm-5.2.0-15.module+el8.4.0+10650+50781ca0.x86_64
4.18.0-304.el8.x86_64
virtio-win-prewhql-0.1-198
82599ES VF driver:https://downloadcenter.intel.com/download/22283



Test vm:
Windows Sever 2012r2



Test step:
(1) start a Windows Sever 2012r2 vm with a failover virtio net device and a failover 82599ES VF
The domain xml:
    <interface type='bridge'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='ua-test'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:aa:1c:ef'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x06' slot='0x10' function='0x0'/>
      </source>
      <teaming type='transient' persistent='ua-test'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

The qemu cmd line:
-netdev tap,fd=39,id=hostua-test,vhost=on,vhostfd=40 \
-device virtio-net-pci,failover=on,netdev=hostua-test,id=ua-test,mac=52:54:00:aa:1c:ef,bus=pci.1,addr=0x0 \
-device vfio-pci,host=0000:06:10.0,id=hostdev0,bus=pci.4,addr=0x0,failover_pair_id=ua-test \

(2) check the ip address of the failover virtio net device and the failover vf in the domain

C:\Windows\system32>ipconfig -all

Ethernet adapter Ethernet 4:

   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Intel(R) 82599 Virtual Function
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:997:fa89:d8df:59cb(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::997:fa89:d8df:59cb%25(Preferred) 
   IPv4 Address. . . . . . . . . . . : 10.73.33.134(Preferred)   <--- the failover vf has a valid ip address
   Subnet Mask . . . . . . . . . . . : 255.255.254.0
   Lease Obtained. . . . . . . . . . : Wednesday, April 21, 2021 8:48:24 AM
   Lease Expires . . . . . . . . . . : Wednesday, April 21, 2021 8:48:24 PM
   Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%25
                                       10.73.33.254
   DHCP Server . . . . . . . . . . . : 10.73.2.108
   DHCPv6 IAID . . . . . . . . . . . : 424825856
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-28-0F-34-72-9A-05-AC-8E-39-90
   DNS Servers . . . . . . . . . . . : 10.73.2.107
                                       10.73.2.108
                                       10.66.127.10
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet 3:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : 
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2  <--- the failover virtio net device does not have any ip address
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes


(3) install the  VIOPROT protocol

C:\Users\Administrator\Desktop\virtio-win-prewhql-0.1-198\Win8.1\amd64>netcfg -v -l vioprot.inf -c p -i VIOPROT
Trying to install VIOPROT ...

... vioprot.inf was copied to C:\Windows\INF\oem10.inf.

... done.


(4)  wait a moment and check the network status again

C:\Users\Administrator\Desktop\virtio-win-prewhql-0.1-198\Win8.1\amd64>ipconfig -all

Ethernet adapter Ethernet 3:

   Connection-specific DNS Suffix  . : lab.eng.pek2.redhat.com
   Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2
   Physical Address. . . . . . . . . : 52-54-00-AA-1C-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:25d3:c106:3122:90a6(Preferred) 
   Link-local IPv6 Address . . . . . : fe80::25d3:c106:3122:90a6%21(Preferred) 
   IPv4 Address. . . . . . . . . . . : 10.73.33.134(Preferred)  <---- the failover virtio net device has a valid ip address now
   Subnet Mask . . . . . . . . . . . : 255.255.254.0
   Lease Obtained. . . . . . . . . . : Wednesday, April 21, 2021 9:39:25 AM
   Lease Expires . . . . . . . . . . : Wednesday, April 21, 2021 9:39:25 PM
   Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%21
                                       10.73.33.254
   DHCP Server . . . . . . . . . . . : 10.73.2.108
   DHCPv6 IAID . . . . . . . . . . . : 357716992
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-28-0F-34-72-9A-05-AC-8E-39-90
   DNS Servers . . . . . . . . . . . : 10.73.2.107
                                       10.73.2.108
                                       10.66.127.10
   NetBIOS over Tcpip. . . . . . . . : Enabled

(5) do some ping tests

(5.1) ping external network successfully
C:\Users\Administrator\Desktop\virtio-win-prewhql-0.1-198\Win8.1\amd64>ping redhat.com

Pinging redhat.com [10.4.204.55] with 32 bytes of data:
Reply from 10.4.204.55: bytes=32 time=251ms TTL=248
Reply from 10.4.204.55: bytes=32 time=252ms TTL=248
Reply from 10.4.204.55: bytes=32 time=251ms TTL=248
Reply from 10.4.204.55: bytes=32 time=252ms TTL=248

(5.2) ping the source bridge on the local host successfully

C:\Users\Administrator\Desktop\virtio-win-prewhql-0.1-198\Win8.1\amd64>ping 10.73.33.202

Pinging 10.73.33.202 with 32 bytes of data:
Reply from 10.73.33.202: bytes=32 time=1ms TTL=64
Reply from 10.73.33.202: bytes=32 time<1ms TTL=64
Reply from 10.73.33.202: bytes=32 time<1ms TTL=64
Reply from 10.73.33.202: bytes=32 time<1ms TTL=64

Ping statistics for 10.73.33.202:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms
Comment 9 Yanghang Liu 2021-04-21 10:57:31 UTC 
Use the Win10(20H2) vm to repeat the step 1 - step 5 in comment 8 and get the same test result.
Comment 10 Yanghang Liu 2021-04-22 02:45:31 UTC 
Hi Yuri,

Could you please help check the test result in the comment 8 and comment 9 ?

Is it enough for QE to verify this bug?

If you have any concerns and need QE to do more tests, please feel free to let me know.

Thanks a lot for your help in advance.
Comment 11 ybendito 2021-04-22 05:43:31 UTC 
(In reply to Yanghang Liu from comment #10)
> Hi Yuri,
> 
> Could you please help check the test result in the comment 8 and comment 9 ?
> 
> Is it enough for QE to verify this bug?
> 

In general this is enough to verify.

> If you have any concerns and need QE to do more tests, please feel free to
> let me know.
> 

Yes, I want more tests without specific relation to this BZ.
We need to sync on testing methodology of failover under Windows, there we have more flows than under Linux:
- disable/enable flows
- install/uninstall protocol flows
- driver update flows

I think I need to create dedicated BZ for test plan of failover feature (like a tracker for all the failover issues) where we'll coordinate the test procedure for different flows/use cases related to this feature
Comment 12 Yanghang Liu 2021-04-25 07:00:24 UTC 
According to comment 8, comment 9 and comment 11 , move the bug status to VERIFIED.
Comment 13 Yanghang Liu 2021-04-25 07:04:49 UTC 
(In reply to ybendito from comment #11)

> Yes, I want more tests without specific relation to this BZ.
> We need to sync on testing methodology of failover under Windows, there we
> have more flows than under Linux:
> - disable/enable flows
> - install/uninstall protocol flows
> - driver update flows
> 
> I think I need to create dedicated BZ for test plan of failover feature
> (like a tracker for all the failover issues) where we'll coordinate the test
> procedure for different flows/use cases related to this feature

Related bug:
Bug 1953171 - [netkvm] SRIOV failover: Align test plan with the SW design under Windows