Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1410076 - [SR-IOV] - in-guest bond with virtio+passthrough slave lose connectivity after hotunplug/hotplug of passthrough slave
Summary: [SR-IOV] - in-guest bond with virtio+passthrough slave lose connectivity afte...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.19.1
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ovirt-4.1.1
: 4.19.5
Assignee: Leon Goldberg
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On: 1341248
Blocks: 868811
TreeView+ depends on / blocked
 
Reported: 2017-01-04 12:07 UTC by Michael Burman
Modified: 2017-04-21 09:51 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-21 09:51:38 UTC
oVirt Team: Network
lgoldber: needinfo-
rule-engine: ovirt-4.1+
rule-engine: blocker+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 71029 0 master MERGED net: modify VF MAC addresses on creation to a valid address 2021-02-18 14:13:41 UTC
oVirt gerrit 71519 0 ovirt-4.1 MERGED net: Set and report device mac address through link.iface 2021-02-18 14:13:41 UTC
oVirt gerrit 71529 0 ovirt-4.1 MERGED net: modify VF MAC addresses on creation to a valid address 2021-02-18 14:13:47 UTC

Description Michael Burman 2017-01-04 12:07:12 UTC
Description of problem:
[SR-IOV] - Connectivity lost when migrating guest with active-backup bond over virtIo+passthrough interfaces.

As part of the sr-iov migration feature - http://www.ovirt.org/develop/release-management/features/network/liveMigrationSupportForSRIOV/ we creating a bond in the guest os using nmcli between virtIo+passthrough vNICs.

During the migration the virtIo vNIC should became the active one in order to keep connectivity during the migration. Once the passthrough vNIC is plugged back we have no connection and we can't get IP again. 


Version-Release number of selected component (if applicable):
vdsm-4.19.1-1.el7ev.x86_64

Steps to Reproduce:
1. Create bond mode 1 using nmcli:
 [1] - Edit the virtIo vNIC profile (eth0) with 'No Filter' filter
 [2]- Run VM with 2 nics. virtIo and sr-iov
 [3] - Remove the NM_CONTROLLED=no line from the ifcfg-eth0 file that generated via dracut installation
 [4] - nmcli con reload eth0 
 [5] - nmcli connection add type ethernet con-name ens3 ifname ens3
 [6] - nmcli connection add type bond con-name bond0 ifname bond0 mode active-    backup primary ens3
 [7] - nmcli connection modify id bond0 ipv4.method auto ipv6.method ignore
 [8] - nmcli connection modify id ens3 ipv4.method disabled ipv6.method ignore
 [9] - nmcli connection modify id eth0 ipv4.method disabled ipv6.method ignore
 [10] - nmcli connection modify id ens3 connection.slave-type bond connection.master bond0 connection.autoconnect yes
 [11] - nmcli connection modify id eth0 connection.slave-type bond connection.master bond0 connection.autoconnect yes
 [12] - nmcli connection down id ens3; nmcli con up id ens3; nmcli con down id eth0; nmcli con up id eth0; nmcli con up bond0

2. Ping the guest IP
3. Migrate the VM

Actual results:
No connection to the VM

Expected results:
Should work as expected.

Comment 1 Dan Kenigsberg 2017-01-04 14:26:44 UTC
I suspect that this is the same issue we see with hotunplug/hotplug (regardless of migration), and that it shows up - sometimes - even when NM is stopped and masked. right?

Comment 2 Michael Burman 2017-01-04 14:31:59 UTC
Correct.

Comment 3 Meni Yakove 2017-01-11 07:02:27 UTC
This only happens when the hosts have more than one VFs enabled and it's not related to plug/unplug vNIC.

It seems that if the vNIC get different VF (slot) we lost connection.

BOND created with: (Ping)
<interface type='hostdev'>
	  <mac address='00:1a:4a:16:20:76'/>
	  <driver name='vfio'/>
	  <source>
		<address type='pci' domain='0x0000' bus='0x05' slot='0x10' function='0x2'/>
	  </source>
	  <alias name='hostdev0'/>
	  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
	</interface>


After migration: (No ping)
<interface type='hostdev'>                                                                                                                                     
	  <mac address='00:1a:4a:16:20:76'/>                                                                                                                         
	  <driver name='vfio'/>                                                                                                                                 
	  <source>                                                                                                                                                 
		<address type='pci' domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>                                                                             
	  </source>                                                                                                                                                  
	  <alias name='hostdev0'/>                                                                                                                                  
	  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>                                                                                
	</interface>


After migrate back to original host: (No ping)
<interface type='hostdev'>                                                                                                                                     
	  <mac address='00:1a:4a:16:20:76'/>                                                                                                                         
	  <driver name='vfio'/>                                                                                                                                 
	  <source>                                                                                                                                                 
		<address type='pci' domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>                                                                             
	  </source>                                                                                                                                                  
	  <alias name='hostdev0'/>                                                                                                                                  
	  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>                                                                                
	</interface>

Comment 4 Dan Kenigsberg 2017-01-11 08:00:13 UTC
(In reply to Meni Yakove from comment #3)
> This only happens when the hosts have more than one VFs enabled and it's not
> related to plug/unplug vNIC.

Not sure I understand. During migration we do plug/unplug, and that's the only way to change the VF connected to the guest

> 
> It seems that if the vNIC get different VF (slot) we lost connection.

Could it be that what kills traffic is having a stale VF on the host with the same mac as the one in the guest?

Comment 5 Meni Yakove 2017-01-15 11:50:41 UTC
1.  Create 3 VMs 
2.  Enable 2 VFs on the host
3.  Add vNIC (passthrough) to VM 1 and start the VM
4.  Make sure that VM 1 got IP and have connectivity
5.  Check which VF the VM get:
    virsh -r dumpxml <vm-name> | grep -A8 "<interface type='hostdev'>"
    <source>
       <address type='pci' domain='0x0000' bus='0x05' slot='0x10' function='0x2'/>
    </source>
    VM 1 got function='0x2'
6.  Unplug the vNIC from VM 1
7.  Add vNIC (passthrough) to VM 2 and start the VM
8.  Add vNIC (passthrough) to VM 3 and start the VM
9.  Check which VM (2 or 3) got the same source VF that was on VM 1
10. Stop the VM that didn't get the same source VF that was on VM 1
11. Plug the vNIC back on VM 1
12. VM 1 should get different source VF and results with no connectivity.

Comment 6 Dan Kenigsberg 2017-01-18 09:48:39 UTC
I am guessing that our problem is due to the former VF staying in the host with the same MAC as the VF owned by the VM (bug 1341248). Can you you try chainging its mac address, or take the vf down?

Comment 7 Dan Kenigsberg 2017-01-18 09:50:30 UTC
To verify my guest, could you repeat the steps with a different nic type and driver?

Comment 8 Dan Kenigsberg 2017-01-19 22:50:58 UTC
Leon, could you try the tricky workaround suggested in https://bugzilla.redhat.com/show_bug.cgi?id=1341248#c17 ?

Comment 9 Yanan Fu 2017-01-26 00:28:09 UTC
Hi Michael and Dan,
I tests in qemu side, according to the comment 5.

Test version:
qemu: qemu-kvm-rhev-2.6.0-27.el7.x86_64
kernel: kernel-3.10.0-514.el7.x86_64
nic driver: qlcnic

Test steps:
1. create 2 VFs, and prepare 3 VMs.
2. add VF 1 to VM 1  ---> VF 1 get ip in VM 1 and work well
3. hotunplg VF 1 from VM 1.
4. add VF 1 to VM 2.  ---> get ip, work well
5. add VF 2 to VM 3.  ---> get ip, work well
6. hot unplug VF 2 from VM 3, then hot plug it to VM 1.
7. VM 1 work with VF 2, can get ip and work well.

In step 2:
VM 1 + VF 1:   ip:10.73.33.183,   Mac:  8a:ea:c2:7b:6e:f1

In step 4:
VM 2 + VF 1:   ip:10.73.33.183,   Mac:  8a:ea:c2:7b:6e:f1

In step 5:
VM 3 + VF 2:   ip:10.73.33.190,   Mac:  66:e3:ce:63:d7:64

In step 6:
VM 1 + VF 2:   ip:10.73.33.190,   Mac:  66:e3:ce:63:d7:64


Conclusion:
VF's mac don't change when add to different VMs, and can get ip normally.
Seems can not reproduce this bug with qlcnic driver. Thanks!

Comment 10 Dan Kenigsberg 2017-02-13 11:49:17 UTC
Let us verify this bug only when https://gerrit.ovirt.org/#/c/72135/ is in.

Comment 11 Michael Burman 2017-02-20 09:56:27 UTC
Verified on - 4.1.1.2-0.1.el7


Note You need to log in before you can comment on or make changes to this bug.