Bug 1819991

Hi

This is exactly how it is  supposed to work.  If you unplug the device, it becomes free when hot-unplug finishes.  Otherwise if you need a physical device right now for another guest and not powering down this one, how one is supposed to do it?

Hi Juan, Sorry for that I can not understand well. Just to confirm is this behavior below is expected by design?

1. For hostdev interface *without* <teaming> setting, after executing hot-unplug succeed: 1)the interface xml is deleted from the live guest xml;  2)if "managed='yes'", the vf is reattached to the host with driver like 'ixgbevf';

2. For hostdev interface *with* <teaming> setting, after executing hot-unplug succeed: 1)the interface xml will still exists in the live guest xml; 2)Even "managed='yes'", the vf is *not* reattached to the host, keep driver as "vfio-pci";

Hi

Laine, could you clarify the meaning of the "persistent" element of teaming.

In the proposed example, I am not clear if the device should be kept on vfio-pci state or go back to the host.

Any suggestion?

"persistent" just means that this (virtio) device is the one of the pair that will remain during migration. The other (hostdev) device is the "transient" device.

These names are only relevant for the auto-unplug/replug that happens during migration though. For either device, if they are explicitly unplugged, then they should be _completely_ unplugged (and what the guest does with that is up to the guest).

Hi

movinvg this to 8.3.0 as we are still discussing what the solution is.

(In reply to yalzhang from comment #2)
> Hi Juan, Sorry for that I can not understand well. Just to confirm is this
> behavior below is expected by design?
> 
> 1. For hostdev interface *without* <teaming> setting, after executing
> hot-unplug succeed: 1)the interface xml is deleted from the live guest xml; 
> 2)if "managed='yes'", the vf is reattached to the host with driver like
> 'ixgbevf';

This is correct.

> 2. For hostdev interface *with* <teaming> setting, after executing
> hot-unplug succeed: 1)the interface xml will still exists in the live guest
> xml; 2)Even "managed='yes'", the vf is *not* reattached to the host, keep
> driver as "vfio-pci";

A: this should return to be ixgbevf
B: this "could" take a while

Investigating _why_ this is not happening, and if this is happening on qemu or libvirt.

Even after a command has been issued to QEMU to detach a device from a guest, libvirt will keep the device in the guest's "status" XML until it receives notification back from QEMU (via the "DEVICE_DELETED" event) that the guest has acted on the request and released the device. Only then will libvirt remove the device from the guest's status, and (assuming the device is configured with "managed='yes'") re-bind the device to the host net driver.

The code that handles this is the same whether the device has <teaming> or not.

Based on this, I'm guessing that QEMU isn't sending the DEVICE_DELETED event.

A quick experiment proves that - just get the file examples/systemtap/qemu-monitor.stp from the libvirt sources, then run (as root, of course) "stap qemu-monitor.stp. It will take quite a long time to start up, but once it's started, you'll get a running log of all communications between libvirt and the qemu monitors it has attached to. So now you can just detach the device in question, and watch for a line like this:

174.444 ! 0x7fc70404a5d0 {"timestamp": {"seconds": 1594141866, "microseconds": 476418}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/ua-backup0/virtio-backend"}}


That's what comes about 6 seconds after detaching the *virtio* device from my test guest. When I detach the hostdev device, though, I don't see any corresponding DEVICE_DETACHED event.

So either the guest OS isn't properly handling and responding to the detach request, or QEMU isn't properly handling/responding when it gets notification from the guest. Maybe it's trying to maintain the same "half detached" status that is used for these devices during migration?

Yalan,

Could you re-test with the latest qemu (5.2) to see if the problem has been fixed in the meantime?

Thanks

Alex,

do you know if the missing "DEVICE_DELETED" event (see comment #7) is a known problem with vfio device?

Thanks

(In reply to Laurent Vivier from comment #15)
> Alex,
> 
> do you know if the missing "DEVICE_DELETED" event (see comment #7) is a
> known problem with vfio device?

There should be nothing special with a vfio device sending a DEVICE_DELETED event.  AFAIK, failover support in QEMU is limited to preventing a migration blocker from being registered for vfio-pci failover devices, specific handling of vfio devices was rejected upstream.  Did QEMU really release the device?  Did the guest?  We should be able to query QEMU's devices or even look to see if the vfio group device file is still opened by QEMU.

Laurent - just to clarify, the test I ran in Comment 7 was with a vfio device that had been created as a pair with a virtio device (i.e. using "<teaming type='transient' persistent='ua-backup0'/>"). The DEVICE_DELETED event *is* properly issued for normal vfio-assigned devices - I never have any problem with that; I only got a missing DEVICE_DELETED event when the vfio device was a part of a qemu failover pair.

My limited understanding of what happens to these devices *during migration* is that they are "half released" (whatever that means?) by QEMU until the migration is finalized. My ignorant guess (absent any real facts/evidence) is that somehow qemu is doing this "half detach" thing rather than a full detach when an unplug is requested.

Have tried on latest qemu 5.2 and hit Bug 1917654. And this hot unplug issue is not fixed yet.

Yalan,

could you provide:
- the full xml file for the domain at boot,
- the command line of qemu that is launched by libvirt (something like "ps -ef|grep qemu-kvm"),
- after the hot unplug of the vfio card the kernel logs from the guest (dmesg or journalctl -b)

Thanks

Created attachment 1751963 [details]
the guest's active xml

Created attachment 1751964 [details]
qemu command line of the guest

Created attachment 1751965 [details]
guest's log after detach the hostdev interface

The above attachment are for comment 19. It is tested with 
# rpm -q libvirt qemu-kvm
libvirt-7.0.0-2.module+el8.4.0+9520+ef609c5f.x86_64
qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64

with this version, the hostdev type interface is missing on the guest, just as described in Bug 1917654.

We should check some values in primary device pci dev data structure 

dev->qdev.pending_deleted_event and dev->partially_hotplugged

after the unplug was done. This could help us figure out what's wrong. Do we have a dump or could you get those
values with an attached gdb? Maybe just print out the virtio-net dev and the PCIDevice structs.

(In reply to yalzhang from comment #0)
> Description of problem:
> Hostdev type interface with teaming setting exists in domain xml and doesn't
> reattach to host after hot-unplug
> 
> Version-Release number of selected component (if applicable):
> # rpm -q libvirt-libs qemu-kvm 
> libvirt-libs-6.0.0-16.module+el8.2.0+6139+d66dece5.x86_64
> qemu-kvm-4.2.0-17.module+el8.2.0+6141+0f540f16.x86_64
> 

What is the version of the guest kernel that fails?
Do you know a version of a guest kernel that works fine ?

It seems kernel 4.18.0-280 behaves correctly but the cleanup is not done correctly at qemu level and that prevents to re-hotplug the vfio card.

For instance:

# dmesg|grep -e fail
[    3.981800] virtio_net virtio1 eth0: failover master:eth0 registered
[    3.981967] virtio_net virtio1 eth0: failover standby slave:eth1 registered
[    4.148746] virtio_net virtio1 eth0: failover primary slave:eth1 registered

# ip a
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group defau0
    link/ether 2c:ea:7f:59:fe:e2 brd ff:ff:ff:ff:ff:ff
    inet 10.19.153.21/21 brd 10.19.159.255 scope global dynamic noprefixroute enp2s0
       valid_lft 86385sec preferred_lft 86385sec
    inet6 2620:52:0:1398:9c72:a3ae:a18b:aeed/64 scope global dynamic noprefixroute 
       valid_lft 2591991sec preferred_lft 604791sec
    inet6 fe80::c282:65cc:bffc:77aa/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp2s0nsby: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master enp2s0 s0
    link/ether 2c:ea:7f:59:fe:e2 brd ff:ff:ff:ff:ff:ff
4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master enp2s0 state UP gr0
    link/ether 2c:ea:7f:59:fe:e2 brd ff:ff:ff:ff:ff:ff

(qemu) info qtree
...
       bus: root.3
          type PCIE
          dev: vfio-pci, id "hostdev0"
            host = "0000:04:00.0"
            sysfsdev = "/sys/bus/pci/devices/0000:04:00.0"
            x-pre-copy-dirty-page-tracking = "on"
            display = "off"
            xres = 0 (0x0)
            yres = 0 (0x0)
            x-intx-mmap-timeout-ms = 1100 (0x44c)
            x-vga = false
            x-req = true
            x-igd-opregion = false
            x-enable-migration = false
            x-no-mmap = false
            x-balloon-allowed = false
            x-no-kvm-intx = false
            x-no-kvm-msi = false
            x-no-kvm-msix = false
            x-no-geforce-quirks = true
            x-assigned-device-limit = 64 (0x40)
            x-no-kvm-ioeventfd = false
            x-no-vfio-ioeventfd = false
            x-pci-vendor-id = 5348 (0x14e4)
            x-pci-device-id = 5727 (0x165f)
            x-pci-sub-vendor-id = 4294967295 (0xffffffff)
            x-pci-sub-device-id = 4294967295 (0xffffffff)
            x-igd-gms = 0 (0x0)
            x-nv-gpudirect-clique = 255 (0xff)
            x-msix-relocation = "off"
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            x-pcie-lnksta-dllla = true
            x-pcie-extcap-init = true
            failover_pair_id = "net0"
            class Ethernet controller, addr 03:00.0, pci id 14e4:165f (sub 1028:001f)
            bar 0: mem at 0xfe600000 [0xfe60ffff]
            bar 2: mem at 0xfe610000 [0xfe61ffff]
            bar 4: mem at 0xfe620000 [0xfe62ffff]
            bar 6: mem at 0xffffffffffffffff [0x3fffe]
...
(qemu) info pci
...
  Bus  0, device   2, function 2:
    PCI bridge: PCI device 1b36:000c
      IRQ 11, pin A
      BUS 0.
      secondary bus 3.
      subordinate bus 3.
      IO range [0x3000, 0x3fff]
      memory range [0xfd400000, 0xfd5fffff]
      prefetchable memory range [0xfe600000, 0xfe7fffff]
      BAR0: 32 bit memory at 0xfda02000 [0xfda02fff].
      id "root.3"
  Bus  3, device   0, function 0:
    Ethernet controller: PCI device 14e4:165f
      PCI subsystem 1028:001f
      IRQ 6, pin A
      BAR0: 64 bit prefetchable memory at 0xfe600000 [0xfe60ffff].
      BAR2: 64 bit prefetchable memory at 0xfe610000 [0xfe61ffff].
      BAR4: 64 bit prefetchable memory at 0xfe620000 [0xfe62ffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "hostdev0"
  Bus  0, device   2, function 3:
...
(qemu) device_del hostdev0
(qemu) info qtree
...
      dev: pcie-root-port, id "root.3"
        x-migrate-msix = true
        bus-reserve = 4294967295 (0xffffffff)
        io-reserve = 18446744073709551615 (16 EiB)
        mem-reserve = 18446744073709551615 (16 EiB)
        pref32-reserve = 18446744073709551615 (16 EiB)
        pref64-reserve = 18446744073709551615 (16 EiB)
        x-speed = "16"
        x-width = "32"
        power_controller_present = true
        disable-acs = false
        chassis = 3 (0x3)
        slot = 0 (0x0)
        hotplug = true
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 02.2
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        x-pcie-lnksta-dllla = true
        x-pcie-extcap-init = true
        failover_pair_id = ""
        class PCI bridge, addr 00:02.2, pci id 1b36:000c (sub 0000:0000)
        bar 0: mem at 0xfda02000 [0xfda02fff]
        bus: root.3
          type PCIE
...
(qemu) info pci
...
  Bus  0, device   2, function 2:
    PCI bridge: PCI device 1b36:000c
      IRQ 11, pin A
      BUS 0.
      secondary bus 3.
      subordinate bus 3.
      IO range [0x3000, 0x3fff]
      memory range [0xfd400000, 0xfd5fffff]
      prefetchable memory range [0xfe600000, 0xfe7fffff]
      BAR0: 32 bit memory at 0xfda02000 [0xfda02fff].
      id "root.3"
  Bus  0, device   2, function 3:
...
# dmesg|tail -3
[  202.998132] pcieport 0000:00:02.2: Slot(0-2): Attention button pressed
[  202.998134] pcieport 0000:00:02.2: Slot(0-2): Powering off due to button press
[  208.437949] virtio_net virtio1 enp2s0: failover primary slave:enp3s0 unregistered

We can see the card has been unplugged by the kernel (dmesg) and qemu (root.3 is empty), but we cannot plug back the card:

(qemu) device_add vfio-pci,host=0000:04:00.0,id=hostdev1,bus=root.3,failover_pair_id=net0
Error: vfio 0000:04:00.0: device is already attached
(qemu) device_add vfio-pci,host=0000:04:00.0,id=hostdev0,bus=root.3,failover_pair_id=net0
Try "help device_add" for more information
(qemu) device_add vfio-pci,host=0000:04:00.0,id=hostdev0,bus=root.3,failover_pair_id=net0
qemu-kvm: Duplicate ID 'hostdev0' for device
Try "help device_add" for more information

I've tested latest upstream qemu (99ae0cd90d) and the problem is the same.

Without the failover device (virtio-net) the device_del/device_add of the vfio device works well. So I guess the failover code in QEMU doesn't release some vfio structures when the card is unplugged.

The fix seems easy: an object_unref() is missing when we add the vfio device in the qdev tree.

The following patch fixes the problem in upstream QEMU for me (tested with a 4.18.0-60 guest kernel because of BZ 1917654):

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 5150f295e8..d9ebb50219 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -863,6 +863,9 @@ static void failover_add_primary(VirtIONet *n, Error **errp)
         if (err) {
             qemu_opts_del(opts);
         }
+        if (dev) {
+            object_unref(OBJECT(dev));
+        }
     } else {
         error_setg(errp, "Primary device not found");
         error_append_hint(errp, "Virtio-net failover will not work. Make "

Hi Laurent,


I have test the build you have provided in the comment 28, the failover vf could be hot-unplugged from the vm successfully.


Test environment:
# rpm -q qemu-kvm
qemu-kvm-5.2.0-5.el8.BZ1819991.x86_64



Test step:
(1) start a vm with a failover vf and a failover virtio-net device

domain xml:
    <interface type='bridge'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source network='failover-bridge' portid='6f70ccf4-dfc1-48d1-9d8f-ac7a7b79416e' bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:aa:1c:ef'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x06' slot='0x10' function='0x0'/>
      </source>
      <teaming type='transient' persistent='net0'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </interface>

qemu cmd line:
-netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 \
-device virtio-net-pci,failover=on,netdev=hostnet0,id=net0,mac=52:54:00:aa:1c:ef,bus=pci.1,addr=0x0 \
-device vfio-pci,host=0000:06:10.0,id=hostdev0,bus=pci.7,addr=0x0,failover_pair_id=net0 \


（2） check the network device info in the vm
# ifconfig 
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.33.140  netmask 255.255.254.0  broadcast 10.73.33.255
        inet6 fe80::146b:9f06:d8cb:a094  prefixlen 64  scopeid 0x20<link>
        inet6 2620:52:0:4920:bf2f:5e18:7523:e255  prefixlen 64  scopeid 0x0<global>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 317  bytes 40122 (39.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 153  bytes 19655 (19.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp1s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::46ee:9377:7d84:93b2  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 180  bytes 16250 (15.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22  bytes 4144 (4.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp7s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.33.140  netmask 255.255.254.0  broadcast 10.73.33.255
        inet6 fe80::9b24:ab5d:16c5:2ec0  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 137  bytes 23872 (23.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 131  bytes 15511 (15.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# dmesg | grep -i failover
[    7.301316] virtio_net virtio0 eth0: failover master:eth0 registered
[    7.308495] virtio_net virtio0 eth0: failover standby slave:eth1 registered
[   12.783587] virtio_net virtio0 enp1s0: failover primary slave:eth0 registered


(3) hot-unplug the failover vf from the vm
# virsh detach-device-alias failover_test hostdev0
Device detach request sent successfully


(3.1) check the guest dmesg after hot-unplug failover vf
# dmesg
[  340.008996] pcieport 0000:00:02.6: Slot(0-6): Attention button pressed
[  340.011152] pcieport 0000:00:02.6: Slot(0-6): Powering off due to button press
[  345.557224] virtio_net virtio0 enp1s0: failover primary slave:enp7s0 unregistered

(3.2) check the host vf driver and host dmesg after hot-unplug failover vf

The failvoer vf has reattached to host 

# dmesg
[11397.033573] ixgbe 0000:06:00.0 enp6s0f0: VF Reset msg received from vf 0
[11398.339450] ixgbe 0000:06:00.0: setting MAC f2:bd:4c:f0:5a:a6 on VF 0
[11398.346645] ixgbe 0000:06:00.0: Reload the VF driver to make this change effective.
[11398.355698] ixgbevf 0000:06:10.0: enabling device (0000 -> 0002)
[11398.362799] ixgbe 0000:06:00.0 enp6s0f0: VF Reset msg received from vf 0
[11398.374401] ixgbevf 0000:06:10.0: f2:bd:4c:f0:5a:a6
[11398.379845] ixgbevf 0000:06:10.0: MAC: 1
[11398.382174] ixgbevf 0000:06:10.0 enp6s0f0v0: renamed from eth0
[11398.384254] ixgbevf 0000:06:10.0: Intel(R) 82599 Virtual Function
[11398.418786] IPv6: ADDRCONF(NETDEV_UP): enp6s0f0v0: link is not ready
[11398.439204] ixgbevf 0000:06:10.0: NIC Link is Up 10 Gbps
[11398.440745] IPv6: ADDRCONF(NETDEV_UP): enp6s0f0v0: link is not ready
[11398.452238] IPv6: ADDRCONF(NETDEV_CHANGE): enp6s0f0v0: link becomes ready


# readlink -f /sys/bus/pci/devices/0000\:06\:10.0/driver
/sys/bus/pci/drivers/ixgbevf

(3.3) check the related qmp info 

1461.845 > {"execute":"device_del","arguments":{"id":"hostdev0"},"id":"libvirt-380"}
1468.682 !  {"timestamp": {"seconds": 1612761713, "microseconds": 585501}, "event": "DEVICE_DELETED", "data": {"device": "hostdev0", "path": "/machine/peripheral/hostdev0"}}

(3.4) check the domain xml
only the failvoer virtio-net device domain xml exists 

# virsh dumpxml $domain
...
    <interface type='bridge'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source network='failover-bridge' portid='6f70ccf4-dfc1-48d1-9d8f-ac7a7b79416e' bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

(In reply to Yanghang Liu from comment #30)
> Hi Laurent,
> 
> 
> I have test the build you have provided in the comment 28, the failover vf
> could be hot-unplugged from the vm successfully.

Thank you Yanghang,

could you also check that migration continues to work as expected with this change?

Once it's done I will post changes downstream.

Thanks,
Laurent

Hi laurent, I have tried with the build on comment 28, the unplug issue is fixed. And I also tested migration, all the results is as expected.
I have another question: is it a valid test scenario to hot unplug the "persistent" interface? I have tried it, unplug the persistent interface will cause the vm to be unreachable.

The scenarios includes:
1. Unplug the hostdev interface, 
   1) then check the live xml, the hostdev interface is gone; 
   2) check the vf's driver, it is reattach to the host; 
   3) check on the guest, the hostdev type interface is gone, and the network works well; 

2. Unplug the hostdev interface, then hotplug this hostdev interface by attach-device(with the failover configuration):
   1) check the network work will on vm;
   2) check the interface occur on the vm;
   3) check the live xml has included the hostdev interface;

3. Migration works well; 

4. unplug the persistent interface:
on vm when unplug the persistent interface, the network will be broken when unplug the 'persistent' interface.

[root@vm ~]# ping www.baidu.com
PING www.a.shifen.com (220.181.38.149) 56(84) bytes of data.
....
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=16 ttl=46 time=3.69 ms
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=17 ttl=46 time=3.73 ms
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=18 ttl=46 time=3.65 ms
[  129.791903] pcieport 0000:00:02.3: Slot(0-3): Attention button pressed
[  129.793228] pcieport 0000:00:02.3: Slot(0-3): Powering off due to button press
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=19 ttl=46 time=3.66 ms
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=20 ttl=46 time=3.96 ms
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=21 ttl=46 time=3.68 ms
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=22 ttl=46 time=3.68 ms
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=23 ttl=46 time=3.57 ms
[  135.137667] virtio_net virtio2 enp4s0: failover standby slave:enp4s0nsby unregistered
[  135.175958] virtio_net virtio2 enp4s0: failover primary slave:enp5s0 unregistered
[  135.177404] virtio_net virtio2 enp4s0: failover master:enp4s0 unregistered
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
...

Add the information about the version for comment 32:

guest kernel: 4.18.0-193.el8.x86_64 (no issue like bug 1917654)

host libvirt and qemu:
# rpm -q libvirt qemu-kvm
libvirt-7.0.0-3.module+el8.4.0+9709+a99efd61.x86_64
qemu-kvm-5.2.0-5.el8.BZ1819991.x86_64

(In reply to yalzhang from comment #32)
...
> I have another question: is it a valid test scenario to hot unplug the
> "persistent" interface? I have tried it, unplug the persistent interface
> will cause the vm to be unreachable.

I'm not a specialist of the domain, but I would guess it's expected.
But I think it has to be tested to be sure we don't crash the kernel doing that...

Perhaps Jens can answer?

The persistent device in the guest is the main device and controls 
the primary and standby device. 
When you unplug the main/persistent device there's no way to control
the two other devices any more. 

So I would say, yes it is expected that the vm becomes unreachable.

Patch sent upsream:

[PATCH v3 2/4] virtio-net: add missing object_unref()
https://patchew.org/QEMU/20210212135250.2738750-1-lvivier@redhat.com/20210212135250.2738750-3-lvivier@redhat.com/

Merged upstream:

00e7b1299599 ("virtio-net: add missing object_unref()")
https://github.com/qemu/qemu/commit/00e7b1299599384dfdda2a2a4570a0fb2d69eb6b

The problem can be reproduced in the following test environment:

host:
4.18.0-287.el8.dt4.x86_64
qemu-kvm-5.2.0-9.module+el8.4.0+10182+4161bd91.x86_64
guest:
4.18.0-291.el8.x86_64


Test step:
(1) start a vm with a failover vf and a failover virtio net device

The domain xml:

    <interface type='bridge'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source network='failover-bridge' portid='064017b3-6021-4d54-9f3a-a108fa05fc12' bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='ua-test'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:aa:1c:ef'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x06' slot='0x10' function='0x0'/>
      </source>
      <teaming type='transient' persistent='ua-test'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </interface>

The qemu cmd line:

-netdev tap,fd=40,id=hostua-test,vhost=on,vhostfd=41 \
-device virtio-net-pci,failover=on,netdev=hostua-test,id=ua-test,mac=52:54:00:aa:1c:ef,bus=pci.1,addr=0x0 \
-device vfio-pci,host=0000:06:10.0,id=hostdev0,bus=pci.7,addr=0x0,failover_pair_id=ua-test \

(2) check the network device info in the vm

# ifconfig 
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.33.140  netmask 255.255.254.0  broadcast 10.73.33.255
        inet6 fe80::13eb:4896:8ea0:3557  prefixlen 64  scopeid 0x20<link>
        inet6 2620:52:0:4920:5af1:27aa:31f1:70a3  prefixlen 64  scopeid 0x0<global>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 106  bytes 15516 (15.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 96  bytes 11569 (11.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp1s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::720c:f7a:cf1e:ff26  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 65  bytes 6936 (6.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11  bytes 1790 (1.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp7s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.33.140  netmask 255.255.254.0  broadcast 10.73.33.255
        inet6 fe80::3521:657:3b91:70fa  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 41  bytes 8580 (8.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 85  bytes 9779 (9.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


# dmesg | grep -i failover
[    3.843022] virtio_net virtio0 eth0: failover master:eth0 registered
[    3.846099] virtio_net virtio0 eth0: failover standby slave:eth1 registered
[    6.579829] virtio_net virtio0 enp1s0: failover primary slave:eth0 registered



(3) hot-unplug the failover vf from the vm
# virsh detach-device-alias failover_test hostdev0
Device detach request sent successfully


(3.1) check the guest dmesg after hot-unplug failover vf
# dmesg
[  128.166973] pcieport 0000:00:02.6: Slot(0-6): Attention button pressed
[  128.169452] pcieport 0000:00:02.6: Slot(0-6): Powering off due to button press
[  133.631315] virtio_net virtio0 enp1s0: failover primary slave:enp7s0 unregistered


(3.2) check the host vf driver after hot-unplug failover vf

# readlink -f /sys/bus/pci/devices/0000\:06\:10.0/driver
/sys/bus/pci/drivers/vfio-pci   <-- The vf driver is still vfio-pci


(3.3) check the related qmp info 

> {"execute":"device_del","arguments":{"id":"hostdev0"},"id":"libvirt-380"}  

I *did not* observe the information as following: 
!  {"timestamp": {"seconds": 1612761713, "microseconds": 585501}, "event": "DEVICE_DELETED", "data": {"device": "hostdev0", "path": "/machine/peripheral/hostdev0"}}

(3.4) check the domain xml
Both the failover virtio-net device and the failover vf exists in the domain xml.


# virsh domiflist failover_test 
 Interface   Type      Source            Model    MAC
---------------------------------------------------------------------
 vnet0       bridge    failover-bridge   virtio   52:54:00:aa:1c:ef
 -           hostdev   -                 -        52:54:00:aa:1c:ef


# virsh dumpxml $domain
...
    <interface type='bridge'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source network='failover-bridge' portid='e04b7327-8a1b-4aa7-bec0-85dce81fe567' bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='ua-test'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:aa:1c:ef'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x06' slot='0x10' function='0x0'/>
      </source>
      <teaming type='transient' persistent='ua-test'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </interface>

The process of verifying this bug:


Test env:
host:
qemu-kvm-5.2.0-10.module+el8.4.0+10217+cbdd2152.x86_64
4.18.0-287.el8.dt4.x86_64
guest:
4.18.0-291.el8.x86_64



Test step:

(1) (2) Repeat the same step 1 and step 2 in comment 47

> (1) start a vm with a failover vf and a failover virtio net device
> 
> The domain xml:
> 
>     <interface type='bridge'>
>       <mac address='52:54:00:aa:1c:ef'/>
>       <source network='failover-bridge'
> portid='064017b3-6021-4d54-9f3a-a108fa05fc12' bridge='br0'/>
>       <target dev='vnet0'/>
>       <model type='virtio'/>
>       <teaming type='persistent'/>
>       <alias name='ua-test'/>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
>     </interface>
> 
>     <interface type='hostdev' managed='yes'>
>       <mac address='52:54:00:aa:1c:ef'/>
>       <driver name='vfio'/>
>       <source>
>         <address type='pci' domain='0x0000' bus='0x06' slot='0x10'
> function='0x0'/>
>       </source>
>       <teaming type='transient' persistent='ua-test'/>
>       <alias name='hostdev0'/>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0'/>
>     </interface>
> 
> The qemu cmd line:
> 
> -netdev tap,fd=40,id=hostua-test,vhost=on,vhostfd=41 \
> -device
> virtio-net-pci,failover=on,netdev=hostua-test,id=ua-test,mac=52:54:00:aa:1c:
> ef,bus=pci.1,addr=0x0 \
> -device
> vfio-pci,host=0000:06:10.0,id=hostdev0,bus=pci.7,addr=0x0,
> failover_pair_id=ua-test \
> 
> (2) check the network device info in the vm
> 
> # ifconfig 
> enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 10.73.33.140  netmask 255.255.254.0  broadcast 10.73.33.255
>         inet6 fe80::13eb:4896:8ea0:3557  prefixlen 64  scopeid 0x20<link>
>         inet6 2620:52:0:4920:5af1:27aa:31f1:70a3  prefixlen 64  scopeid
> 0x0<global>
>         ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
>         RX packets 106  bytes 15516 (15.1 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 96  bytes 11569 (11.2 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> enp1s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet6 fe80::720c:f7a:cf1e:ff26  prefixlen 64  scopeid 0x20<link>
>         ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
>         RX packets 65  bytes 6936 (6.7 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 11  bytes 1790 (1.7 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> enp7s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 10.73.33.140  netmask 255.255.254.0  broadcast 10.73.33.255
>         inet6 fe80::3521:657:3b91:70fa  prefixlen 64  scopeid 0x20<link>
>         ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
>         RX packets 41  bytes 8580 (8.3 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 85  bytes 9779 (9.5 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> 
> # dmesg | grep -i failover
> [    3.843022] virtio_net virtio0 eth0: failover master:eth0 registered
> [    3.846099] virtio_net virtio0 eth0: failover standby slave:eth1
> registered
> [    6.579829] virtio_net virtio0 enp1s0: failover primary slave:eth0
> registered

(3) hot-unplug the failover vf from the vm
# virsh detach-device-alias failover_test hostdev0
Device detach request sent successfully


(3.1) check the guest dmesg after hot-unplug failover vf
# dmesg
[  216.309928] pcieport 0000:00:02.6: Slot(0-6): Attention button pressed
[  216.312100] pcieport 0000:00:02.6: Slot(0-6): Powering off due to button press
[  221.696435] virtio_net virtio0 enp1s0: failover primary slave:enp7s0 unregistered

# ifconfig 
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.33.140  netmask 255.255.254.0  broadcast 10.73.33.255
        inet6 fe80::13eb:4896:8ea0:3557  prefixlen 64  scopeid 0x20<link>
        inet6 2620:52:0:4920:5af1:27aa:31f1:70a3  prefixlen 64  scopeid 0x0<global>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 244  bytes 24161 (23.5 KiB)
        RX errors 0  dropped 52  overruns 0  frame 0
        TX packets 60  bytes 11696 (11.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp1s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 332  bytes 25769 (25.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 57  bytes 9828 (9.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


(3.2) check the host vf driver after hot-unplug failover vf

The vf has been reattached to the host

# readlink -f /sys/bus/pci/devices/0000\:06\:10.0/driver
/sys/bus/pci/drivers/ixgbevf  


(3.3) check the related qmp info 

>  {"execute":"device_del","arguments":{"id":"hostdev0"},"id":"libvirt-385"}
!  {"timestamp": {"seconds": 1614828583, "microseconds": 553675}, "event": "DEVICE_DELETED", "data": {"device": "hostdev0", "path": "/machine/peripheral/hostdev0"}}



(3.4) check the domain xml
only the failover virtio-net device exists in the domain xml.

# virsh domiflist failover_test 
 Interface   Type     Source            Model    MAC
--------------------------------------------------------------------
 vnet0       bridge   failover-bridge   virtio   52:54:00:aa:1c:ef



# virsh dumpxml $domain
...
    <interface type='bridge'>
      <mac address='52:54:00:aa:1c:ef'/>
      <source network='failover-bridge' portid='dbc88dba-92d8-4e9c-8700-5f425c1eca5b' bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='ua-test'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

According to comment 47 and comment 48, move the bug status to VERIFIED.

*** Bug 1867048 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2098