Bug 1531393

Didn't reproduce the issue without -device intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on 

The qemu command line:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu Haswell-noTSX \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-global isa-debugcon.iobase=0x402 \
-boot menu=on \
-qmp tcp:0:6667,server,nowait \
-usb \
-device usb-tablet \
-vga qxl \
-device pcie-root-port,bus=pcie.0,id=root0,multifunction=on,chassis=1,addr=0xa.0 \
-drive file=/home/sriov/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=root0 \
-device pcie-root-port,bus=pcie.0,id=root1,chassis=11,addr=0xa.1 \
-device virtio-net-pci,netdev=tap10,mac=00:52:68:26:31:03,bus=root1,id=net0 -netdev tap,id=tap10 \
-device pcie-root-port,bus=pcie.0,id=root2,chassis=12,addr=0xa.2 \
-device virtio-net-pci,netdev=tap11,mac=00:52:68:26:31:00,bus=root2,id=net1 -netdev tap,id=tap11 \
-device pcie-root-port,bus=pcie.0,id=root6,chassis=15,addr=0xa.5 \
-device vfio-pci,host=04:02.0,id=vf1,bus=root6,x-req=off \
-device pcie-root-port,bus=pcie.0,id=root7,chassis=4 \
-device vfio-pci,host=04:02.1,id=vf2,bus=root7,x-req=off \
-monitor stdio \
-vnc :1 \

Marked it as regression since didn't reproduce it with qemu-kvm-rhev-2.9.0-16.el7_4.13.x86_64

Hi Peter,

Following the logs, the Guest OS finishes the hot-unplug sequence
and powers down the device. It seems something on QEMU's IOMMU side
prevents the hot-unplug to finish.

Can you please have a look?
Thanks,
Marcel

(In reply to Marcel Apfelbaum from comment #8)
> Hi Peter,
> 
> Following the logs, the Guest OS finishes the hot-unplug sequence
> and powers down the device. It seems something on QEMU's IOMMU side
> prevents the hot-unplug to finish.
> 
> Can you please have a look?
> Thanks,
> Marcel

Hi, Marcel,

The event is sent in RCU thread:

#0  qapi_event_send_device_deleted (has_device=true, device=0x55c2500b81f0 "vfio0", path=0x55c2513530d0 "/machine/peripheral/vfio0", 
    errp=0x55c24f3f3d10 <error_abort>) at qapi-event.c:412
#1  0x000055c24e3ca86c in device_finalize (obj=0x55c25128b270) at /root/git/qemu/hw/core/qdev.c:1087
#2  0x000055c24e5ad6db in object_deinit (obj=0x55c25128b270, type=0x55c250094700) at /root/git/qemu/qom/object.c:462
#3  0x000055c24e5ad709 in object_deinit (obj=0x55c25128b270, type=0x55c2500a40c0) at /root/git/qemu/qom/object.c:466
#4  0x000055c24e5ad709 in object_deinit (obj=0x55c25128b270, type=0x55c250083030) at /root/git/qemu/qom/object.c:466
#5  0x000055c24e5ad74c in object_finalize (data=0x55c25128b270) at /root/git/qemu/qom/object.c:476
#6  0x000055c24e5ae6db in object_unref (obj=0x55c25128b270) at /root/git/qemu/qom/object.c:911
#7  0x000055c24e206988 in memory_region_unref (mr=0x55c25128b4e0) at /root/git/qemu/memory.c:1747
#8  0x000055c24e209d16 in do_address_space_destroy (as=0x55c25128b488) at /root/git/qemu/memory.c:2787
#9  0x000055c24e6da1c8 in call_rcu_thread (opaque=0x0) at /root/git/qemu/util/rcu.c:278
#10 0x00007f78faef4dd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f78fac1e94d in clone () from /lib64/libc.so.6

Luckily I still have the debugging stuff on jinzhao's host due to the other bug, so I did a quick test.  I think it's because some references of vfio-pci device is leaked somehow due to existance of VT-d, since this is the backtrace I got when the error happens (or say, when the event is missed):

(gdb) bt
#0  object_unref (obj=0x5559618ef0c0) at /root/git/qemu/qom/object.c:904
#1  0x000055595d710988 in memory_region_unref (mr=0x5559618ef330) at /root/git/qemu/memory.c:1747
#2  0x000055595d713d16 in do_address_space_destroy (as=0x5559618ef2d8) at /root/git/qemu/memory.c:2787
#3  0x000055595dbe41c8 in call_rcu_thread (opaque=0x0) at /root/git/qemu/util/rcu.c:278
#4  0x00007f85ee1b5dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f85ededf94d in clone () from /lib64/libc.so.6
(gdb) p obj->ref
$2 = 3

Basically here obj->ref should be 1 otherwise device_finalize() for vfio-pci device won't be called, and event won't be sent.

I'll dig more tomorrow.  For now, I'll take over the bug since it's for sure vt-d related.

Thanks,
Peter

This is reproducable upstream too, and even without XL710 - I tested
with X540, which also hit this.

A funny thing is that upstream QEMU doesn't have this problem until
commit f7b879e072 ("qdev: defer DEVICE_DEL event until
instance_finalize()", 2017-10-18).  However I don't think that's the
problematic commit.  That commit just revealed another issue after it
fixed one.

After I built a very customized QEMU (and with some script for parsing) I think
these are the two refcounts that are missing (via MemoryRegion
referencing, since memory_region_ref() will add ref onto its owner):

Remaining refcount holders:            
{                                      
    "0x5639af8f8000": {                
        "count": 1,                    
        "devid": 1,                    
        "name": "0000:05:10.2 BAR 0 mmaps[0]",                                 
        "mr": "0x5639af8f8000"         
    },                                 
    "0x5639af8f86d0": {                
        "count": 1,                    
        "devid": 1,                    
        "name": "0000:05:10.2 BAR 3 mmaps[0]",                                 
        "mr": "0x5639af8f86d0"         
    }                                  
}                                      

It means that we referenced these two regions one more time for the
2nd VF (and the 1st VF have paired unrefs, since we have no problem with it):

  "0000:05:10.2 BAR 0 mmaps[0]"
  "0000:05:10.2 BAR 3 mmaps[0]"
  
They are the two BARs for the X540 VF device.  Though, I am still
investigating on how the VT-d device can affect them to leak that refcount.

Peter

Well... this funny refcount leakage seems to be a more general issue on memory API rather than VT-d itself.  I posted my thoughts and solution upstream as RFC (since it touches memory core and I haven't tested too much):

  http://lists.gnu.org/archive/html/qemu-devel/2018-01/msg04703.html

I'll wait for some feedback on the patches before I decide what to do next.

Peter

Please follow 0day process, check the link
https://mojo.redhat.com/docs/DOC-1022552

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443