Bug 1371930

Steps to Reproduce for scenario 2:
1.Bring up vfs, and bind them to vfio-pci

2.Boot guest with not only one vf assigned, for example:
    -device vfio-pci,host=04:02.0,id=vf-02.0  \
    -device vfio-pci,host=04:02.1,id=vf-02.1  \

3.after guest boot up, unload the pf's driver in the host:
# modprobe -r i40e
This operation will be blocked, and  dmesg show:  
kernel: vfio-pci 0000:04:02.0: Relaying device request to user (#0)
kernel: iommu: Removing device 0000:04:02.0 from group 56
kernel: vfio-pci 0000:04:02.1: Relaying device request to user (#0)
kernel: vfio-pci 0000:04:02.1: Relaying device request to user (#10)
...
kernel: vfio-pci 0000:04:02.1: Relaying device request to user (#30)

The first vf "vf-02.0" disappear from "info pci", some times it disappear from guest too, but some times it keep in the guest.
The second vf "vf-02.1", exist in both "info pci" and guest.

4.device_del the second vf with qmp
{"execute":"device_del","arguments":{"id":"vf-02.1"}}
This operation return with no error output, but "info pci" still can get the vf.
In the guest, vf exist too.

5.quit the guest forcibly, then "modprobe -r i40e" will finished,vf disappear in host as execpt, and dmesg show as execpt:
kernel: iommu: Removing device 0000:04:02.1 from group 57

Hi Yanan,

I am not able to reproduce case 2). I launch the VM through the virt-manager with the 2 VFs assigned. When unprobing the I40e driver on host, the operation is not blocked as you mention. On the dmesg I can immediatly see:

[ 2422.411811] i40e 0000:03:00.1: i40e_ptp_stop: removed PHC on enp3s0f1
[ 2422.412030] vfio-pci 0000:03:0a.0: Relaying device request to user (#0)
[ 2422.615941] iommu: Removing device 0000:03:0a.0 from group 33
[ 2423.709047] i40e 0000:03:00.1: Deleted LAN device PF1 bus=0x00 func=0x01
[ 2423.711468] i40e 0000:03:00.0: i40e_ptp_stop: removed PHC on enp3s0f0
[ 2423.711683] vfio-pci 0000:03:02.0: Relaying device request to user (#0)
[ 2423.914928] iommu: Removing device 0000:03:02.0 from group 32
[ 2425.007045] i40e 0000:03:00.0: Deleted LAN device PF0 bus=0x00 func=0x00

In the virtual machine "Details" window, I see the 2 VFs disappear as expected. My guest is a win2012r2 as you mentioned + installed i40eVF drivers.

I am using 3.10.0-514.el7.x86_64 + qemu-kvm-rhev-2.6.0-27.el7

As for case 1 using x-req=false (non default), my understanding is it is never used by virt-tools. For my knowmledge, what is rationale behind using it? I will try to run it though...

Thanks

Eric

For use case 1), I don't reproduce either. At the moment I issue the device_del command in QEMU, the modprobe -r completes. See actions and traces below.

# modprobe -v vfio-pci
# modprobe -v i40e
insmod /lib/modules/3.10.0-514.el7.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko
# ifconfig enp3s0f0 up
# ifconfig enp3s0f1 up
# echo 1 > /sys/bus/pci/devices/0000\:03\:00.0/sriov_numvfs
# echo 1 > /sys/bus/pci/devices/0000\:03\:00.1/sriov_numvfs

# ./tools/dpdk-devbind.py --bind=vfio-pci 03:02.0
# ./tools/dpdk-devbind.py --bind=vfio-pci 03:0a.0

# /usr/libexec/qemu-kvm-rhev \
-name win2k12r2 \
-machine pc \
-sandbox off  \
-cpu SandyBridge,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \
-enable-kvm \
-m 2048 -realtime mlock=off \
-smp 8,sockets=8,cores=1,threads=1 \
-nodefaults \
-qmp unix:./qmp-sock,server,nowait \
-rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown \
-global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 \
-drive file=/var/lib/libvirt/images/win2k12r2.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 \
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-drive file=/var/lib/libvirt/images/en_windows_server_2012_r2_x64_dvd_2707946.iso,format=raw,if=none,id=drive-ide0-0-1,readonly=on \
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 \
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vga qxl \
-vnc :0 \
-device vfio-pci,host=03:02.0,id=vf0,x-req=off

# modprobe -r i40e
dmesg output:
[11400.162981] i40e 0000:03:00.1: i40e_ptp_stop: removed PHC on enp3s0f1
[11400.163311] iommu: Removing device 0000:03:0a.0 from group 33
[11401.261076] i40e 0000:03:00.1: Deleted LAN device PF1 bus=0x00 func=0x01
[11401.263594] i40e 0000:03:00.0: i40e_ptp_stop: removed PHC on enp3s0f0
[11401.263794] vfio-pci 0000:03:02.0: No device request channel registered, blocked until released by user

#qemu-kvm/scripts/qmp/qmp-shell  /home/augere/qmp-sock
QMP> device_del id=vf0

At that moment, modprobe -r completes.
dmesg contains
[11476.022236] iommu: Removing device 0000:03:02.0 from group 32
[11477.120047] i40e 0000:03:00.0: Deleted LAN device PF0 bus=0x00 func=0x00

Please can you check the bug still exists?

Thanks

Eric

Hi Alex, Yanan,

So I was eventually able to reproduce the issue on Yanan's machine and added some few traces in QEMU.

Looks the vfio_req_notifier_handler is properly called on both QEMU VFIO-PCI devices. for the first one, 0000:04:02.0 I can see the vfio_instance_finalize function being called. However for the second one it is never called.

In both cases it takes the following path:

- qdev unplug with hotplug_dev=PIIX4_PM
- acpi_pcihp_device_unplug_cb and acpi_send_gpe_event is sent. Eventually there are interrupts sent (SCI?, docs/specs/acpi_pci_hotplug.txt).

Can't it be a BIOS issue (note: the same use case 2 works fine for me with another win2012r2 image).

Thanks

Eric

(In reply to Eric Auger from comment #11)
> Can't it be a BIOS issue (note: the same use case 2 works fine for me with
> another win2012r2 image).

A system BIOS issue?  How is the system BIOS involved?  Or do you mean a SeaBIOS issue?

I meant a SeaBios issue.

thanks

Eric

Hi Igor,

OK. Actually we have not been able yet to identify the source of the hang. On the last SOS report we got "missed kernel messages" at crucial debug moment. We should reproduce the manipulation requested in https://bugzilla.redhat.com/show_bug.cgi?id=1371930#c20, and make sure we get all kernel logs in the sos or /var/log/messages.

I think this is a matter of journald config.
1) save existing config file:
cp /etc/systemd/journald.conf /tmp/journald.conf.sauv 
2) edit /etc/systemd/journald.conf
Lines with RateLimitInterval and RateLimitBurst should be commented
If yes, add the following lines, otherwise change the value.
RateLimitInterval=0
RateLimitBurst=0
3) restart systemd-journald:
systemctl restart systemd-journald
4) reproduce manip detailed in comment #20:
on the node where it hangs, once the hang is observed, issue:
echo t >  /proc/sysrq-trigger
5) generate the SOS report. It shouldn't have any messages like
"Aug  2 11:21:38 compute-1 journal: Missed 130 kernel messages"
or invoke:
journalctl -k
6) send the SOS report or output of journalctl
7) restore the original journald.conf file
mv /tmp/journald.conf.sauv /etc/systemd/journald.conf
8) restart systemd-journald:
systemctl restart systemd-journald

The above change requires root access though. Obviously if you know an easier/cleaner manner to do that change, please correct. But getting a full log would be useful we think.

Thanks

Eric

The original issue cannot be reproduced anymore on original qemu/kernel combo (after mandated NIC driver update in the guest) and cannot be reproduced with RHEL7.4 either. So after discussion with Yanan, we suggest to close it.