Bug 1520352

Here's what happening. VIOSerialPortEvtDeviceD0Exit disables interrupts on the read queue but it doesn't stop it. The host side is still sending data. This causes VIOSerialDiscardPortDataLocked, called from VIOSerialPortEvtDeviceD0Exit, to loop extensively and potentially indefinitely. As long as the device has data to send, the loop runs, re-queuing the returned buffers.

Then, when VIOSerialDiscardPortDataLocked convinces itself that it's processed everything, VIOSerialPortEvtDeviceD0Exit proceeds to call VIOSerialDrainQueue which uses virtqueue_detach_unused_buf to essentially un-push already pushed buffers to the queue. Obviously this must not be done on a live queue. The driver is racing here with the device which is still trying to fill buffers and return them to the driver. This is where the QEMU asserts fire.

This has been fixed in:

https://github.com/virtio-win/kvm-guest-drivers-windows/commit/dd278961d0c2ff969e9f388c06ea9e447f0daffa

Hi Ladi,

I try to reproduce this bug as comment#0 steps, but didn't reproduce it. Could you help to check my test steps?

Test Steps:
================================================================================
1. boot a guest with cli and install a new win2016 guest:
/usr/libexec/qemu-kvm \
-name win2016 \
-enable-kvm -machine pc,accel=kvm -m 4G -smp 2,maxcpus=8,sockets=8,cores=1,threads=1 -cpu host,hv_time,+kvm_pv_eoi,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -uuid ea78071a-f6e4-4347-8077-9cb9f7953a84 \
-nodefconfig --nodefaults -boot order=cd,menu=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 -device usb-tablet,id=input0 \
-drive file=win2016.raw,if=none,id=drive-ide0-0-1,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 \
-drive file=/home/images/en_windows_server_2016_x64_dvd_9718492.iso,media=cdrom,id=cdrom,if=none -device ide-drive,drive=cdrom,bootindex=1 \
-netdev tap,id=hostnet0,vhost=on,vhostforce=off \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:83:66:77:88:66,bus=pci.0,addr=0x3,status=on \
-vnc 0.0.0.0:1 -vga std -monitor stdio -qmp tcp:0:4446,server,nowait \
-cdrom /home/virtio-win-prewhql-141.iso \
-device virtio-serial-pci,id=virtio-serial0,max_ports=511 \
-chardev socket,host=localhost,port=4467,server,nowait,id=ser \
-device virtserialport,chardev=ser,name=port1 \

2. Install the virtio-win-prewhql-141 vioser driver.

3. In host:
$ nc localhost 4467 </dev/zero

4. in the Windows VM simply open the port:

  C:\>copy con \\.\port1
  hello<enter>

5. hot-add CPUs:
  (qemu) cpu-add 2
  (qemu) cpu-add 3
  (qemu) cpu-add 4
  (qemu) cpu-add 5
  (qemu) cpu-add 6
  (qemu) cpu-add 7
6. checked the qemu info, no qemu crash
================================================================================

I also tried to test with follow matrix:
1. qemu-kvm-rhev-2.10.0-8.el7  + virtio-win-prewhql-141, not reproduced(tried about 5 times).
2. qemu-kvm-rhev-2.10.0-8.el7  + virtio-win-prewhql-145, not reproduced(tried 1 1 time).
3. qemu-kvm-rhev-2.6.0-29.el7  + virtio-win-prewhql-141, not reproduced(tried about 5 times).
4. qemu-kvm-rhev-2.6.0-29.el7  + virtio-win-prewhql-145, not reproduced(tried 1 time).

Other versions:
kernel-3.10.0-798.el7.x86_64
seabios-1.10.2-5.el7.x86_64


Thanks a lot~
Peixiu

Hi Peixiu Hou,

(In reply to Peixiu Hou from comment #5)
> 4. in the Windows VM simply open the port:
> 
>   C:\>copy con \\.\port1
>   hello<enter>

What I meant here is typing hello, or any arbitrary string, and pressing Enter. Without Enter the copy command won't actually open the port the way we want.

If you did press Enter and still couldn't reproduce it, can you please give me access to the repro host?

Thank you!
Ladi

Also, do you see the added CPUs in the guest? In Device Manager or even in Task Manager. Every time you do cpu-add, a new CPU should appear.

If not, you'll have to apply the workaround from https://bugzilla.redhat.com/show_bug.cgi?id=1377155#c12

I should have mentioned it before, sorry about that.

Thanks! It looks like in your case the corruption is not detected by QEMU but by Windows on D0 re-entry.

Out of curiosity, can you please try again with a modern-only virtio-serial:

-device virtio-serial-pci,...,disable-legacy=on

Thank you,
Ladi

(In reply to Ladi Prosek from comment #9)
> Thanks! It looks like in your case the corruption is not detected by QEMU
> but by Windows on D0 re-entry.
> 
> Out of curiosity, can you please try again with a modern-only virtio-serial:
> 
> -device virtio-serial-pci,...,disable-legacy=on
> 
Tried with disable-legacy=on commands:
-device virtio-serial-pci,id=virtio-serial0,max_ports=511,disable-legacy=on \
-chardev socket,host=localhost,port=4467,server,nowait,id=ser \
-device virtserialport,chardev=ser,name=port1 \

1. Also hit BSOD with virtio-win-prewhql-141, qemu report follow error message. comment#8 also hit this error.
(qemu) cpu-add 2
(qemu) qemu-kvm: Guest moved used index from 131 to 129
qemu-kvm: Guest moved used index from 131 to 129

2. Not hit BSOD or qemu crash with virtio-win-prewhql-145.

Best Regard~
Peixiu

> Thank you,
> Ladi

Great, thank you. The results you're getting are expected then.

change status to verified according to comment#10 and comment#11

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0657