Bug 1372583

Created attachment 1197051 [details]
virt-viewer-keyboard.log

Created attachment 1197052 [details]
qemu-keyboard.log

So, I'm able to reproduce this bug, but the root cause is still eluding me at the moment. There seems to be some interaction between the client and the server. As far as I can tell, it only happens when you reset the guest while connected with a certain spice client. New clients (e.g. spice-gtk built from git) do not trigger the problem, but older ones do. 

However, when you trigger the problem (by rebooting the guest while connected with an older client), the keyboard will still be unusable even if you disconnect that client and re-connect with a newer client that would not otherwise exhibit this problem. That implies that the buggy client somehow affects the guest or host in such a way that the keyboard becomes disabled on the server side.

After bisecting spice-gtk to see where the client problem was fixed, I find this spice-gtk commit:

commit 73cd553fb0fbd213b64d72f8b4289ed8a17fc6c0
Author: Frediano Ziglio <fziglio>
Date:   Wed Sep 7 11:56:36 2016 +0100

    Ignore modifiers messages if no modifiers changed
    
    This avoid keep sending modifiers changes if guest is not
    synchronising the changes.
    
    I consider this as an improving as this avoids client to try again
    and again to force synchronisation however this does not prevent
    every unwanted keystroke insertion which possibly can be a real
    problem on some configurations.
    
    For instance if guest do not handle caps lock as the client do
    if client uses another modifiers (as num lock) this can force
    inserting virtual caps keypress.
    
    Signed-off-by: Frediano Ziglio <fziglio>
    Acked-by: Marc-André Lureau <mlureau>


I'm still trying to investigate how a client that sends a few too many keyboard modifier messages would end up totally disabling the guest's keyboard functionality. Or what it might have to do with SATA cdrom...

While the above commit does make the bug more difficult (or impossible?) to reproduce, it can't be the full solution. For one thing, the keyboard remains disabled after disconnecting and reconnecting a client, so there must be a bug on the server. After investigating a bit more, I've noticed a difference in the keyboard state between working and non-working resets. Specifically, when I set a breakpoint within ps2_put_keycode() after a reset when the keyboard is not working, I can see that PS2KbdState::scan_enabled is false. After a successful reboot, this value is true. A little more investigation shows that during a reboot where the keyboard works properly, ps2_write_keyboard() is called once with a value of KBD_CMD_RESET_DISABLE, and once with a value of KBD_CMD_RESET_ENABLE. In the case where the keyboard does not work, ps2_write_keyboard() is only called with a value of KBD_CMD_RESET_DISABLE.

Can somebody from the qemu team that is more familiar with this part of the code investigate a little bit further?

> I'm still trying to investigate how a client that sends a few too many
> keyboard modifier messages would end up totally disabling the guest's
> keyboard functionality. Or what it might have to do with SATA cdrom...

I guess it is the keyboard buffer overflowing, ps/2 has 16 bytes only which isn't that much.  Using SATA might change the timings enough that this actually becomes a problem.

KBD_CMD_RESET_DISABLE and KBD_CMD_RESET_ENABLE are guest writes.  Probably the guest considers the ps/2 keyboard dead.  Due to the buffer overflowing it doesn't get the expected replies.

Try this ...

@@ -556,8 +556,10 @@ void ps2_queue(void *opaque, int b)
     PS2State *s = (PS2State *)opaque;
     PS2Queue *q = &s->queue;
 
-    if (q->count >= PS2_QUEUE_SIZE - 1)
+    if (q->count >= PS2_QUEUE_SIZE - 1) {
+        fprintf(stderr, "%s: queue full\n", __func__);
         return;
+    }
     q->data[q->wptr] = b;
     if (++q->wptr == PS2_QUEUE_SIZE)
         q->wptr = 0;


... and see if it triggers.

Yes, when the problem is reproduced, that statement does indeed get printed.

please try this (clears queue on keyboard reset)
https://www.kraxel.org/cgit/qemu/log/?h=work/ps2-queue

Yes, that series does appear to fix this bug.

patches sent upstream

pull request sent.

(In reply to Gerd Hoffmann from comment #13)
> pull request sent.

Landed upstream.

Reproduce this issue against qemu-kvm-rhev-2.9.0-16.el7_4.13.x86_64.

Try to install a guest with qemu cli:
/usr/libexec/qemu-kvm \
-name guest=generic,debug-threads=on \
-S \
-machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,vmport=off \
-cpu IvyBridge \
-m 1024 \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 97c2c535-7489-4c4c-8e71-fc58c98f3e01 \
-no-user-config \
-nodefaults \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 \
-device ahci,id=sata0,bus=pci.0,addr=0x5 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 \
-drive file=/home/1372583.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 \
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 \
-drive file=/home/RHEL-7.4-20170711.0-Server-x86_64-dvd1.iso,format=raw,if=none,media=cdrom,id=drive-sata0-0-0,readonly=on \
-device ide-cd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 \
-netdev tap,fd=29,id=hostnet0 \
-device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:ee:97:eb,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-spice port=5900,addr=10.66.8.189,disable-ticketing,image-compression=off,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 \
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 \
-monitor stdio \

steps:
1. Try to install guest with above cli, in languages selection UI, try to input some characters, if you can input, then system_reset and retry.

I meet this issue 4 times in 10 trials.

Verify this issue against qemu-kvm-rhev-2.10.0-12.el7.x86_64, no such issue happen after 20 trials.

Verified per comment 16

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104