Bug 1372583 - Keyboard can't be used when install rhel7 in guest which has SATA CDROM and spice+qxl mode sometimes
Summary: Keyboard can't be used when install rhel7 in guest which has SATA CDROM and s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Gerd Hoffmann
QA Contact: Guo, Zhiyi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-02 06:40 UTC by mxie@redhat.com
Modified: 2018-04-11 00:09 UTC (History)
19 users (show)

Fixed In Version: qemu-kvm-rhev-2.10.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-11 00:09:32 UTC


Attachments (Terms of Use)
virt-manager-keyboard.log (46.53 KB, text/plain)
2016-09-02 06:40 UTC, mxie@redhat.com
no flags Details
virt-viewer-keyboard.log (82.70 KB, text/plain)
2016-09-02 06:41 UTC, mxie@redhat.com
no flags Details
qemu-keyboard.log (21.21 KB, text/plain)
2016-09-02 06:41 UTC, mxie@redhat.com
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1104 normal SHIPPED_LIVE Important: qemu-kvm-rhev security, bug fix, and enhancement update 2018-04-10 22:54:38 UTC

Description mxie@redhat.com 2016-09-02 06:40:25 UTC
Created attachment 1197050 [details]
virt-manager-keyboard.log

Description of problem:
Keyboard can't be used when install rhel7 in guest which has SATA CDROM and spice+qxl mode sometimes


Version-Release number of selected component (if applicable):
libvirt-2.0.0-6.el7.x86_64
qemu-kvm-rhev-2.6.0-22.el7.x86_64
kernel: 3.10.0-495.el7.x86_64 
spice-gtk3-0.31-5.el7.x86_64
virt-viewer-2.0-11.el7.x86_64
virt-manager-1.4.0-1.el7.noarch


How reproducible:
50%

Steps to Reproduce:
1.Create a new guest in virt-manager
Open virt-manager-> create a new vm-> select rhel7 iso image to install guest->default setting until select "custom configuration before install"-> change CDROM to SATA type -> Begin installation

2.Enter into installation, at language interface,found the keyboard couldn't used but mouse could be used, I could reproduce this problem at this step sometimes, if you couldn't reproduce this problem at this step, pls go on doing below steps

3.Force off the guest and reconnect the rhel7 iso to CDROM and make CDROM at first order in boot options, and then power on the guest and check the keyboard status after entering into language interface of installation, if the keyboard could be used at this step, pls go on doing step4

4.Force reset the guest and then check the keyboard status after entering into language interface of installation, if the keyboard still could be used, pls repeat step3 and step4 patiently until you could reproduce this problem, I have attached the details log"virt-manager-keyboard.log", "virt-viewer-keyboard.log", "qemu-keyboard.log" for your reference

Actual results:
As above description

Expected results:
Keyboard can be used when install rhel7 in guest which has SATA CDROM and spice+qxl mode at 100%

Comment 1 mxie@redhat.com 2016-09-02 06:41:13 UTC
Created attachment 1197051 [details]
virt-viewer-keyboard.log

Comment 2 mxie@redhat.com 2016-09-02 06:41:44 UTC
Created attachment 1197052 [details]
qemu-keyboard.log

Comment 6 Jonathon Jongsma 2017-05-19 16:10:07 UTC
So, I'm able to reproduce this bug, but the root cause is still eluding me at the moment. There seems to be some interaction between the client and the server. As far as I can tell, it only happens when you reset the guest while connected with a certain spice client. New clients (e.g. spice-gtk built from git) do not trigger the problem, but older ones do. 

However, when you trigger the problem (by rebooting the guest while connected with an older client), the keyboard will still be unusable even if you disconnect that client and re-connect with a newer client that would not otherwise exhibit this problem. That implies that the buggy client somehow affects the guest or host in such a way that the keyboard becomes disabled on the server side.

After bisecting spice-gtk to see where the client problem was fixed, I find this spice-gtk commit:

commit 73cd553fb0fbd213b64d72f8b4289ed8a17fc6c0
Author: Frediano Ziglio <fziglio@redhat.com>
Date:   Wed Sep 7 11:56:36 2016 +0100

    Ignore modifiers messages if no modifiers changed
    
    This avoid keep sending modifiers changes if guest is not
    synchronising the changes.
    
    I consider this as an improving as this avoids client to try again
    and again to force synchronisation however this does not prevent
    every unwanted keystroke insertion which possibly can be a real
    problem on some configurations.
    
    For instance if guest do not handle caps lock as the client do
    if client uses another modifiers (as num lock) this can force
    inserting virtual caps keypress.
    
    Signed-off-by: Frediano Ziglio <fziglio@redhat.com>
    Acked-by: Marc-André Lureau <mlureau@redhat.com>


I'm still trying to investigate how a client that sends a few too many keyboard modifier messages would end up totally disabling the guest's keyboard functionality. Or what it might have to do with SATA cdrom...

Comment 7 Jonathon Jongsma 2017-05-23 21:51:06 UTC
While the above commit does make the bug more difficult (or impossible?) to reproduce, it can't be the full solution. For one thing, the keyboard remains disabled after disconnecting and reconnecting a client, so there must be a bug on the server. After investigating a bit more, I've noticed a difference in the keyboard state between working and non-working resets. Specifically, when I set a breakpoint within ps2_put_keycode() after a reset when the keyboard is not working, I can see that PS2KbdState::scan_enabled is false. After a successful reboot, this value is true. A little more investigation shows that during a reboot where the keyboard works properly, ps2_write_keyboard() is called once with a value of KBD_CMD_RESET_DISABLE, and once with a value of KBD_CMD_RESET_ENABLE. In the case where the keyboard does not work, ps2_write_keyboard() is only called with a value of KBD_CMD_RESET_DISABLE.

Can somebody from the qemu team that is more familiar with this part of the code investigate a little bit further?

Comment 8 Gerd Hoffmann 2017-05-29 07:41:41 UTC
> I'm still trying to investigate how a client that sends a few too many
> keyboard modifier messages would end up totally disabling the guest's
> keyboard functionality. Or what it might have to do with SATA cdrom...

I guess it is the keyboard buffer overflowing, ps/2 has 16 bytes only which isn't that much.  Using SATA might change the timings enough that this actually becomes a problem.

KBD_CMD_RESET_DISABLE and KBD_CMD_RESET_ENABLE are guest writes.  Probably the guest considers the ps/2 keyboard dead.  Due to the buffer overflowing it doesn't get the expected replies.

Try this ...

@@ -556,8 +556,10 @@ void ps2_queue(void *opaque, int b)
     PS2State *s = (PS2State *)opaque;
     PS2Queue *q = &s->queue;
 
-    if (q->count >= PS2_QUEUE_SIZE - 1)
+    if (q->count >= PS2_QUEUE_SIZE - 1) {
+        fprintf(stderr, "%s: queue full\n", __func__);
         return;
+    }
     q->data[q->wptr] = b;
     if (++q->wptr == PS2_QUEUE_SIZE)
         q->wptr = 0;


... and see if it triggers.

Comment 9 Jonathon Jongsma 2017-05-30 20:42:30 UTC
Yes, when the problem is reproduced, that statement does indeed get printed.

Comment 10 Gerd Hoffmann 2017-05-31 09:51:13 UTC
please try this (clears queue on keyboard reset)
https://www.kraxel.org/cgit/qemu/log/?h=work/ps2-queue

Comment 11 Jonathon Jongsma 2017-06-01 21:00:21 UTC
Yes, that series does appear to fix this bug.

Comment 12 Gerd Hoffmann 2017-06-06 11:22:59 UTC
patches sent upstream

Comment 13 Gerd Hoffmann 2017-06-23 11:15:57 UTC
pull request sent.

Comment 14 Gerd Hoffmann 2017-07-12 13:25:59 UTC
(In reply to Gerd Hoffmann from comment #13)
> pull request sent.

Landed upstream.

Comment 16 Guo, Zhiyi 2017-12-24 16:14:34 UTC
Reproduce this issue against qemu-kvm-rhev-2.9.0-16.el7_4.13.x86_64.

Try to install a guest with qemu cli:
/usr/libexec/qemu-kvm \
-name guest=generic,debug-threads=on \
-S \
-machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,vmport=off \
-cpu IvyBridge \
-m 1024 \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 97c2c535-7489-4c4c-8e71-fc58c98f3e01 \
-no-user-config \
-nodefaults \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 \
-device ahci,id=sata0,bus=pci.0,addr=0x5 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 \
-drive file=/home/1372583.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 \
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 \
-drive file=/home/RHEL-7.4-20170711.0-Server-x86_64-dvd1.iso,format=raw,if=none,media=cdrom,id=drive-sata0-0-0,readonly=on \
-device ide-cd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 \
-netdev tap,fd=29,id=hostnet0 \
-device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:ee:97:eb,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-spice port=5900,addr=10.66.8.189,disable-ticketing,image-compression=off,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 \
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 \
-monitor stdio \

steps:
1. Try to install guest with above cli, in languages selection UI, try to input some characters, if you can input, then system_reset and retry.

I meet this issue 4 times in 10 trials.

Verify this issue against qemu-kvm-rhev-2.10.0-12.el7.x86_64, no such issue happen after 20 trials.

Comment 17 Guo, Zhiyi 2017-12-24 16:15:41 UTC
Verified per comment 16

Comment 19 errata-xmlrpc 2018-04-11 00:09:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104


Note You need to log in before you can comment on or make changes to this bug.