Bug 1448810
| Summary: | Guest loses keyboard and/or mouse after migration after hotplug | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Min Deng <mdeng> | ||||
| Component: | qemu-kvm-rhev | Assignee: | Laurent Vivier <lvivier> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.4 | CC: | dgibson, dgilbert, knoel, kraxel, lvivier, mdeng, peterx, quintela, qzhang, virt-maint | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | ppc64le | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-06-07 08:32:51 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Min Deng
2017-05-08 08:54:04 UTC
For x86 test results QE will update it here as soon as possible. (In reply to Min Deng from comment #2) > For x86 test results QE will update it here as soon as possible. Tried the bug on x86 platform with rhel74 guest but it was not reproducible. Update steps for comment0 Migrate guest with the following cli. migrate -d tcp:0:5555 (In reply to Min Deng from comment #3) > (In reply to Min Deng from comment #2) > > For x86 test results QE will update it here as soon as possible. > > Tried the bug on x86 platform with rhel74 guest but it was not reproducible. > > Update steps for comment0 > Migrate guest with the following cli. > migrate -d tcp:0:5555 Between which steps of comment #0 do you start migration? (In reply to Laurent Vivier from comment #5) > (In reply to Min Deng from comment #3) > > (In reply to Min Deng from comment #2) > > > For x86 test results QE will update it here as soon as possible. > > > > Tried the bug on x86 platform with rhel74 guest but it was not reproducible. > > > > Update steps for comment0 > > Migrate guest with the following cli. > > migrate -d tcp:0:5555 > > Between which steps of comment #0 do you start migration? Laurent, Min is on pto today. I just tested this bug, started migration after step 4. In my experiment, guest screen doesn't change to black after migration but both mouse and keyboard could not be used. The issue happens on RHEL6 guest, same as Min reported. Re-test RHEL7.4 guest, can not reproduce the issue. We know migration of hotplugged device can be broken on pseries. There is a series on QEMU mailing list to address the problem: http://www.mail-archive.com/qemu-devel@nongnu.org/msg448397.html I'm not able to reproduce the problem. Do you have X11 running in the guest? Does the guest really hang or you can log from the serial console or with ssh? (In reply to Laurent Vivier from comment #8) > I'm not able to reproduce the problem. > > Do you have X11 running in the guest? Desktop was installed. > Does the guest really hang or you can log from the serial console or with > ssh? The guest could be reached by ssh so it was not really hang from QE's perspective.Already uploaded a relevant screen shot.After rebooting the guest via ssh the guest's keyboard and mouse worked.Thanks ! Created attachment 1277730 [details]
hang
Could you check this BZ 1448810 and BZ 1451631 are not the same bug? If so, please set this one as a duplicate of BZ 1451631 Thanks (In reply to Laurent Vivier from comment #11) > Could you check this BZ 1448810 and BZ 1451631 are not the same bug? > If so, please set this one as a duplicate of BZ 1451631 They are not the same bug as BZ 1451631 doesn't involve memory hotplug. I also haven't been able to reproduce bz 1451631, so the triggering circumstances seem to be complex. I'm a bit confused here - the initial comments say that the screen is black after migration, later comments and the attechement suggest that the screen is not blank, but doesn't respond to keyboard/mouse. Which is the case? I'm wondering if there is some problem with the keyboard after migration but with some complicated non-obvious triggering circumstances. Both 1451631 and this bug might be results of the same thing but via different paths. (In reply to David Gibson from comment #13) > I also haven't been able to reproduce bz 1451631, so the triggering > circumstances seem to be complex. Did you install a guest with Desktop ? > > I'm a bit confused here - the initial comments say that the screen is black > after migration, later comments and the attechement suggest that the screen > is not blank, but doesn't respond to keyboard/mouse. Which is the case? Let me make it more clearly here.Initially,it was not black and after a while it became a blank screen there. > I'm wondering if there is some problem with the keyboard after migration but > with some complicated non-obvious triggering circumstances. Both 1451631 > and this bug might be results of the same thing but via different paths. To be honest,my keyboard works well while doing other migration tests. Updating summary. I don't think this is really about the blank screen - I think that's just the screesaver kicking in because it's not seeing any keyboard or mouse input for a while. I do suspect this has the same root cause as bug 1451631. However it looks like this one has more specific triggering sequence, so we should probably try to debug this first, then see if the fix solves 1451631 as well. As Laurent says there are known problems with migration after hotplug, but it's not clear why those would cause this specific symptom. Can you try the following things: 1) After the migration, with the keyboard in non-working state, log in with ssh and get the output of "lsusb -v" in the guest. 2) Again, with the keyboard in non-working state, see if you are able to issue keystrokes in the guest using "sendkey" from the qemu monitor. e.g. (qemu) sendkey x (qemu) sendkey ctrl-alt-f4 3) Does the problem occur if you switch the guest to a text console instead of the desktop environment before attempting the migration? > Can you try the following things: > 1) After the migration, with the keyboard in non-working state, log in with > ssh and get the output of "lsusb -v" in the guest. Output is here #lsusb -v ------------------------------------------------------------------------------- Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.00 bDeviceClass 9 Hub bDeviceSubClass 0 Unused bDeviceProtocol 1 Single TT bMaxPacketSize0 64 idVendor 0x1d6b Linux Foundation idProduct 0x0002 2.0 root hub bcdDevice 2.06 iManufacturer 3 Linux 2.6.32-696.3.1.el6.ppc64 xhci_hcd iProduct 2 xHCI Host Controller iSerial 1 0000:00:03.0 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 25 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 9 Hub bInterfaceSubClass 0 Unused bInterfaceProtocol 0 Full speed (or root) hub iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0004 1x 4 bytes bInterval 12 Hub Descriptor: bLength 9 bDescriptorType 41 nNbrPorts 4 wHubCharacteristic 0x000a No power switching (usb 1.0) Per-port overcurrent protection TT think time 8 FS bits bPwrOn2PwrGood 10 * 2 milli seconds bHubContrCurrent 0 milli Ampere DeviceRemovable 0x00 PortPwrCtrlMask 0xff Hub Port Status: Port 1: 0000.0503 highspeed power enable connect Port 2: 0000.0503 highspeed power enable connect Port 3: 0000.0100 power Port 4: 0000.0503 highspeed power enable connect Device Status: 0x0001 Self Powered Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 3.00 bDeviceClass 9 Hub bDeviceSubClass 0 Unused bDeviceProtocol 3 bMaxPacketSize0 9 idVendor 0x1d6b Linux Foundation idProduct 0x0003 3.0 root hub bcdDevice 2.06 iManufacturer 3 Linux 2.6.32-696.3.1.el6.ppc64 xhci_hcd iProduct 2 xHCI Host Controller iSerial 1 0000:00:03.0 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 25 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 9 Hub bInterfaceSubClass 0 Unused bInterfaceProtocol 0 Full speed (or root) hub iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0004 1x 4 bytes bInterval 12 Hub Descriptor: bLength 12 bDescriptorType 42 nNbrPorts 4 wHubCharacteristic 0x000a No power switching (usb 1.0) Per-port overcurrent protection bPwrOn2PwrGood 10 * 2 milli seconds bHubContrCurrent 0 milli Ampere bHubDecLat 0.0 micro seconds wHubDelay 0 nano seconds DeviceRemovable 0x00 Hub Port Status: Port 1: 0000.02a0 5Gbps power Rx.Detect Port 2: 0000.02a0 5Gbps power Rx.Detect Port 3: 0000.02a0 5Gbps power Rx.Detect Port 4: 0000.02a0 5Gbps power Rx.Detect Binary Object Store Descriptor: bLength 5 bDescriptorType 15 wTotalLength 15 bNumDeviceCaps 1 SuperSpeed USB Device Capability: bLength 10 bDescriptorType 16 bDevCapabilityType 3 bmAttributes 0x00 Latency Tolerance Messages (LTM) Supported wSpeedsSupported 0x0008 Device can operate at SuperSpeed (5Gbps) bFunctionalitySupport 0 Lowest fully-functional device speed is Low Speed (1Mbps) bU1DevExitLat 3 micro seconds bU2DevExitLat 0 micro seconds Device Status: 0x0001 Self Powered Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.00 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 64 idVendor 0x0627 Adomax Technology Co., Ltd idProduct 0x0001 bcdDevice 0.00 iManufacturer 1 QEMU iProduct 4 QEMU USB Keyboard iSerial 5 42 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 34 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 8 HID Keyboard bmAttributes 0xa0 (Bus Powered) Remote Wakeup MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 3 Human Interface Device bInterfaceSubClass 1 Boot Interface Subclass bInterfaceProtocol 1 Keyboard iInterface 0 HID Device Descriptor: bLength 9 bDescriptorType 33 bcdHID 1.11 bCountryCode 0 Not supported bNumDescriptors 1 bDescriptorType 34 Report wDescriptorLength 63 Report Descriptors: ** UNAVAILABLE ** Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0008 1x 8 bytes bInterval 7 Device Qualifier (for other device speed): bLength 10 bDescriptorType 6 bcdUSB 1.00 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 8 bNumConfigurations 1 Device Status: 0x0000 (Bus Powered) Bus 001 Device 003: ID 0627:0001 Adomax Technology Co., Ltd Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.00 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 64 idVendor 0x0627 Adomax Technology Co., Ltd idProduct 0x0001 bcdDevice 0.00 iManufacturer 1 QEMU iProduct 2 QEMU USB Mouse iSerial 5 42 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 34 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 6 HID Mouse bmAttributes 0xa0 (Bus Powered) Remote Wakeup MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 3 Human Interface Device bInterfaceSubClass 1 Boot Interface Subclass bInterfaceProtocol 2 Mouse iInterface 0 HID Device Descriptor: bLength 9 bDescriptorType 33 bcdHID 0.01 bCountryCode 0 Not supported bNumDescriptors 1 bDescriptorType 34 Report wDescriptorLength 52 Report Descriptors: ** UNAVAILABLE ** Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0004 1x 4 bytes bInterval 7 Device Qualifier (for other device speed): bLength 10 bDescriptorType 6 bcdUSB 1.00 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 8 bNumConfigurations 1 Device Status: 0x0000 (Bus Powered) Bus 001 Device 004: ID 0627:0001 Adomax Technology Co., Ltd Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.00 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 64 idVendor 0x0627 Adomax Technology Co., Ltd idProduct 0x0001 bcdDevice 0.00 iManufacturer 1 QEMU iProduct 3 QEMU USB Tablet iSerial 5 42 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 34 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 7 HID Tablet bmAttributes 0xa0 (Bus Powered) Remote Wakeup MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 3 Human Interface Device bInterfaceSubClass 0 No Subclass bInterfaceProtocol 0 None iInterface 0 HID Device Descriptor: bLength 9 bDescriptorType 33 bcdHID 0.01 bCountryCode 0 Not supported bNumDescriptors 1 bDescriptorType 34 Report wDescriptorLength 74 Report Descriptors: ** UNAVAILABLE ** Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0008 1x 8 bytes bInterval 4 Device Qualifier (for other device speed): bLength 10 bDescriptorType 6 bcdUSB 1.00 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 8 bNumConfigurations 1 Device Status: 0x0000 (Bus Powered) > > 2) Again, with the keyboard in non-working state, see if you are able to > issue keystrokes in the guest using "sendkey" from the qemu monitor. e.g. > (qemu) sendkey x > (qemu) sendkey ctrl-alt-f4 It doesn't work even if sendkey from HMP (qemu) info mice Mouse #3: QEMU HID Tablet (absolute) * Mouse #2: QEMU HID Mouse > > 3) Does the problem occur if you switch the guest to a text console instead > of the desktop environment before attempting the migration? Yes,it still occurred after changing it to a text console. Guest is rhel6.9 guest with kernel-696.3.1.el6.pp64 installed Thanks I'm able to reproduce the problem. If I unplug and re-plug the keyboard after migration I can use it again: device_del input0 device_add usb-kbd,id=input0 There is no need of memory hotplug to have the bug, so I think BZ1451631 is a duplicate. Bisected to: commit 243afe858b95765b98d16a1f0dd50dca262858ad Author: Gerd Hoffmann <kraxel> Date: Fri Mar 31 12:25:21 2017 +0200 xhci: flush dequeue pointer to endpoint context When done processing a endpoint ring we must update the dequeue pointer in the endpoint context in guest memory. This is needed to make sure the guest has a correct view of things and also to make live migration work properly, because xhci post_load restores alot of the state from xhci data structures in guest memory. Add xhci_set_ep_state() call to do that. The recursive calls stopped by commit ddb603ab6c981c1d67cb42266fc700c33e5b2d8f had the (unintentional) side effect to hiding this bug. xhci_set_ep_state() was called before processing, to set the state to running, which updated the dequeue pointer too. Reported-by: Dr. David Alan Gilbert <dgilbert> Signed-off-by: Gerd Hoffmann <kraxel> Tested-by: Dr. David Alan Gilbert <dgilbert> Message-id: 20170331102521.29253-1-kraxel Gerd, any idea? (In reply to Laurent Vivier from comment #19) > Gerd, any idea? No clue offhand. There is bug 1454580 which looks simliar. Not debugged yet. Min Deng, As there is no reason this problem is ppc64le only, could you recheck on x86_64, being sure you use nec-usb-xhci device? (In reply to Laurent Vivier from comment #21) > Min Deng, > > As there is no reason this problem is ppc64le only, could you recheck on > x86_64, being sure you use nec-usb-xhci device? Thanks for developer's reply. Do not involve memory hot_plug test steps any more at this time.Again,QE double confirmed the issue on x86 platform on rhel6.9 guest.Basically,they had the *similar* issue but there was still some difference between power and x86 platform. In brief, *x86 platform,keyboard doesn't work but mouse can work after migration *power,neither keyboard nor mouse can work after migration X86 platform Build info, kernel-3.10.0-671.el7.x86_64 (host and guest) qemu-kvm-rhev-2.9.0-7.el7.x86_64 Power platform Build info, kernel-3.10.0-675.el7.ppc64le (host and guest) qemu-img-rhev-2.9.0-7.el7.ppc64le Anyway,we still need a bug to trace the difference,if I was wrong please correct me.Feel free to let me know if you have any concerns.Thanks a lot. Relevant clis, 1.boot up SRC guest with /usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc -nodefaults -vga std -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 -device usb-kbd,id=input0 -device usb-mouse,id=input1 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=rhel69-64-virtio.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=1,bus=pci.0,addr=0x4 -device virtio-net-pci,mac=9a:93:94:95:96:97,id=idpWdRJ9,vectors=4,netdev=idkTBKmY,bus=pci.0,addr=0x5 -netdev tap,id=idkTBKmY,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-down,id=hostnet1 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x6 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=4 -vnc :0 -rtc base=utc,clock=host -boot order=cdn,once=d,menu=off,strict=off -enable-kvm -monitor stdio -m 4G,slots=32,maxmem=80G -numa node,nodeid=0 -numa node,nodeid=1 -monitor unix:/tmp/monitor3,server,nowait -qmp tcp:0:4444,server,nowait 2.Boot up DST side /usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc -nodefaults -vga std -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 -device usb-kbd,id=input0 -device usb-mouse,id=input1 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=rhel69-64-virtio.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=1,bus=pci.0,addr=0x4 -device virtio-net-pci,mac=9a:93:94:95:96:17,id=idpWdRJ9,vectors=4,netdev=idkTBKmY,bus=pci.0,addr=0x5 -netdev tap,id=idkTBKmY,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-down,id=hostnet1 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x6 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=4 -vnc :2 -rtc base=utc,clock=host -boot order=cdn,once=d,menu=off,strict=off -enable-kvm -monitor stdio -m 4G,slots=32,maxmem=80G -numa node,nodeid=0 -numa node,nodeid=1 -monitor unix:/tmp/monitor3,server,nowait -qmp tcp:0:4446,server,nowait -incoming tcp:0:1212 I set this BZ as a duplicate of BZ1451631, as the problem is with xhci device migration, not with memory hotplugging. *** This bug has been marked as a duplicate of bug 1451631 *** (In reply to Gerd Hoffmann from comment #20) > (In reply to Laurent Vivier from comment #19) > > Gerd, any idea? > > No clue offhand. > There is bug 1454580 which looks simliar. > Not debugged yet. Can you give this one a spin on ppc64? https://www.kraxel.org/cgit/qemu/log/?h=work/xhci-hid-migration (In reply to Gerd Hoffmann from comment #24) > (In reply to Gerd Hoffmann from comment #20) > > (In reply to Laurent Vivier from comment #19) > > > Gerd, any idea? > > > > No clue offhand. > > There is bug 1454580 which looks simliar. > > Not debugged yet. > > Can you give this one a spin on ppc64? > https://www.kraxel.org/cgit/qemu/log/?h=work/xhci-hid-migration Yes, it fixes the problem. |