For x86 test results QE will update it here as soon as possible.

(In reply to Min Deng from comment #2)
> For x86 test results QE will update it here as soon as possible.

Tried the bug on x86 platform with rhel74 guest but it was not reproducible.

Update steps for comment0
Migrate guest with the following cli.
migrate -d tcp:0:5555

(In reply to Min Deng from comment #3)
> (In reply to Min Deng from comment #2)
> > For x86 test results QE will update it here as soon as possible.
> 
> Tried the bug on x86 platform with rhel74 guest but it was not reproducible.
> 
> Update steps for comment0
> Migrate guest with the following cli.
> migrate -d tcp:0:5555

Between which steps of comment #0 do you start migration?

(In reply to Laurent Vivier from comment #5)
> (In reply to Min Deng from comment #3)
> > (In reply to Min Deng from comment #2)
> > > For x86 test results QE will update it here as soon as possible.
> > 
> > Tried the bug on x86 platform with rhel74 guest but it was not reproducible.
> > 
> > Update steps for comment0
> > Migrate guest with the following cli.
> > migrate -d tcp:0:5555
> 
> Between which steps of comment #0 do you start migration?

Laurent, 

Min is on pto today. I just tested this bug, started migration after step 4. In my experiment, guest screen doesn't change to black after migration but both mouse and keyboard could not be used.

The issue happens on RHEL6 guest, same as Min reported.

Re-test RHEL7.4 guest, can not reproduce the issue.

We know migration of hotplugged device can be broken on pseries.

There is a series on QEMU mailing list to address the problem:

http://www.mail-archive.com/qemu-devel@nongnu.org/msg448397.html

I'm not able to reproduce the problem.

Do you have X11 running in the guest?
Does the guest really hang or you can log from the serial console or with ssh?

(In reply to Laurent Vivier from comment #8)
> I'm not able to reproduce the problem.
> 
> Do you have X11 running in the guest?
  Desktop was installed.
> Does the guest really hang or you can log from the serial console or with
> ssh?
  The guest could be reached by ssh so it was not really hang from QE's perspective.Already uploaded a relevant screen shot.After rebooting the guest via ssh the guest's keyboard and mouse worked.Thanks !

Created attachment 1277730 [details]
hang

Could you check this BZ 1448810 and BZ 1451631 are not the same bug?
If so, please set this one as a duplicate of BZ 1451631

Thanks

(In reply to Laurent Vivier from comment #11)
> Could you check this BZ 1448810 and BZ 1451631 are not the same bug?
> If so, please set this one as a duplicate of BZ 1451631

They are not the same bug as BZ 1451631 doesn't involve memory hotplug.

I also haven't been able to reproduce bz 1451631, so the triggering circumstances seem to be complex.

I'm a bit confused here - the initial comments say that the screen is black after migration, later comments and the attechement suggest that the screen is not blank, but doesn't respond to keyboard/mouse.  Which is the case?

I'm wondering if there is some problem with the keyboard after migration but with some complicated non-obvious triggering circumstances.  Both 1451631 and this bug might be results of the same thing but via different paths.

(In reply to David Gibson from comment #13)
> I also haven't been able to reproduce bz 1451631, so the triggering
> circumstances seem to be complex.

  Did you install a guest with Desktop ?
> 
> I'm a bit confused here - the initial comments say that the screen is black
> after migration, later comments and the attechement suggest that the screen
> is not blank, but doesn't respond to keyboard/mouse.  Which is the case?
  
  Let me make it more clearly here.Initially,it was not black and after a while it became a blank screen there.

> I'm wondering if there is some problem with the keyboard after migration but
> with some complicated non-obvious triggering circumstances.  Both 1451631
> and this bug might be results of the same thing but via different paths.

  To be honest,my keyboard works well while doing other migration tests.

Updating summary.  I don't think this is really about the blank screen - I think that's just the screesaver kicking in because it's not seeing any keyboard or mouse input for a while.

I do suspect this has the same root cause as bug 1451631.  However it looks like this one has more specific triggering sequence, so we should probably try to debug this first, then see if the fix solves 1451631 as well.

As Laurent says there are known problems with migration after hotplug, but it's not clear why those would cause this specific symptom.

Can you try the following things:

1) After the migration, with the keyboard in non-working state, log in with ssh and get the output of "lsusb -v" in the guest.

2) Again, with the keyboard in non-working state, see if you are able to issue keystrokes in the guest using "sendkey" from the qemu monitor.  e.g.
     (qemu) sendkey x
     (qemu) sendkey ctrl-alt-f4

3) Does the problem occur if you switch the guest to a text console instead of the desktop environment before attempting the migration?

> Can you try the following things:
> 1) After the migration, with the keyboard in non-working state, log in with
> ssh and get the output of "lsusb -v" in the guest.
Output is here
#lsusb -v
-------------------------------------------------------------------------------
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            2.06
  iManufacturer           3 Linux 2.6.32-696.3.1.el6.ppc64 xhci_hcd
  iProduct                2 xHCI Host Controller
  iSerial                 1 0000:00:03.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           25
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             4
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
    TT think time 8 FS bits
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0503 highspeed power enable connect
   Port 2: 0000.0503 highspeed power enable connect
   Port 3: 0000.0100 power
   Port 4: 0000.0503 highspeed power enable connect
Device Status:     0x0001
  Self Powered

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         3 
  bMaxPacketSize0         9
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0003 3.0 root hub
  bcdDevice            2.06
  iManufacturer           3 Linux 2.6.32-696.3.1.el6.ppc64 xhci_hcd
  iProduct                2 xHCI Host Controller
  iSerial                 1 0000:00:03.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           25
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
Hub Descriptor:
  bLength              12
  bDescriptorType      42
  nNbrPorts             4
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  bHubDecLat          0.0 micro seconds
  wHubDelay             0 nano seconds
  DeviceRemovable    0x00
 Hub Port Status:
   Port 1: 0000.02a0 5Gbps power Rx.Detect
   Port 2: 0000.02a0 5Gbps power Rx.Detect
   Port 3: 0000.02a0 5Gbps power Rx.Detect
   Port 4: 0000.02a0 5Gbps power Rx.Detect
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength           15
  bNumDeviceCaps          1
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
      Latency Tolerance Messages (LTM) Supported
    wSpeedsSupported   0x0008
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   0
      Lowest fully-functional device speed is Low Speed (1Mbps)
    bU1DevExitLat           3 micro seconds
    bU2DevExitLat           0 micro seconds
Device Status:     0x0001
  Self Powered

Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0627 Adomax Technology Co., Ltd
  idProduct          0x0001 
  bcdDevice            0.00
  iManufacturer           1 QEMU
  iProduct                4 QEMU USB Keyboard
  iSerial                 5 42
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           34
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          8 HID Keyboard
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      1 Keyboard
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.11
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      63
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               7
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               1.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  bNumConfigurations      1
Device Status:     0x0000
  (Bus Powered)

Bus 001 Device 003: ID 0627:0001 Adomax Technology Co., Ltd 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0627 Adomax Technology Co., Ltd
  idProduct          0x0001 
  bcdDevice            0.00
  iManufacturer           1 QEMU
  iProduct                2 QEMU USB Mouse
  iSerial                 5 42
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           34
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          6 HID Mouse
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      2 Mouse
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               0.01
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      52
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval               7
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               1.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  bNumConfigurations      1
Device Status:     0x0000
  (Bus Powered)

Bus 001 Device 004: ID 0627:0001 Adomax Technology Co., Ltd 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0627 Adomax Technology Co., Ltd
  idProduct          0x0001 
  bcdDevice            0.00
  iManufacturer           1 QEMU
  iProduct                3 QEMU USB Tablet
  iSerial                 5 42
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           34
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          7 HID Tablet
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               0.01
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      74
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               4
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               1.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  bNumConfigurations      1
Device Status:     0x0000
  (Bus Powered) 
> 
> 2) Again, with the keyboard in non-working state, see if you are able to
> issue keystrokes in the guest using "sendkey" from the qemu monitor.  e.g.
>      (qemu) sendkey x
>      (qemu) sendkey ctrl-alt-f4

  It doesn't work even if sendkey from HMP
  (qemu) info mice 
  Mouse #3: QEMU HID Tablet (absolute)
* Mouse #2: QEMU HID Mouse
> 
> 3) Does the problem occur if you switch the guest to a text console instead
> of the desktop environment before attempting the migration?
  Yes,it still occurred after changing it to a text console.

  Guest is rhel6.9 guest with kernel-696.3.1.el6.pp64 installed
  Thanks

I'm able to reproduce the problem.

If I unplug and re-plug the keyboard after migration I can use it again:
 device_del input0
 device_add usb-kbd,id=input0

There is no need of memory hotplug to have the bug, so I think BZ1451631 is a duplicate.

Bisected to:

commit 243afe858b95765b98d16a1f0dd50dca262858ad
Author: Gerd Hoffmann <kraxel>
Date:   Fri Mar 31 12:25:21 2017 +0200

    xhci: flush dequeue pointer to endpoint context
    
    When done processing a endpoint ring we must update the dequeue pointer
    in the endpoint context in guest memory.  This is needed to make sure
    the guest has a correct view of things and also to make live migration
    work properly, because xhci post_load restores alot of the state from
    xhci data structures in guest memory.
    
    Add xhci_set_ep_state() call to do that.
    
    The recursive calls stopped by commit
    ddb603ab6c981c1d67cb42266fc700c33e5b2d8f had the (unintentional) side
    effect to hiding this bug.  xhci_set_ep_state() was called before
    processing, to set the state to running, which updated the dequeue
    pointer too.
    
    Reported-by: Dr. David Alan Gilbert <dgilbert>
    Signed-off-by: Gerd Hoffmann <kraxel>
    Tested-by: Dr. David Alan Gilbert <dgilbert>
    Message-id: 20170331102521.29253-1-kraxel

Gerd, any idea?

(In reply to Laurent Vivier from comment #19)
> Gerd, any idea?

No clue offhand.
There is bug 1454580 which looks simliar.
Not debugged yet.

Min Deng,

As there is no reason this problem is ppc64le only, could you recheck on x86_64, being sure you use nec-usb-xhci device?

(In reply to Laurent Vivier from comment #21)
> Min Deng,
> 
> As there is no reason this problem is ppc64le only, could you recheck on
> x86_64, being sure you use nec-usb-xhci device?

Thanks for developer's reply.
Do not involve memory hot_plug test steps any more at this time.Again,QE double confirmed the issue on x86 platform on rhel6.9 guest.Basically,they had the *similar* issue but there was still some difference between power and x86 platform.

In brief,
*x86 platform,keyboard doesn't work but mouse can work after migration
*power,neither keyboard nor mouse can work after migration

X86 platform Build info,
kernel-3.10.0-671.el7.x86_64 (host and guest)
qemu-kvm-rhev-2.9.0-7.el7.x86_64

Power platform Build info,
kernel-3.10.0-675.el7.ppc64le (host and guest)
qemu-img-rhev-2.9.0-7.el7.ppc64le

Anyway,we still need a bug to trace the difference,if I was wrong please correct me.Feel free to let me know if you have any concerns.Thanks a lot.
 
Relevant clis, 
1.boot up SRC guest with 
  /usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc -nodefaults -vga std -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 -device usb-kbd,id=input0 -device usb-mouse,id=input1 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=rhel69-64-virtio.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=1,bus=pci.0,addr=0x4 -device virtio-net-pci,mac=9a:93:94:95:96:97,id=idpWdRJ9,vectors=4,netdev=idkTBKmY,bus=pci.0,addr=0x5 -netdev tap,id=idkTBKmY,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-down,id=hostnet1 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x6 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=4 -vnc :0 -rtc base=utc,clock=host -boot order=cdn,once=d,menu=off,strict=off -enable-kvm -monitor stdio -m 4G,slots=32,maxmem=80G -numa node,nodeid=0 -numa node,nodeid=1 -monitor unix:/tmp/monitor3,server,nowait -qmp tcp:0:4444,server,nowait
2.Boot up DST side 
  /usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc -nodefaults -vga std -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 -device usb-kbd,id=input0 -device usb-mouse,id=input1 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=rhel69-64-virtio.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=1,bus=pci.0,addr=0x4 -device virtio-net-pci,mac=9a:93:94:95:96:17,id=idpWdRJ9,vectors=4,netdev=idkTBKmY,bus=pci.0,addr=0x5 -netdev tap,id=idkTBKmY,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-down,id=hostnet1 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x6 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=4 -vnc :2 -rtc base=utc,clock=host -boot order=cdn,once=d,menu=off,strict=off -enable-kvm -monitor stdio -m 4G,slots=32,maxmem=80G -numa node,nodeid=0 -numa node,nodeid=1 -monitor unix:/tmp/monitor3,server,nowait -qmp tcp:0:4446,server,nowait -incoming tcp:0:1212

I set this BZ as a duplicate of BZ1451631, as the problem is with xhci device migration, not with memory hotplugging.

*** This bug has been marked as a duplicate of bug 1451631 ***

(In reply to Gerd Hoffmann from comment #20)
> (In reply to Laurent Vivier from comment #19)
> > Gerd, any idea?
> 
> No clue offhand.
> There is bug 1454580 which looks simliar.
> Not debugged yet.

Can you give this one a spin on ppc64?
https://www.kraxel.org/cgit/qemu/log/?h=work/xhci-hid-migration

(In reply to Gerd Hoffmann from comment #24)
> (In reply to Gerd Hoffmann from comment #20)
> > (In reply to Laurent Vivier from comment #19)
> > > Gerd, any idea?
> > 
> > No clue offhand.
> > There is bug 1454580 which looks simliar.
> > Not debugged yet.
> 
> Can you give this one a spin on ppc64?
> https://www.kraxel.org/cgit/qemu/log/?h=work/xhci-hid-migration

Yes, it fixes the problem.