821655 – Guest sometimes hangs after migrating from RHEL6.1 host to RHEL6.3 host with -M rhel6.1.0 with spice+qxl

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 821655 - Guest sometimes hangs after migrating from RHEL6.1 host to RHEL6.3 host with -M rhel6.1.0 with spice+qxl

Summary: Guest sometimes hangs after migrating from RHEL6.1 host to RHEL6.3 host with ...

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	6.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	David Blechter
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-05-15 09:22 UTC by Qunfang Zhang
Modified:	2017-12-06 11:44 UTC (History)
CC List:	12 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-12-06 11:44:30 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
qemu monitor log (3.92 KB, text/plain) 2012-07-05 04:16 UTC, Qunfang Zhang	no flags	Details
spice log (Xorg.0.log in guest) (76.20 KB, application/octet-stream) 2012-07-05 04:17 UTC, Qunfang Zhang	no flags	Details
strace -p `pidof qemu-kvm` when the guest hangs (1.73 MB, text/plain) 2012-07-05 04:19 UTC, Qunfang Zhang	no flags	Details
View All

Description Qunfang Zhang 2012-05-15 09:22:19 UTC

Description of problem:
Hit this issue during verifying bug 698936. After migrate a rhel6.3 guest from rhel6.1 host to rhel6.3 host with "-M rhel6.1.0", guest sometimes hangs and consumes 100% cpu. Remove "-vga qxl" did not hit the issue after more than 15 times attempts. 

Version-Release number of selected component (if applicable):
RHEL6.1 host:
kernel-2.6.32-131.0.15.el6.x86_64
qemu-kvm-0.12.1.2-2.160.el6.x86_64
seabios-0.6.1.2-3.el6.x86_64
spice-server-0.8.0-1.el6.x86_64

RHEL6.3 host:
kernel-2.6.32-270.el6.x86_64
qemu-kvm-0.12.1.2-2.293.el6.x86_64
seabios-0.6.1.2-19.el6.x86_64
spice-server-0.10.1-6.el6.x86_64

Guest:
kernel-2.6.32-269.el6.x86_64

How reproducible:
Sometimes. about 1/10 or 2/10

Steps to Reproduce:
1. Boot guest on rhel6.1 host with the following command:

/usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -m 1024 -smp 2 -name rhel6.3-64 -uuid 4c21b4e8-de85-4e31-89b6-8b746d1eeca2 -rtc base=localtime,clock=host,driftfix=slew -boot c  -drive file=/mnt/RHEL-Server-6.3-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,werror=stop,rerror=stop -device ide-drive,bus=ide.0,unit=1,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet1,script=/etc/qemu-ifup,downscript=no -device rtl8139,netdev=hostnet1,id=net1,mac=64:31:50:41:e1:13 -netdev tap,id=hostnet3,script=/etc/qemu-ifup,downscript=no -device e1000,netdev=hostnet3,id=net3,mac=64:31:50:41:b1:e3  -monitor stdio -qmp tcp:0:5555,server,nowait -spice port=9000,disable-ticketing -vga qxl -device qxl,vram_size=67108864  -device virtio-balloon-pci,id=balloon0,bus=pci.0,id=0x6 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=17,bus=pci.0,addr=0x7

2. Boot with listening mode on dst rhel6.3 host:
Append "-incoming tcp:0:5800"

3. Ping-pong migration between the 2 hosts.
  
Actual results:
Sometimes guest hangs, mostly on the dst rhel6.3 host. And only for once I hit it on the source rhel6.1 host.

Expected results:
Guest should not hang. And should migrate successfully.

Additional info:
Tried the additional scenarios after talked with uri:
(1) Migrate between 2 rhel6.3 host, tried both "-M rhel6.1.0" and "-M rhel6.3.0".
==> Passed.
(2) Before migration, using "__com.redhat_spice_migrate_info" first. Then migrate guest.
==> Reproduced.
(3) Close the spice client during migration, and open it again after finish migration.
==> Reproduced.

Comment 1 Qunfang Zhang 2012-05-15 09:30:17 UTC

Additional phenomenon:
After migrate, the guest sometimes log out, and I need to re-login again.

Comment 3 Karen Noel 2012-06-22 17:09:49 UTC

Can you provide more info about what the qemu thread is doing when it consumes 100% cpu? 

Also, please let us know the hardware info for the source and target host. What type of processors? Can you give us login access?

Comment 7 Orit Wasserman 2012-07-03 13:51:46 UTC

Hi,
your hosts time is not synced (around 30s diff) ,this can cause the hangs when migrating see https://bugzilla.redhat.com/show_bug.cgi?id=731100.
Can you sync the hosts clock and see if it helps ?

Comment 8 Qunfang Zhang 2012-07-04 02:41:21 UTC

(In reply to comment #7)
> Hi,
> your hosts time is not synced (around 30s diff) ,this can cause the hangs
> when migrating see https://bugzilla.redhat.com/show_bug.cgi?id=731100.
> Can you sync the hosts clock and see if it helps ?

Hi, Orit
Actually the issue can be reproduced when sync the time of the two hosts. It happened when I reported the bug, and also happened on my re-test just now. :-)
Sorry I'm not aware the two hosts time are different when I provided the environment. 

Could you help check it again? 

Thanks.

Comment 9 Orit Wasserman 2012-07-04 09:38:22 UTC

Can you provide with the output of qemu monitor when this happens ?
Can you provide me with spice logs too ?

Comment 10 Orit Wasserman 2012-07-04 09:39:26 UTC

Also strace output of qemu when the guest hangs .

Comment 11 Qunfang Zhang 2012-07-05 04:16:22 UTC

Created attachment 596316 [details]
qemu monitor log

Comment 12 Qunfang Zhang 2012-07-05 04:17:18 UTC

Created attachment 596317 [details]
spice log (Xorg.0.log in guest)

Comment 13 Qunfang Zhang 2012-07-05 04:19:10 UTC

Created attachment 596319 [details]
strace -p `pidof qemu-kvm` when the guest hangs

Hi, please help check the logs.

And when the guest hangs, top inside guest shows Xorg process consume 100% guest cpu.
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2629 root      20   0  277m  51m 7076 R 100.0  5.1   0:17.91 Xorg              
 3096 root      20   0 15028 1196  892 R  0.3  0.1   0:00.06 top                
    1 root      20   0 19348 1524 1212 S  0.0  0.1   0:00.66 init

Comment 14 Qunfang Zhang 2012-07-05 04:22:43 UTC

xorg-x11-drv-qxl version of guest:
xorg-x11-drv-qxl-0.0.14-13.el6_2.x86_64

Comment 15 RHEL Program Management 2012-07-10 06:27:33 UTC

This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 16 Alon Levy 2012-07-10 08:34:16 UTC

(In reply to comment #13)
> Created attachment 596319 [details]
> strace -p `pidof qemu-kvm` when the guest hangs
> 
> Hi, please help check the logs.
> 
> And when the guest hangs, top inside guest shows Xorg process consume 100%
> guest cpu.
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND        
> 
>  2629 root      20   0  277m  51m 7076 R 100.0  5.1   0:17.91 Xorg          
> 
>  3096 root      20   0 15028 1196  892 R  0.3  0.1   0:00.06 top            
> 
>     1 root      20   0 19348 1524 1212 S  0.0  0.1   0:00.66 init

I tried reproducing with your machines and failed (i.e. successfull migration) but only tried once. What is the frequency/percent of the time this happens?

Can you provide output of perf top or just attach with gdb and break (hoping it will be relevant) to Xorg when the 100% happens? Can you ssh to the vm when this happens?

Thanks,
Alon

Comment 18 RHEL Program Management 2012-07-11 02:07:53 UTC

This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.

Comment 19 RHEL Program Management 2012-12-14 07:23:44 UTC

This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 23 Søren Sandmann Pedersen 2013-01-11 13:45:42 UTC

This question from Alon is the first thing we need an answer to.

> Can you provide output of perf top or just attach with gdb and break (hoping it will be relevant) to Xorg when the 100% happens? Can you ssh to the vm when this happens?

Adding needinfo(areis).

Comment 25 Jan Kurik 2017-12-06 11:44:30 UTC

Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available.

The official life cycle policy can be reviewed here:

http://redhat.com/rhel/lifecycle

This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL:

https://access.redhat.com/

Note You need to log in before you can comment on or make changes to this bug.