RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 823813 - Qemu-kvm abort when do ping-pong migration while guest install
Summary: Qemu-kvm abort when do ping-pong migration while guest install
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.3
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: ---
Assignee: Orit Wasserman
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-05-22 08:37 UTC by Joy Pu
Modified: 2014-03-04 00:24 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-06-21 04:43:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Joy Pu 2012-05-22 08:37:30 UTC
Description of problem:
Do ping-pong migrate during guest install. It will failed to create a guest with a abort message after several round.

do_spice_init: starting 0.10.1
reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
spice_server_add_interface: SPICE_INTERFACE_MIGRATION
spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
spice_server_add_interface: SPICE_INTERFACE_MOUSE
Failed to allocate 3246391296 B: Cannot allocate memory
/bin/sh: line 1: 17381 Aborted                 (core dumped)

And the call trace from the core file is just like this:
(gdb) bt
#0  0x00007fd9e50838a5 in raise () from /lib64/libc.so.6
#1  0x00007fd9e5085085 in abort () from /lib64/libc.so.6
#2  0x00007fd9e77310d8 in qemu_memalign (alignment=2097152, size=3246391296) at osdep.c:112
#3  0x00007fd9e77a5a17 in qemu_ram_alloc_from_ptr (dev=<value optimized out>, name=<value optimized out>, 
    size=3246391296, host=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/exec.c:2756
#4  0x00007fd9e785aa03 in pc_init1 (ram_size=3246391296, boot_device=0x7fff421d4df0 "d", 
    kernel_filename=0x7fff421d6ebb "/root/autotest-devel/client/tests/kvm/images/rhel62-64/vmlinuz", 
    kernel_cmdline=0x7fff421d6f02 "ks=cdrom nicdelay=60 console=ttyS0,115200 console=tty0", 
    initrd_filename=0x7fff421d6f41 "/root/autotest-devel/client/tests/kvm/images/rhel62-64/initrd.img", 
    cpu_model=0x7fff421d6d92 "Opteron_G2", pci_enabled=1) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/pc.c:1122
#5  0x00007fd9e76f83c8 in main (argc=20, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6220

Version-Release number of selected component (if applicable):
host kernel: 2.6.32-269.el6.x86_64 
rpm -qa |grep qemu
qemu-kvm-tools-0.12.1.2-2.290.el6.x86_64
qemu-kvm-0.12.1.2-2.290.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.290.el6.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
qemu-img-0.12.1.2-2.290.el6.x86_64


How reproducible:
once

Steps to Reproduce:
1. Install a guest with cdrom and -kernel -initrd and -append options
2. Do ping-pong migrate during install
  
Actual results:
guest create failed with abort message

Expected results:
guest install succuss during migrate.

Additional info:
processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 107
model name	: AMD Athlon(tm) Dual Core Processor 5400B
stepping	: 2
cpu MHz		: 1000.000
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch lbrv
bogomips	: 2004.11
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps

Comment 3 Orit Wasserman 2012-05-23 05:02:50 UTC
(In reply to comment #0)
> Description of problem:
> Do ping-pong migrate during guest install. It will failed to create a guest
> with a abort message after several round.
> 
> do_spice_init: starting 0.10.1
> reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
> reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
> reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
> spice_server_add_interface: SPICE_INTERFACE_MIGRATION
> spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
> spice_server_add_interface: SPICE_INTERFACE_MOUSE
> Failed to allocate 3246391296 B: Cannot allocate memory
> /bin/sh: line 1: 17381 Aborted                 (core dumped)
> 
> And the call trace from the core file is just like this:
> (gdb) bt
> #0  0x00007fd9e50838a5 in raise () from /lib64/libc.so.6
> #1  0x00007fd9e5085085 in abort () from /lib64/libc.so.6
> #2  0x00007fd9e77310d8 in qemu_memalign (alignment=2097152, size=3246391296)
> at osdep.c:112
> #3  0x00007fd9e77a5a17 in qemu_ram_alloc_from_ptr (dev=<value optimized
> out>, name=<value optimized out>, 
>     size=3246391296, host=<value optimized out>) at
> /usr/src/debug/qemu-kvm-0.12.1.2/exec.c:2756
> #4  0x00007fd9e785aa03 in pc_init1 (ram_size=3246391296,
> boot_device=0x7fff421d4df0 "d", 
>     kernel_filename=0x7fff421d6ebb
> "/root/autotest-devel/client/tests/kvm/images/rhel62-64/vmlinuz", 
>     kernel_cmdline=0x7fff421d6f02 "ks=cdrom nicdelay=60 console=ttyS0,115200
> console=tty0", 
>     initrd_filename=0x7fff421d6f41
> "/root/autotest-devel/client/tests/kvm/images/rhel62-64/initrd.img", 
>     cpu_model=0x7fff421d6d92 "Opteron_G2", pci_enabled=1) at
> /usr/src/debug/qemu-kvm-0.12.1.2/hw/pc.c:1122
> #5  0x00007fd9e76f83c8 in main (argc=20, argv=<value optimized out>,
> envp=<value optimized out>)
>     at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6220
> 
> Version-Release number of selected component (if applicable):
> host kernel: 2.6.32-269.el6.x86_64 
> rpm -qa |grep qemu
> qemu-kvm-tools-0.12.1.2-2.290.el6.x86_64
> qemu-kvm-0.12.1.2-2.290.el6.x86_64
> qemu-kvm-debuginfo-0.12.1.2-2.290.el6.x86_64
> gpxe-roms-qemu-0.9.7-6.9.el6.noarch
> qemu-img-0.12.1.2-2.290.el6.x86_64
> 
> 
> How reproducible:
> once
> 
> Steps to Reproduce:
> 1. Install a guest with cdrom and -kernel -initrd and -append options
> 2. Do ping-pong migrate during install
>   
> Actual results:
> guest create failed with abort message
> 
> Expected results:
> guest install succuss during migrate.
> 
> Additional info:
> processor	: 1
> vendor_id	: AuthenticAMD
> cpu family	: 15
> model		: 107
> model name	: AMD Athlon(tm) Dual Core Processor 5400B
> stepping	: 2
> cpu MHz		: 1000.000
> cache size	: 512 KB
> physical id	: 0
> siblings	: 2
> core id		: 1
> cpu cores	: 2
> apicid		: 1
> initial apicid	: 1
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 1
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36
> clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext
> 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic
> cr8_legacy 3dnowprefetch lbrv
> bogomips	: 2004.11
> TLB size	: 1024 4K pages
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 40 bits physical, 48 bits virtual
> power management: ts fid vid ttp tm stc 100mhzsteps

Can you provide qemu logs ?
what are the dest host cpuinfo ?
what are the hosts memory ?

Comment 4 Dor Laor 2012-05-23 06:40:13 UTC
Also, please report whether a similar scenario but w/o the usage of -kernel/initrd work.

Orit, we have a qemu crash due to alignment issue, it's not related to migration (at least directly). Do you have any ideas?

I moving to 6.4 since it does not look critical.

Comment 5 Orit Wasserman 2012-05-23 07:07:06 UTC
(In reply to comment #4)
> Also, please report whether a similar scenario but w/o the usage of
> -kernel/initrd work.
> 
> Orit, we have a qemu crash due to alignment issue, it's not related to
> migration (at least directly). Do you have any ideas?
> 
> I moving to 6.4 since it does not look critical.

This actually looks like out of memory error (posix_memalign allocates and align the memory ) "Failed to allocate 3246391296 B: Cannot allocate memory".

Comment 6 Joy Pu 2012-05-23 09:55:44 UTC
Hi Orit and Dor,

The migrate is in one host with 4G memory. The guset has 3G memory. And this is all qemu will output when it abort:
do_spice_init: starting 0.10.1
reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
spice_server_add_interface: SPICE_INTERFACE_MIGRATION
spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
spice_server_add_interface: SPICE_INTERFACE_MOUSE
Failed to allocate 3246391296 B: Cannot allocate memory
/bin/sh: line 1: 17381 Aborted                 (core dumped)

I also tried when the guest has 2G and 1G memory. It will not abort.

I try to do migrate without -kernel/initrd. It will abort with 3G memory too while install around 10~15 round. But for a normal guest with 3G memory that is already boot up. It will not abort and can do migrate for 20 round.  So it is not related with the -kernel/initrd options but with the memory size.

Comment 8 Orit Wasserman 2012-06-21 04:43:03 UTC
This behavior is expected as we may exceed the host memory (3G + 3G > 4G) .
We migration a guest on the same host we need at least twice the memory of the guest to make sure you have enough memory.


Note You need to log in before you can comment on or make changes to this bug.