Bug 823813 - Qemu-kvm abort when do ping-pong migration while guest install
Qemu-kvm abort when do ping-pong migration while guest install
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.3
Unspecified Unspecified
high Severity medium
: rc
: ---
Assigned To: Orit Wasserman
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-22 04:37 EDT by Joy Pu
Modified: 2014-03-03 19:24 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-21 00:43:03 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Joy Pu 2012-05-22 04:37:30 EDT
Description of problem:
Do ping-pong migrate during guest install. It will failed to create a guest with a abort message after several round.

do_spice_init: starting 0.10.1
reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
spice_server_add_interface: SPICE_INTERFACE_MIGRATION
spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
spice_server_add_interface: SPICE_INTERFACE_MOUSE
Failed to allocate 3246391296 B: Cannot allocate memory
/bin/sh: line 1: 17381 Aborted                 (core dumped)

And the call trace from the core file is just like this:
(gdb) bt
#0  0x00007fd9e50838a5 in raise () from /lib64/libc.so.6
#1  0x00007fd9e5085085 in abort () from /lib64/libc.so.6
#2  0x00007fd9e77310d8 in qemu_memalign (alignment=2097152, size=3246391296) at osdep.c:112
#3  0x00007fd9e77a5a17 in qemu_ram_alloc_from_ptr (dev=<value optimized out>, name=<value optimized out>, 
    size=3246391296, host=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/exec.c:2756
#4  0x00007fd9e785aa03 in pc_init1 (ram_size=3246391296, boot_device=0x7fff421d4df0 "d", 
    kernel_filename=0x7fff421d6ebb "/root/autotest-devel/client/tests/kvm/images/rhel62-64/vmlinuz", 
    kernel_cmdline=0x7fff421d6f02 "ks=cdrom nicdelay=60 console=ttyS0,115200 console=tty0", 
    initrd_filename=0x7fff421d6f41 "/root/autotest-devel/client/tests/kvm/images/rhel62-64/initrd.img", 
    cpu_model=0x7fff421d6d92 "Opteron_G2", pci_enabled=1) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/pc.c:1122
#5  0x00007fd9e76f83c8 in main (argc=20, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6220

Version-Release number of selected component (if applicable):
host kernel: 2.6.32-269.el6.x86_64 
rpm -qa |grep qemu
qemu-kvm-tools-0.12.1.2-2.290.el6.x86_64
qemu-kvm-0.12.1.2-2.290.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.290.el6.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
qemu-img-0.12.1.2-2.290.el6.x86_64


How reproducible:
once

Steps to Reproduce:
1. Install a guest with cdrom and -kernel -initrd and -append options
2. Do ping-pong migrate during install
  
Actual results:
guest create failed with abort message

Expected results:
guest install succuss during migrate.

Additional info:
processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 107
model name	: AMD Athlon(tm) Dual Core Processor 5400B
stepping	: 2
cpu MHz		: 1000.000
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch lbrv
bogomips	: 2004.11
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps
Comment 3 Orit Wasserman 2012-05-23 01:02:50 EDT
(In reply to comment #0)
> Description of problem:
> Do ping-pong migrate during guest install. It will failed to create a guest
> with a abort message after several round.
> 
> do_spice_init: starting 0.10.1
> reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
> reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
> reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
> spice_server_add_interface: SPICE_INTERFACE_MIGRATION
> spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
> spice_server_add_interface: SPICE_INTERFACE_MOUSE
> Failed to allocate 3246391296 B: Cannot allocate memory
> /bin/sh: line 1: 17381 Aborted                 (core dumped)
> 
> And the call trace from the core file is just like this:
> (gdb) bt
> #0  0x00007fd9e50838a5 in raise () from /lib64/libc.so.6
> #1  0x00007fd9e5085085 in abort () from /lib64/libc.so.6
> #2  0x00007fd9e77310d8 in qemu_memalign (alignment=2097152, size=3246391296)
> at osdep.c:112
> #3  0x00007fd9e77a5a17 in qemu_ram_alloc_from_ptr (dev=<value optimized
> out>, name=<value optimized out>, 
>     size=3246391296, host=<value optimized out>) at
> /usr/src/debug/qemu-kvm-0.12.1.2/exec.c:2756
> #4  0x00007fd9e785aa03 in pc_init1 (ram_size=3246391296,
> boot_device=0x7fff421d4df0 "d", 
>     kernel_filename=0x7fff421d6ebb
> "/root/autotest-devel/client/tests/kvm/images/rhel62-64/vmlinuz", 
>     kernel_cmdline=0x7fff421d6f02 "ks=cdrom nicdelay=60 console=ttyS0,115200
> console=tty0", 
>     initrd_filename=0x7fff421d6f41
> "/root/autotest-devel/client/tests/kvm/images/rhel62-64/initrd.img", 
>     cpu_model=0x7fff421d6d92 "Opteron_G2", pci_enabled=1) at
> /usr/src/debug/qemu-kvm-0.12.1.2/hw/pc.c:1122
> #5  0x00007fd9e76f83c8 in main (argc=20, argv=<value optimized out>,
> envp=<value optimized out>)
>     at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6220
> 
> Version-Release number of selected component (if applicable):
> host kernel: 2.6.32-269.el6.x86_64 
> rpm -qa |grep qemu
> qemu-kvm-tools-0.12.1.2-2.290.el6.x86_64
> qemu-kvm-0.12.1.2-2.290.el6.x86_64
> qemu-kvm-debuginfo-0.12.1.2-2.290.el6.x86_64
> gpxe-roms-qemu-0.9.7-6.9.el6.noarch
> qemu-img-0.12.1.2-2.290.el6.x86_64
> 
> 
> How reproducible:
> once
> 
> Steps to Reproduce:
> 1. Install a guest with cdrom and -kernel -initrd and -append options
> 2. Do ping-pong migrate during install
>   
> Actual results:
> guest create failed with abort message
> 
> Expected results:
> guest install succuss during migrate.
> 
> Additional info:
> processor	: 1
> vendor_id	: AuthenticAMD
> cpu family	: 15
> model		: 107
> model name	: AMD Athlon(tm) Dual Core Processor 5400B
> stepping	: 2
> cpu MHz		: 1000.000
> cache size	: 512 KB
> physical id	: 0
> siblings	: 2
> core id		: 1
> cpu cores	: 2
> apicid		: 1
> initial apicid	: 1
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 1
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36
> clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext
> 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic
> cr8_legacy 3dnowprefetch lbrv
> bogomips	: 2004.11
> TLB size	: 1024 4K pages
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 40 bits physical, 48 bits virtual
> power management: ts fid vid ttp tm stc 100mhzsteps

Can you provide qemu logs ?
what are the dest host cpuinfo ?
what are the hosts memory ?
Comment 4 Dor Laor 2012-05-23 02:40:13 EDT
Also, please report whether a similar scenario but w/o the usage of -kernel/initrd work.

Orit, we have a qemu crash due to alignment issue, it's not related to migration (at least directly). Do you have any ideas?

I moving to 6.4 since it does not look critical.
Comment 5 Orit Wasserman 2012-05-23 03:07:06 EDT
(In reply to comment #4)
> Also, please report whether a similar scenario but w/o the usage of
> -kernel/initrd work.
> 
> Orit, we have a qemu crash due to alignment issue, it's not related to
> migration (at least directly). Do you have any ideas?
> 
> I moving to 6.4 since it does not look critical.

This actually looks like out of memory error (posix_memalign allocates and align the memory ) "Failed to allocate 3246391296 B: Cannot allocate memory".
Comment 6 Joy Pu 2012-05-23 05:55:44 EDT
Hi Orit and Dor,

The migrate is in one host with 4G memory. The guset has 3G memory. And this is all qemu will output when it abort:
do_spice_init: starting 0.10.1
reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
spice_server_add_interface: SPICE_INTERFACE_MIGRATION
spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
spice_server_add_interface: SPICE_INTERFACE_MOUSE
Failed to allocate 3246391296 B: Cannot allocate memory
/bin/sh: line 1: 17381 Aborted                 (core dumped)

I also tried when the guest has 2G and 1G memory. It will not abort.

I try to do migrate without -kernel/initrd. It will abort with 3G memory too while install around 10~15 round. But for a normal guest with 3G memory that is already boot up. It will not abort and can do migrate for 20 round.  So it is not related with the -kernel/initrd options but with the memory size.
Comment 8 Orit Wasserman 2012-06-21 00:43:03 EDT
This behavior is expected as we may exceed the host memory (3G + 3G > 4G) .
We migration a guest on the same host we need at least twice the memory of the guest to make sure you have enough memory.

Note You need to log in before you can comment on or make changes to this bug.