Bug 823813

Summary: Qemu-kvm abort when do ping-pong migration while guest install
Product: Red Hat Enterprise Linux 6 Reporter: Joy Pu <ypu>
Component: qemu-kvmAssignee: Orit Wasserman <owasserm>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 6.3CC: acathrow, areis, bsarathy, dyasny, hhuang, juzhang, knoel, michen, mkenneth, tburke, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-21 04:43:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joy Pu 2012-05-22 08:37:30 UTC
Description of problem:
Do ping-pong migrate during guest install. It will failed to create a guest with a abort message after several round.

do_spice_init: starting 0.10.1
reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
spice_server_add_interface: SPICE_INTERFACE_MIGRATION
spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
spice_server_add_interface: SPICE_INTERFACE_MOUSE
Failed to allocate 3246391296 B: Cannot allocate memory
/bin/sh: line 1: 17381 Aborted                 (core dumped)

And the call trace from the core file is just like this:
(gdb) bt
#0  0x00007fd9e50838a5 in raise () from /lib64/libc.so.6
#1  0x00007fd9e5085085 in abort () from /lib64/libc.so.6
#2  0x00007fd9e77310d8 in qemu_memalign (alignment=2097152, size=3246391296) at osdep.c:112
#3  0x00007fd9e77a5a17 in qemu_ram_alloc_from_ptr (dev=<value optimized out>, name=<value optimized out>, 
    size=3246391296, host=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/exec.c:2756
#4  0x00007fd9e785aa03 in pc_init1 (ram_size=3246391296, boot_device=0x7fff421d4df0 "d", 
    kernel_filename=0x7fff421d6ebb "/root/autotest-devel/client/tests/kvm/images/rhel62-64/vmlinuz", 
    kernel_cmdline=0x7fff421d6f02 "ks=cdrom nicdelay=60 console=ttyS0,115200 console=tty0", 
    initrd_filename=0x7fff421d6f41 "/root/autotest-devel/client/tests/kvm/images/rhel62-64/initrd.img", 
    cpu_model=0x7fff421d6d92 "Opteron_G2", pci_enabled=1) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/pc.c:1122
#5  0x00007fd9e76f83c8 in main (argc=20, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6220

Version-Release number of selected component (if applicable):
host kernel: 2.6.32-269.el6.x86_64 
rpm -qa |grep qemu
qemu-kvm-tools-0.12.1.2-2.290.el6.x86_64
qemu-kvm-0.12.1.2-2.290.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.290.el6.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
qemu-img-0.12.1.2-2.290.el6.x86_64


How reproducible:
once

Steps to Reproduce:
1. Install a guest with cdrom and -kernel -initrd and -append options
2. Do ping-pong migrate during install
  
Actual results:
guest create failed with abort message

Expected results:
guest install succuss during migrate.

Additional info:
processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 107
model name	: AMD Athlon(tm) Dual Core Processor 5400B
stepping	: 2
cpu MHz		: 1000.000
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch lbrv
bogomips	: 2004.11
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps

Comment 3 Orit Wasserman 2012-05-23 05:02:50 UTC
(In reply to comment #0)
> Description of problem:
> Do ping-pong migrate during guest install. It will failed to create a guest
> with a abort message after several round.
> 
> do_spice_init: starting 0.10.1
> reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
> reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
> reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
> spice_server_add_interface: SPICE_INTERFACE_MIGRATION
> spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
> spice_server_add_interface: SPICE_INTERFACE_MOUSE
> Failed to allocate 3246391296 B: Cannot allocate memory
> /bin/sh: line 1: 17381 Aborted                 (core dumped)
> 
> And the call trace from the core file is just like this:
> (gdb) bt
> #0  0x00007fd9e50838a5 in raise () from /lib64/libc.so.6
> #1  0x00007fd9e5085085 in abort () from /lib64/libc.so.6
> #2  0x00007fd9e77310d8 in qemu_memalign (alignment=2097152, size=3246391296)
> at osdep.c:112
> #3  0x00007fd9e77a5a17 in qemu_ram_alloc_from_ptr (dev=<value optimized
> out>, name=<value optimized out>, 
>     size=3246391296, host=<value optimized out>) at
> /usr/src/debug/qemu-kvm-0.12.1.2/exec.c:2756
> #4  0x00007fd9e785aa03 in pc_init1 (ram_size=3246391296,
> boot_device=0x7fff421d4df0 "d", 
>     kernel_filename=0x7fff421d6ebb
> "/root/autotest-devel/client/tests/kvm/images/rhel62-64/vmlinuz", 
>     kernel_cmdline=0x7fff421d6f02 "ks=cdrom nicdelay=60 console=ttyS0,115200
> console=tty0", 
>     initrd_filename=0x7fff421d6f41
> "/root/autotest-devel/client/tests/kvm/images/rhel62-64/initrd.img", 
>     cpu_model=0x7fff421d6d92 "Opteron_G2", pci_enabled=1) at
> /usr/src/debug/qemu-kvm-0.12.1.2/hw/pc.c:1122
> #5  0x00007fd9e76f83c8 in main (argc=20, argv=<value optimized out>,
> envp=<value optimized out>)
>     at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6220
> 
> Version-Release number of selected component (if applicable):
> host kernel: 2.6.32-269.el6.x86_64 
> rpm -qa |grep qemu
> qemu-kvm-tools-0.12.1.2-2.290.el6.x86_64
> qemu-kvm-0.12.1.2-2.290.el6.x86_64
> qemu-kvm-debuginfo-0.12.1.2-2.290.el6.x86_64
> gpxe-roms-qemu-0.9.7-6.9.el6.noarch
> qemu-img-0.12.1.2-2.290.el6.x86_64
> 
> 
> How reproducible:
> once
> 
> Steps to Reproduce:
> 1. Install a guest with cdrom and -kernel -initrd and -append options
> 2. Do ping-pong migrate during install
>   
> Actual results:
> guest create failed with abort message
> 
> Expected results:
> guest install succuss during migrate.
> 
> Additional info:
> processor	: 1
> vendor_id	: AuthenticAMD
> cpu family	: 15
> model		: 107
> model name	: AMD Athlon(tm) Dual Core Processor 5400B
> stepping	: 2
> cpu MHz		: 1000.000
> cache size	: 512 KB
> physical id	: 0
> siblings	: 2
> core id		: 1
> cpu cores	: 2
> apicid		: 1
> initial apicid	: 1
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 1
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36
> clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext
> 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic
> cr8_legacy 3dnowprefetch lbrv
> bogomips	: 2004.11
> TLB size	: 1024 4K pages
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 40 bits physical, 48 bits virtual
> power management: ts fid vid ttp tm stc 100mhzsteps

Can you provide qemu logs ?
what are the dest host cpuinfo ?
what are the hosts memory ?

Comment 4 Dor Laor 2012-05-23 06:40:13 UTC
Also, please report whether a similar scenario but w/o the usage of -kernel/initrd work.

Orit, we have a qemu crash due to alignment issue, it's not related to migration (at least directly). Do you have any ideas?

I moving to 6.4 since it does not look critical.

Comment 5 Orit Wasserman 2012-05-23 07:07:06 UTC
(In reply to comment #4)
> Also, please report whether a similar scenario but w/o the usage of
> -kernel/initrd work.
> 
> Orit, we have a qemu crash due to alignment issue, it's not related to
> migration (at least directly). Do you have any ideas?
> 
> I moving to 6.4 since it does not look critical.

This actually looks like out of memory error (posix_memalign allocates and align the memory ) "Failed to allocate 3246391296 B: Cannot allocate memory".

Comment 6 Joy Pu 2012-05-23 09:55:44 UTC
Hi Orit and Dor,

The migrate is in one host with 4G memory. The guset has 3G memory. And this is all qemu will output when it abort:
do_spice_init: starting 0.10.1
reds_init_ssl: Loaded certificates from /tmp/spice_x509d/server-cert.pem
reds_init_ssl: Using private key from /tmp/spice_x509d/server-key.pem
reds_init_ssl: Loaded CA certificates from /tmp/spice_x509d/ca-cert.pem
spice_server_add_interface: SPICE_INTERFACE_MIGRATION
spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
spice_server_add_interface: SPICE_INTERFACE_MOUSE
Failed to allocate 3246391296 B: Cannot allocate memory
/bin/sh: line 1: 17381 Aborted                 (core dumped)

I also tried when the guest has 2G and 1G memory. It will not abort.

I try to do migrate without -kernel/initrd. It will abort with 3G memory too while install around 10~15 round. But for a normal guest with 3G memory that is already boot up. It will not abort and can do migrate for 20 round.  So it is not related with the -kernel/initrd options but with the memory size.

Comment 8 Orit Wasserman 2012-06-21 04:43:03 UTC
This behavior is expected as we may exceed the host memory (3G + 3G > 4G) .
We migration a guest on the same host we need at least twice the memory of the guest to make sure you have enough memory.