Bug 716792 - qemu-kvm core dumped when installing guest with big image (2300G).
Summary: qemu-kvm core dumped when installing guest with big image (2300G).
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-06-27 05:37 UTC by YangFeng
Modified: 2014-06-18 03:12 UTC (History)
10 users (show)

Fixed In Version: qemu-1.5
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-13 12:56:37 UTC
Target Upstream Version:


Attachments (Terms of Use)
ks.cfg in ks.iso (923 bytes, application/octet-stream)
2011-06-27 05:37 UTC, YangFeng
no flags Details

Description YangFeng 2011-06-27 05:37:06 UTC
Created attachment 510021 [details]
ks.cfg in ks.iso

Description of problem:
When try to installing rhel6.1 guest on a 2300G image, qemu-kvm core dumped.

Version-Release number of selected component (if applicable):
host kernel: kernel-2.6.32-161.el6.x86_64
qemu-kvm: qemu-kvm-0.12.1.2-2.165.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. Make a 2300G qcow2 image, cluster_size=512.
  # qemu-img create -f qcow2 /usr/code/autotest-patch/client/tests/kvm/images/RHEL-Server-6.1-64-virtio.qcow2 -o cluster_size=512 2300G
2. Create ks.iso which contains ks.cfg file.
3. Install rhel6.1 guest with following command.
   # qemu-kvm -name 'vm1' -chardev socket,id=qmp_monitor_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20110623-143356-eaBW,server,nowait -mon chardev=qmp_monitor_id_qmpmonitor1,mode=control -chardev socket,id=serial_id_20110623-143356-eaBW,path=/tmp/serial-20110623-143356-eaBW,server,nowait -device isa-serial,chardev=serial_id_20110623-143356-eaBW -drive file='/usr/code/autotest-patch/client/tests/kvm/images/RHEL-Server-6.1-64-virtio.qcow2',index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -device virtio-net-pci,netdev=idxohpsa,mac=9a:7a:b9:37:ae:25,id=ndev00idxohpsa,bus=pci.0,addr=0x3 -netdev tap,id=idxohpsa,ifname='t0-143356-eaBW',script='/usr/code/autotest-patch/client/tests/kvm/scripts/qemu-ifup-switch',downscript='no' -m 2048 -smp 2,cores=1,threads=1,sockets=2 -drive file='/usr/code/autotest-patch/client/tests/kvm/isos/linux/RHEL6.1-Server-x86_64.iso',index=1,if=none,id=drive-ide0-0-0,media=cdrom,readonly=on,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file='/usr/code/autotest-patch/client/tests/kvm/isos/../images/rhel61-64/ks.iso',index=2,if=none,id=drive-ide0-0-1,media=cdrom,readonly=on,format=raw -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -cpu cpu64-rhel6,+sse2,+x2apic -kernel '/usr/code/autotest-patch/client/tests/kvm/images/rhel61-64/vmlinuz' -initrd '/usr/code/autotest-patch/client/tests/kvm/images/rhel61-64/initrd.img' -vnc :0 -rtc base=utc,clock=host,driftfix=none -M rhel6.1.0 -boot order=cdn,once=d,menu=off   -usbdevice tablet -no-kvm-pit-reinjection --append 'ks=cdrom nicdelay=60 console=ttyS0,115200 console=tty0' -enable-kvm

Actual results:
Install need long time to create ext4 in guest image.  About 2 hours later, qemu-kvm core dumped.

Expected results:
Install could finish successfully.

Additional info:
(gdb) bt
#0  0x0000003ba7c32a45 in raise () from /lib64/libc.so.6
#1  0x0000003ba7c34225 in abort () from /lib64/libc.so.6
#2  0x0000003ba7c6fdfb in __libc_message () from /lib64/libc.so.6
#3  0x0000003ba7c75716 in malloc_printerr () from /lib64/libc.so.6
#4  0x000000000049550b in alloc_refcount_block (bs=0x243d010, offset=3120536064, length=<value optimized out>, addend=1) at block/qcow2-refcount.c:352
#5  update_refcount (bs=0x243d010, offset=3120536064, length=<value optimized out>, addend=1) at block/qcow2-refcount.c:459
#6  0x0000000000495df0 in qcow2_alloc_clusters (bs=0x243d010, size=32768) at block/qcow2-refcount.c:576
#7  0x0000000000496b0d in qcow2_alloc_cluster_offset (bs=0x243d010, offset=146579226624, n_start=0, n_end=19008, num=0x26fa86c, m=0x26fa8b0) at block/qcow2-cluster.c:806
#8  0x0000000000492648 in qcow2_aio_write_cb (opaque=0x26fa830, ret=<value optimized out>) at block/qcow2.c:666
#9  0x000000000048443a in qemu_laio_process_completion (s=<value optimized out>, laiocb=0x26fac70) at linux-aio.c:68
#10 0x000000000048464f in qemu_laio_enqueue_completed (opaque=0x243ac50) at linux-aio.c:107
#11 qemu_laio_completion_cb (opaque=0x243ac50) at linux-aio.c:144
#12 0x000000000040b9df in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4430
#13 0x000000000042b5ca in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2164
#14 0x000000000040ef05 in main_loop (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4640
#15 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6845


qemu-kvm print following on terminal when core dump:
*** glibc detected *** qemu-kvm: double free or corruption (!prev): 0x00000000026e14a0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3ba7c75716]
qemu-kvm[0x49550b]
qemu-kvm[0x495df0]
qemu-kvm[0x496b0d]
qemu-kvm[0x492648]
qemu-kvm[0x48443a]
qemu-kvm[0x48464f]
qemu-kvm[0x40b9df]
qemu-kvm[0x42b5ca]
qemu-kvm[0x40ef05]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3ba7c1ec9d]
qemu-kvm[0x4080a9]
======= Memory map: ========
00400000-006c1000 r-xp 00000000 fd:00 2498340                            /usr/libexec/qemu-kvm
008c0000-008e4000 rw-p 002c0000 fd:00 2498340                            /usr/libexec/qemu-kvm
008e4000-00ccc000 rw-p 00000000 00:00 0 
00ee3000-00ee9000 rw-p 002e3000 fd:00 2498340                            /usr/libexec/qemu-kvm
0241f000-02e99000 rw-p 00000000 00:00 0 
02e99000-02ea9000 rw-p 00000000 00:00 0 
02ea9000-02ebc000 rw-p 00000000 00:00 0 
02ebc000-02ecc000 rw-p 00000000 00:00 0 
02ecc000-03710000 rw-p 00000000 00:00 0 
3592400000-3592416000 r-xp 00000000 fd:00 2101038                        /lib64/libgcc_s-4.4.5-20110214.so.1
3592416000-3592615000 ---p 00016000 fd:00 2101038                        /lib64/libgcc_s-4.4.5-20110214.so.1
3592615000-3592616000 rw-p 00015000 fd:00 2101038                        /lib64/libgcc_s-4.4.5-20110214.so.1
36e6200000-36e6201000 r-xp 00000000 fd:00 2098932                        /lib64/libaio.so.1.0.1
36e6201000-36e6400000 ---p 00001000 fd:00 2098932                        /lib64/libaio.so.1.0.1
36e6400000-36e6401000 rw-p 00000000 fd:00 2098932                        /lib64/libaio.so.1.0.1
3ba7800000-3ba7820000 r-xp 00000000 fd:00 2100933                        /lib64/ld-2.12.so
3ba7a1f000-3ba7a20000 r--p 0001f000 fd:00 2100933                        /lib64/ld-2.12.so
3ba7a20000-3ba7a21000 rw-p 00020000 fd:00 2100933                        /lib64/ld-2.12.so
3ba7a21000-3ba7a22000 rw-p 00000000 00:00 0 
3ba7c00000-3ba7d87000 r-xp 00000000 fd:00 2100934                        /lib64/libc-2.12.so
3ba7d87000-3ba7f87000 ---p 00187000 fd:00 2100934                        /lib64/libc-2.12.so
3ba7f87000-3ba7f8b000 r--p 00187000 fd:00 2100934                        /lib64/libc-2.12.so
3ba7f8b000-3ba7f8c000 rw-p 0018b000 fd:00 2100934                        /lib64/libc-2.12.so
3ba7f8c000-3ba7f91000 rw-p 00000000 00:00 0 
3ba8000000-3ba8002000 r-xp 00000000 fd:00 2100938                        /lib64/libdl-2.12.so
3ba8002000-3ba8202000 ---p 00002000 fd:00 2100938                        /lib64/libdl-2.12.so
3ba8202000-3ba8203000 r--p 00002000 fd:00 2100938                        /lib64/libdl-2.12.so
3ba8203000-3ba8204000 rw-p 00003000 fd:00 2100938                        /lib64/libdl-2.12.so
3ba8400000-3ba8417000 r-xp 00000000 fd:00 2100936                        /lib64/libpthread-2.12.so
3ba8417000-3ba8617000 ---p 00017000 fd:00 2100936                        /lib64/libpthread-2.12.so
3ba8617000-3ba8618000 r--p 00017000 fd:00 2100936                        /lib64/libpthread-2.12.so
3ba8618000-3ba8619000 rw-p 00018000 fd:00 2100936                        /lib64/libpthread-2.12.so
3ba8619000-3ba861d000 rw-p 00000000 00:00 0 
3ba8800000-3ba8807000 r-xp 00000000 fd:00 2100945                        /lib64/librt-2.12.so
3ba8807000-3ba8a06000 ---p 00007000 fd:00 2100945                        /lib64/librt-2.12.so
3ba8a06000-3ba8a07000 r--p 00006000 fd:00 2100945                        /lib64/librt-2.12.so
3ba8a07000-3ba8a08000 rw-p 00007000 fd:00 2100945                        /lib64/librt-2.12.so
3ba8c00000-3ba8c83000 r-xp 00000000 fd:00 2100935                        /lib64/libm-2.12.so
3ba8c83000-3ba8e82000 ---p 00083000 fd:00 2100935                        /lib64/libm-2.12.so
3ba8e82000-3ba8e83000 r--p 00082000 fd:00 2100935                        /lib64/libm-2.12.so
3ba8e83000-3ba8e84000 rw-p 00083000 fd:00 2100935                        /lib64/libm-2.12.so
3ba9000000-3ba9003000 r-xp 00000000 fd:00 2525319                        /usr/lib64/libpulse-simple.so.0.0.3
3ba9003000-3ba9203000 ---p 00003000 fd:00 2525319                        /usr/lib64/libpulse-simple.so.0.0.3
3ba9203000-3ba9204000 rw-p 00003000 fd:00 2525319                        /usr/lib64/libpulse-simple.so.0.0.3
3ba9400000-3ba941d000 r-xp 00000000 fd:00 2100955                        /lib64/libselinux.so.1
3ba941d000-3ba961c000 ---p 0001d000 fd:00 2100955                        /lib64/libselinux.so.1
3ba961c000-3ba961d000 r--p 0001c000 fd:00 2100955                        /lib64/libselinux.so.1
3ba961d000-3ba961e000 rw-p 0001d000 fd:00 2100955                        /lib64/libselinux.so.1
3ba961e000-3ba961f000 rw-p 00000000 00:00 0 
3ba9800000-3ba9815000 r-xp 00000000 fd:00 2100940                        /lib64/libz.so.1.2.3
3ba9815000-3ba9a14000 ---p 00015000 fd:00 2100940                        /lib64/libz.so.1.2.3
3ba9a14000-3ba9a15000 rw-p 00014000 fd:00 2100940                        /lib64/libz.so.1.2.3
3ba9c00000-3ba9cdc000 r-xp 00000000 fd:00 2527222                        /usr/lib64/libspice-server.so.1.0.2
3ba9cdc000-3ba9edc000 ---p 000dc000 fd:00 2527222                        /usr/lib64/libspice-server.so.1.0.2
3ba9edc000-3ba9ede000 rw-p 000dc000 fd:00 2527222                        /usr/lib64/libspice-server.so.1.0.2
3ba9ede000-3ba9ee3000 rw-p 00000000 00:00 0 
3baa000000-3baa016000 r-xp 00000000 fd:00 2097538                        /lib64/libresolv-2.12.so
3baa016000-3baa216000 ---p 00016000 fd:00 2097538                        /lib64/libresolv-2.12.so
3baa216000-3baa217000 r--p 00016000 fd:00 2097538                        /lib64/libresolv-2.12.so
3baa217000-3baa218000 rw-p 00017000 fd:00 2097538                        /lib64/libresolv-2.12.so
3baa218000-3baa21a000 rw-p 00000000 00:00 0 
3baa400000-3baa440000 r-xp 00000000 fd:00 2100947                        /lib64/libdbus-1.so.3.4.0
3baa440000-3baa63f000 ---p 00040000 fd:00 2100947                        /lib64/libdbus-1.so.3.4.0
3baa63f000-3baa640000 r--p 0003f000 fd:00 2100947                        /lib64/libdbus-1.so.3.4.0
3baa640000-3baa641000 rw-p 00040000 fd:00 2100947                        /lib64/libdbus-1.so.3.4.0
3baa800000-3baa80f000 r-xp 00000000 fd:00 2495457                        /usr/lib64/libcelt051.so.0.0.0
3baa80f000-3baaa0e000 ---p 0000f000 fd:00 2495457                        /usr/lib64/libcelt051.so.0.0.0
3baaa0e000-3baaa0f000 rw-p 0000e000 fd:00 2495457                        /usr/lib64/libcelt051.so.0.0.0
3bab400000-3bab539000 r-xp 00000000 fd:00 2510624                        /usr/lib64/libX11.so.6.3.0
3bab539000-3bab739000 ---p 00139000 fd:00 2510624                        /usr/lib64/libX11.so.6.3.0
3bab739000-3bab73f000 rw-p 00139000 fd:00 2510624                        /usr/lib64/libX11.so.6.3.0Aborted (core dumped)

Comment 7 RHEL Program Management 2012-07-10 07:23:58 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 8 RHEL Program Management 2012-07-11 02:00:00 UTC
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.

Comment 13 juzhang 2013-05-20 07:39:22 UTC
Hi Sluo,

Please re-test this issue if the internal build which is based qemu1.5 comes out.

Comment 17 mazhang 2014-01-16 02:33:45 UTC
Test this problem on RHEL-7.0-20130306.0.

Host:
qemu-kvm-1.3.0-6.el7.x86_64
kernel-3.7.0-0.36.el7.x86_64


Steps:
1. Create a 2300G image with 512 byte clusters.
# qemu-img create -f qcow2 rhel6u1-64.qcow2 -o cluster_size=512 2300G
Formatting 'rhel6u1-64.qcow2', fmt=qcow2 size=2469606195200 encryption=off cluster_size=512 lazy_refcounts=off 

2. Install rhel6.1 guest.
gdb --args /usr/libexec/qemu-kvm \
-name 'vm1' \
-nodefaults \
-chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20130218-133213-tne4yYwu,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-drive file=/home/rhel6u1-64.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:39:13:2c \
-m 4096 \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-cpu 'SandyBridge',hv_relaxed \
-M pc \
-rtc base=localtime,clock=host,driftfix=slew \
-boot menu=on \
-enable-kvm \
-monitor stdio \
-vga qxl \
-spice port=5900,disable-ticketing \
-drive file=/home/boot.iso,if=none,id=drive-ide0-0-1,media=cdrom,format=raw \
-device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,bootindex=0 \

Result:
After about 18 hours guest can't finish make ext4 filesystem, but still alive, and no core dumped.

Comment 18 mazhang 2014-01-17 05:47:49 UTC
qemu-kvm-1.3.0-3.el7.x86_64 can't reproduce this bug, about 20 hours guest can't finish make ext4 filesystem.

Comment 20 mazhang 2014-01-23 02:14:42 UTC
Installed the latest package and test this bug.

Host:
kernel-3.10.0-78.el7.x86_64
qemu-kvm-tools-1.5.3-39.el7.x86_64
ipxe-roms-qemu-20130517-1.gitc4bce43.el7.noarch
qemu-kvm-common-1.5.3-39.el7.x86_64
qemu-kvm-1.5.3-39.el7.x86_64
qemu-kvm-debuginfo-1.5.3-39.el7.x86_64
qemu-img-1.5.3-39.el7.x86_64

Guest:
RHEL7-64

Cli:
#qemu-img create -f qcow2 rhel7-64.qcow2 -o cluster_size=512 2300G

#gdb --args /usr/libexec/qemu-kvm \
-name 'vm1' \
-nodefaults \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:39:13:2d \
-m 4096 \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-cpu Opteron_G1 \
-M pc \
-rtc base=localtime,clock=host,driftfix=slew \
-boot menu=on \
-enable-kvm \
-monitor stdio \
-vga qxl \
-spice port=5900,disable-ticketing \
-qmp tcp:0:6666,server,nowait \
-drive file=/home/rhel7-64.qcow2,index=0,if=none,id=drive-virtio-disk0,media=disk,cache=none,format=qcow2,aio=native \
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 \
-drive file=/home/boot.iso,if=none,id=drive-ide0-0-1,media=cdrom,format=raw \
-device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,bootindex=0 \

Result:
Qemu-kvm works well, no crash, installation takes very long time.

Comment 21 juzhang 2014-01-24 04:33:39 UTC
(In reply to mazhang from comment #20)
> Installed the latest package and test this bug.
> 
> Host:
> kernel-3.10.0-78.el7.x86_64
> qemu-kvm-tools-1.5.3-39.el7.x86_64
> ipxe-roms-qemu-20130517-1.gitc4bce43.el7.noarch
> qemu-kvm-common-1.5.3-39.el7.x86_64
> qemu-kvm-1.5.3-39.el7.x86_64
> qemu-kvm-debuginfo-1.5.3-39.el7.x86_64
> qemu-img-1.5.3-39.el7.x86_64
> 
> Guest:
> RHEL7-64
> 
> Cli:
> #qemu-img create -f qcow2 rhel7-64.qcow2 -o cluster_size=512 2300G
> 
> #gdb --args /usr/libexec/qemu-kvm \
> -name 'vm1' \
> -nodefaults \
> -netdev tap,id=hostnet0 \
> -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:39:13:2d \
> -m 4096 \
> -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
> -cpu Opteron_G1 \
> -M pc \
> -rtc base=localtime,clock=host,driftfix=slew \
> -boot menu=on \
> -enable-kvm \
> -monitor stdio \
> -vga qxl \
> -spice port=5900,disable-ticketing \
> -qmp tcp:0:6666,server,nowait \
> -drive
> file=/home/rhel7-64.qcow2,index=0,if=none,id=drive-virtio-disk0,media=disk,
> cache=none,format=qcow2,aio=native \
> -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 \
> -drive file=/home/boot.iso,if=none,id=drive-ide0-0-1,media=cdrom,format=raw \
> -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,bootindex=0 \
> 
> Result:
> Qemu-kvm works well, no crash, installation takes very long time.
Long time is expected since cluster_size is 512

Comment 23 Ludek Smid 2014-06-13 12:56:37 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.


Note You need to log in before you can comment on or make changes to this bug.