Hide Forgot
Created attachment 476591 [details] gdb output Description of problem: running host (RHEl6.1) with 175 guests (qemu-processes) noticed that several guests died during time, from digging further, it appears that block layer crashes when max user processes limit is reached. working with libvirt and vdsm above. some more info below. #0 0x00000030b1e329a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00000030b1e34185 in abort () at abort.c:92 #2 0x00000000004819dc in die2 (err=<value optimized out>, what=0x62c0de "pthread_create") at posix-aio-compat.c:80 #3 0x0000000000481d6c in thread_create (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:118 #4 spawn_thread (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:379 #5 qemu_paio_submit (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:390 #6 0x0000000000481ecb in paio_submit (bs=<value optimized out>, fd=9, sector_num=<value optimized out>, qiov=0x7f6e2800f070, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=0x7f6e2800f010, type=2) at posix-aio-compat.c:584 #7 0x00000000004979d7 in raw_aio_submit (bs=0x135b920, sector_num=65224, qiov=0x7f6e2800f070, nb_sectors=8, cb=0x4906e0 <qcow_aio_write_cb>, opaque=<value optimized out>, type=2) at block/raw-posix.c:546 #8 0x0000000000497a50 in raw_aio_writev (bs=<value optimized out>, sector_num=<value optimized out>, qiov=<value optimized out>, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>) at block/raw-posix.c:562 #9 0x000000000047c641 in bdrv_aio_writev (bs=0x135b920, sector_num=65224, qiov=0x7f6e2800f070, nb_sectors=8, cb=<value optimized out>, opaque=<value optimized out>) at block.c:1923 #10 0x0000000000490902 in qcow_aio_write_cb (opaque=0x7f6e2800f010, ret=0) at block/qcow2.c:657 #11 0x0000000000490a74 in qcow_aio_writev (bs=<value optimized out>, sector_num=<value optimized out>, qiov=<value optimized out>, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>) at block/qcow2.c:691 #12 0x000000000047c641 in bdrv_aio_writev (bs=0x135b010, sector_num=1352008, qiov=0x7f6e280606c0, nb_sectors=8, cb=<value optimized out>, opaque=<value optimized out>) at block.c:1923 #13 0x000000000047d5dc in bdrv_aio_multiwrite (bs=0x135b010, reqs=0x7f6e2e33b5f0, num_reqs=<value optimized out>) at block.c:2132 #14 0x000000000041dd9e in do_multiwrite (bs=<value optimized out>, blkreq=0x7f6e2e33b5f0, num_writes=3) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:236 #15 0x000000000041e448 in virtio_blk_handle_output (vdev=0x13da010, vq=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:363 #16 0x000000000042af59 in kvm_handle_io (env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:538 #17 kvm_run (env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:975 #18 0x000000000042aff9 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1664 #19 0x000000000042bd2f in kvm_main_loop_cpu (_env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1932 #20 ap_main_loop (_env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1982 #21 0x00000030b22077e1 in start_thread (arg=0x7f6e2e33c710) at pthread_create.c:301 #22 0x00000030b1ee153d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 # ps -elf|grep root|wc -l # 1644 # ulimit -u # 1024 repro: 1) running about 170 guests (installed with RHEL6 O.S)
So please increase the ulimit higher. How many processes to you have on the host?
(In reply to comment #4) > So please increase the ulimit higher. How many processes to you have on the > host? vdsm will increase ulimit, but still, why does it crash ? i have tried to run 180 guests on single host, in any manner, qemu shouldn't die; please revise