Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 476591[details]
gdb output
Description of problem:
running host (RHEl6.1) with 175 guests (qemu-processes) noticed that several guests died during time, from digging further, it appears that block layer crashes when max user processes limit is reached.
working with libvirt and vdsm above.
some more info below.
#0 0x00000030b1e329a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00000030b1e34185 in abort () at abort.c:92
#2 0x00000000004819dc in die2 (err=<value optimized out>, what=0x62c0de "pthread_create") at posix-aio-compat.c:80
#3 0x0000000000481d6c in thread_create (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:118
#4 spawn_thread (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:379
#5 qemu_paio_submit (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:390
#6 0x0000000000481ecb in paio_submit (bs=<value optimized out>, fd=9, sector_num=<value optimized out>, qiov=0x7f6e2800f070,
nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=0x7f6e2800f010, type=2) at posix-aio-compat.c:584
#7 0x00000000004979d7 in raw_aio_submit (bs=0x135b920, sector_num=65224, qiov=0x7f6e2800f070, nb_sectors=8, cb=0x4906e0 <qcow_aio_write_cb>,
opaque=<value optimized out>, type=2) at block/raw-posix.c:546
#8 0x0000000000497a50 in raw_aio_writev (bs=<value optimized out>, sector_num=<value optimized out>, qiov=<value optimized out>,
nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>) at block/raw-posix.c:562
#9 0x000000000047c641 in bdrv_aio_writev (bs=0x135b920, sector_num=65224, qiov=0x7f6e2800f070, nb_sectors=8, cb=<value optimized out>,
opaque=<value optimized out>) at block.c:1923
#10 0x0000000000490902 in qcow_aio_write_cb (opaque=0x7f6e2800f010, ret=0) at block/qcow2.c:657
#11 0x0000000000490a74 in qcow_aio_writev (bs=<value optimized out>, sector_num=<value optimized out>, qiov=<value optimized out>,
nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>) at block/qcow2.c:691
#12 0x000000000047c641 in bdrv_aio_writev (bs=0x135b010, sector_num=1352008, qiov=0x7f6e280606c0, nb_sectors=8, cb=<value optimized out>,
opaque=<value optimized out>) at block.c:1923
#13 0x000000000047d5dc in bdrv_aio_multiwrite (bs=0x135b010, reqs=0x7f6e2e33b5f0, num_reqs=<value optimized out>) at block.c:2132
#14 0x000000000041dd9e in do_multiwrite (bs=<value optimized out>, blkreq=0x7f6e2e33b5f0, num_writes=3)
at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:236
#15 0x000000000041e448 in virtio_blk_handle_output (vdev=0x13da010, vq=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:363
#16 0x000000000042af59 in kvm_handle_io (env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:538
#17 kvm_run (env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:975
#18 0x000000000042aff9 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1664
#19 0x000000000042bd2f in kvm_main_loop_cpu (_env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1932
#20 ap_main_loop (_env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1982
#21 0x00000030b22077e1 in start_thread (arg=0x7f6e2e33c710) at pthread_create.c:301
#22 0x00000030b1ee153d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
# ps -elf|grep root|wc -l
# 1644
# ulimit -u
# 1024
repro:
1) running about 170 guests (installed with RHEL6 O.S)
(In reply to comment #4)
> So please increase the ulimit higher. How many processes to you have on the
> host?
vdsm will increase ulimit, but still, why does it crash ? i have tried to run 180 guests on single host, in any manner, qemu shouldn't die; please revise
Created attachment 476591 [details] gdb output Description of problem: running host (RHEl6.1) with 175 guests (qemu-processes) noticed that several guests died during time, from digging further, it appears that block layer crashes when max user processes limit is reached. working with libvirt and vdsm above. some more info below. #0 0x00000030b1e329a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00000030b1e34185 in abort () at abort.c:92 #2 0x00000000004819dc in die2 (err=<value optimized out>, what=0x62c0de "pthread_create") at posix-aio-compat.c:80 #3 0x0000000000481d6c in thread_create (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:118 #4 spawn_thread (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:379 #5 qemu_paio_submit (aiocb=0x7f6e2800fa40) at posix-aio-compat.c:390 #6 0x0000000000481ecb in paio_submit (bs=<value optimized out>, fd=9, sector_num=<value optimized out>, qiov=0x7f6e2800f070, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=0x7f6e2800f010, type=2) at posix-aio-compat.c:584 #7 0x00000000004979d7 in raw_aio_submit (bs=0x135b920, sector_num=65224, qiov=0x7f6e2800f070, nb_sectors=8, cb=0x4906e0 <qcow_aio_write_cb>, opaque=<value optimized out>, type=2) at block/raw-posix.c:546 #8 0x0000000000497a50 in raw_aio_writev (bs=<value optimized out>, sector_num=<value optimized out>, qiov=<value optimized out>, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>) at block/raw-posix.c:562 #9 0x000000000047c641 in bdrv_aio_writev (bs=0x135b920, sector_num=65224, qiov=0x7f6e2800f070, nb_sectors=8, cb=<value optimized out>, opaque=<value optimized out>) at block.c:1923 #10 0x0000000000490902 in qcow_aio_write_cb (opaque=0x7f6e2800f010, ret=0) at block/qcow2.c:657 #11 0x0000000000490a74 in qcow_aio_writev (bs=<value optimized out>, sector_num=<value optimized out>, qiov=<value optimized out>, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>) at block/qcow2.c:691 #12 0x000000000047c641 in bdrv_aio_writev (bs=0x135b010, sector_num=1352008, qiov=0x7f6e280606c0, nb_sectors=8, cb=<value optimized out>, opaque=<value optimized out>) at block.c:1923 #13 0x000000000047d5dc in bdrv_aio_multiwrite (bs=0x135b010, reqs=0x7f6e2e33b5f0, num_reqs=<value optimized out>) at block.c:2132 #14 0x000000000041dd9e in do_multiwrite (bs=<value optimized out>, blkreq=0x7f6e2e33b5f0, num_writes=3) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:236 #15 0x000000000041e448 in virtio_blk_handle_output (vdev=0x13da010, vq=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:363 #16 0x000000000042af59 in kvm_handle_io (env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:538 #17 kvm_run (env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:975 #18 0x000000000042aff9 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1664 #19 0x000000000042bd2f in kvm_main_loop_cpu (_env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1932 #20 ap_main_loop (_env=0x138e2f0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1982 #21 0x00000030b22077e1 in start_thread (arg=0x7f6e2e33c710) at pthread_create.c:301 #22 0x00000030b1ee153d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 # ps -elf|grep root|wc -l # 1644 # ulimit -u # 1024 repro: 1) running about 170 guests (installed with RHEL6 O.S)