Hide Forgot
Description of problem: qemu-kvm core dump when start from libvirt. qemu log says "pthread_create failed: Resource temporarily unavailable". Version-Release number of selected component (if applicable): libvirt-0.10.2-23.el6.x86_64 qemu-kvm-0.12.1.2-2.398.el6.x86_64 How reproducible: always Steps to Reproduce: 1. # virsh dumpxml r6 <domain type='kvm'> <name>r6</name> <uuid>fbd9d4c5-ae9c-2e3c-0265-894fbe093d7a</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='rhel6.5.0'>hvm</type> <loader>/usr/share/seabios/bios.bin</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/var/lib/libvirt/images/kvm-rhel6.4.z-x86_64-raw.img'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0' model='piix3-uhci'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/r6.agent'/> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'/> </domain> 2. # virsh start r6 Domain r6 started The guest will shutdown after grub stage, that caused by qemu-kvm coredump. Actual results: coredump Expected results: no error Additional info: I also tried old version qemu-kvm-0.12.1.2-2.377.el6.x86_64, still coredump.
Created attachment 789494 [details] /var/log/libvirt/qemu/r6.log
the coredump file is too large, and the backtrace is: (gdb) t a a bt Thread 20 (Thread 15216): #0 0x00007f3001f6cec3 in ?? () #1 0x0000000000000000 in ?? () Thread 19 (Thread 15215): #0 0x00007f3001f6cec3 in ?? () #1 0x0000000000000000 in ?? () Thread 18 (Thread 15214): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000000 in ?? () Thread 17 (Thread 15213): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000000 in ?? () Thread 16 (Thread 15202): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000000 in ?? () Thread 15 (Thread 15212): #0 0x00007f3001f6cec3 in ?? () #1 0x0000000000000000 in ?? () Thread 14 (Thread 15211): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000000 in ?? () Thread 13 (Thread 15210): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000000 in ?? () Thread 12 (Thread 15209): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000000 in ?? () Thread 11 (Thread 15208): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000000 in ?? () Thread 10 (Thread 15207): #0 0x00007f3001f6cec3 in ?? () #1 0x0000000000000000 in ?? () Thread 9 (Thread 15206): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000000 in ?? () Thread 8 (Thread 15205): #0 0x00007f3001f6cec3 in ?? () #1 0x0000000000000000 in ?? () Thread 7 (Thread 15204): #0 0x00007f3001f6cec3 in ?? () #1 0x0000000000000000 in ?? () Thread 6 (Thread 15201): #0 0x00007f3001f6cec3 in ?? () #1 0x0000000000000000 in ?? () Thread 5 (Thread 15200): #0 0x00007f2fff5d2084 in ?? () #1 0x0000000000000200 in ?? () #2 0x0000000000002000 in ?? () #3 0x0000000000000000 in ?? () Thread 4 (Thread 15203): #0 0x00007f3001f6cec3 in ?? () #1 0x0000000000000000 in ?? () Thread 3 (Thread 15198): #0 0x00007f2fff5d1a47 in ?? () #1 0x00007f30026517fa in kvm_run (env=0x7f30049c2720) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1015 #2 0x00007f3002651cb9 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1744 #3 0x00007f3002652b9d in kvm_main_loop_cpu (_env=0x7f30049c2720) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2005 #4 ap_main_loop (_env=0x7f30049c2720) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2061 #5 0x00007f3001f65851 in ?? () #6 0x00007f2ff9bb1700 in ?? () #7 0x0000000000000000 in ?? () Thread 2 (Thread 15199): #0 0x00007f2fff5d0253 in ?? () #1 0x00007f2faaa5fbc0 in ?? () #2 0x00007f300000000a in ?? () #3 0x0000000000000014 in ?? () #4 0x00007f2fa40008f8 in ?? () #5 0xffffbe989e3eadf7 in ?? () #6 0x00007f2fffd5a9f4 in ?? () #7 0x0000000000000000 in ?? () Thread 1 (Thread 15196): #0 0x00007f2fff5238a5 in ?? () #1 0x00007f2fff525085 in ?? () #2 0x00007f3006749f28 in ?? () #3 0x00007f2fff87e860 in ?? () #4 0x00007f300674a080 in ?? () #5 0x0000000000001e60 in ?? () #6 0x0000000000000002 in ?? () #7 0x00007f300674a200 in ?? () #8 0x0000000000000001 in ?? () #9 0x00007f2fff5f0dcb in ?? () #10 0x0000003000000028 in ?? () #11 0x00007f3006749ff0 in ?? () #12 0x00007f3006749f20 in ?? () #13 0x00007f2fff648be0 in ?? () #14 0x0000000000000003 in ?? () #15 0x00007f30023948d9 in _dl_allocate_tls () from /lib64/ld-linux-x86-64.so.2 #16 0x0000000000000000 in ?? ()
Can you attach "ps -eLf" output?
Hello Paolo, there is the output of "ps -eLf". I remember I touch the hugepage, "sysctl vm.nr_hugepages=100". is that the reason?
Created attachment 790212 [details] ps output
I capture this abort from gdb: (gdb) c Continuing. [New Thread 0x7fbc74caa700 (LWP 20348)] [New Thread 0x7fbc279fc700 (LWP 20349)] [New Thread 0x7fbc26ffb700 (LWP 20350)] [New Thread 0x7fbc265fa700 (LWP 20351)] [New Thread 0x7fbc25bf9700 (LWP 20352)] [New Thread 0x7fbc251f8700 (LWP 20353)] [New Thread 0x7fbc1ffff700 (LWP 20354)] [New Thread 0x7fbc1f5fe700 (LWP 20355)] [New Thread 0x7fbc1ebfd700 (LWP 20356)] [New Thread 0x7fbc1e1fc700 (LWP 20357)] [New Thread 0x7fbc1d7fb700 (LWP 20358)] [New Thread 0x7fbc1cdfa700 (LWP 20359)] [New Thread 0x7fbc17fff700 (LWP 20360)] [New Thread 0x7fbc175fe700 (LWP 20361)] [New Thread 0x7fbc16bfd700 (LWP 20362)] [New Thread 0x7fbc161fc700 (LWP 20363)] [New Thread 0x7fbc157fb700 (LWP 20364)] Program received signal SIGABRT, Aborted. 0x00007fbc7dbe98a5 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007fbc7dbe98a5 in raise () from /lib64/libc.so.6 #1 0x00007fbc7dbeb085 in abort () from /lib64/libc.so.6 #2 0x00007fbc80d39371 in die2 (err=<value optimized out>, what=0x7fbc80eb39e4 "pthread_create") at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:79 #3 0x00007fbc80d3991e in thread_create (aiocb=0x7fbc824227b0) at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:117 #4 spawn_thread (aiocb=0x7fbc824227b0) at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:397 #5 qemu_paio_submit (aiocb=0x7fbc824227b0) at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:408 #6 0x00007fbc80d39a9b in paio_submit (bs=<value optimized out>, fd=10, sector_num=4196904, qiov=0x7fbc887ddc80, nb_sectors=8, cb=<value optimized out>, opaque=0x7fbc872f9110, type=2) at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:595 #7 0x00007fbc80d57153 in raw_aio_submit (bs=0x7fbc8216e9f0, sector_num=4196904, qiov=0x7fbc887ddc80, nb_sectors=8, cb=0x7fbc80d2cce0 <bdrv_co_io_em_complete>, opaque=<value optimized out>, type=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block/raw-posix.c:448 #8 0x00007fbc80d57200 in raw_aio_writev (bs=<value optimized out>, sector_num=<value optimized out>, qiov=<value optimized out>, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block/raw-posix.c:464 #9 0x00007fbc80d2dd43 in bdrv_co_io_em (bs=0x7fbc8216e9f0, sector_num=4196904, nb_sectors=8, iov=0x7fbc887ddc80, is_write=true) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4109 #10 0x00007fbc80d2f206 in bdrv_co_do_writev (bs=0x7fbc8216e9f0, sector_num=4196904, nb_sectors=8, qiov=0x7fbc887ddc80, flags=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2298 #11 0x00007fbc80d2f206 in bdrv_co_do_writev (bs=0x7fbc8216e010, sector_num=4196904, nb_sectors=8, qiov=0x7fbc887ddc80, flags=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2298 #12 0x00007fbc80d2f2c1 in bdrv_co_do_rw (opaque=0x7fbc8245e5d0) ---Type <return> to continue, or q <return> to quit--- at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3962 #13 0x00007fbc80d3889b in coroutine_trampoline (i0=<value optimized out>, i1=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/coroutine-ucontext.c:129 #14 0x00007fbc7dbfab70 in ?? () from /lib64/libc.so.6 #15 0x00007fff173c9310 in ?? () #16 0x0000000000000000 in ?? ()
hmm, I think I found the problem: When I check /proc/20307/task/, I found there are 20 threads. So I remember that I have configured "max_processes = 20" in /etc/libvirt/qemu.conf. After comment this configuration and restart libvirt, qemu-kvm can start. I think I should close this BZ as NOTABUG.
I'm not sure what max_processes is for... Jiri, can you check this bug? You added it in commit 87e78b2 (qemu: Support for overriding NPROC limit, 2011-04-05).
# If max_processes is set to a positive integer, libvirt will use # it to set the maximum number of processes that can be run by qemu # user. This can be used to override default value set by host OS. # The same applies to max_files which sets the limit on the maximum # number of opened files. The reason for adding it was that for big machines with lots of running qemu domains (possibly also on a lots of CPUs) the default system limit on the number of processes that any user can run was too low. One can use max_processes to increase this limit (which of course must be done prior to starting any domains). Setting max_processes to 20 is insane (except for testing that max_processes option actually influences the system limit).