| Summary: | qemu-kvm pthread_create failed: Resource temporarily unavailable | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Jincheng Miao <jmiao> | ||||||
| Component: | qemu-kvm | Assignee: | Virtualization Maintenance <virt-maint> | ||||||
| Status: | CLOSED NOTABUG | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 6.5 | CC: | acathrow, bsarathy, dyuan, gsun, jdenemar, jiahu, juzhang, mkenneth, mzhan, pbonzini, qzhang, virt-maint | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2013-08-26 04:05:28 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Attachments: |
|
||||||||
Created attachment 789494 [details]
/var/log/libvirt/qemu/r6.log
the coredump file is too large, and the backtrace is:
(gdb) t a a bt
Thread 20 (Thread 15216):
#0 0x00007f3001f6cec3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (Thread 15215):
#0 0x00007f3001f6cec3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (Thread 15214):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (Thread 15213):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (Thread 15202):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 15 (Thread 15212):
#0 0x00007f3001f6cec3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (Thread 15211):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (Thread 15210):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (Thread 15209):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (Thread 15208):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (Thread 15207):
#0 0x00007f3001f6cec3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (Thread 15206):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 8 (Thread 15205):
#0 0x00007f3001f6cec3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (Thread 15204):
#0 0x00007f3001f6cec3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (Thread 15201):
#0 0x00007f3001f6cec3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 5 (Thread 15200):
#0 0x00007f2fff5d2084 in ?? ()
#1 0x0000000000000200 in ?? ()
#2 0x0000000000002000 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 4 (Thread 15203):
#0 0x00007f3001f6cec3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (Thread 15198):
#0 0x00007f2fff5d1a47 in ?? ()
#1 0x00007f30026517fa in kvm_run (env=0x7f30049c2720)
at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1015
#2 0x00007f3002651cb9 in kvm_cpu_exec (env=<value optimized out>)
at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1744
#3 0x00007f3002652b9d in kvm_main_loop_cpu (_env=0x7f30049c2720)
at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2005
#4 ap_main_loop (_env=0x7f30049c2720)
at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2061
#5 0x00007f3001f65851 in ?? ()
#6 0x00007f2ff9bb1700 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (Thread 15199):
#0 0x00007f2fff5d0253 in ?? ()
#1 0x00007f2faaa5fbc0 in ?? ()
#2 0x00007f300000000a in ?? ()
#3 0x0000000000000014 in ?? ()
#4 0x00007f2fa40008f8 in ?? ()
#5 0xffffbe989e3eadf7 in ?? ()
#6 0x00007f2fffd5a9f4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (Thread 15196):
#0 0x00007f2fff5238a5 in ?? ()
#1 0x00007f2fff525085 in ?? ()
#2 0x00007f3006749f28 in ?? ()
#3 0x00007f2fff87e860 in ?? ()
#4 0x00007f300674a080 in ?? ()
#5 0x0000000000001e60 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x00007f300674a200 in ?? ()
#8 0x0000000000000001 in ?? ()
#9 0x00007f2fff5f0dcb in ?? ()
#10 0x0000003000000028 in ?? ()
#11 0x00007f3006749ff0 in ?? ()
#12 0x00007f3006749f20 in ?? ()
#13 0x00007f2fff648be0 in ?? ()
#14 0x0000000000000003 in ?? ()
#15 0x00007f30023948d9 in _dl_allocate_tls () from /lib64/ld-linux-x86-64.so.2
#16 0x0000000000000000 in ?? ()
Can you attach "ps -eLf" output? Hello Paolo, there is the output of "ps -eLf". I remember I touch the hugepage, "sysctl vm.nr_hugepages=100". is that the reason? Created attachment 790212 [details]
ps output
I capture this abort from gdb:
(gdb) c
Continuing.
[New Thread 0x7fbc74caa700 (LWP 20348)]
[New Thread 0x7fbc279fc700 (LWP 20349)]
[New Thread 0x7fbc26ffb700 (LWP 20350)]
[New Thread 0x7fbc265fa700 (LWP 20351)]
[New Thread 0x7fbc25bf9700 (LWP 20352)]
[New Thread 0x7fbc251f8700 (LWP 20353)]
[New Thread 0x7fbc1ffff700 (LWP 20354)]
[New Thread 0x7fbc1f5fe700 (LWP 20355)]
[New Thread 0x7fbc1ebfd700 (LWP 20356)]
[New Thread 0x7fbc1e1fc700 (LWP 20357)]
[New Thread 0x7fbc1d7fb700 (LWP 20358)]
[New Thread 0x7fbc1cdfa700 (LWP 20359)]
[New Thread 0x7fbc17fff700 (LWP 20360)]
[New Thread 0x7fbc175fe700 (LWP 20361)]
[New Thread 0x7fbc16bfd700 (LWP 20362)]
[New Thread 0x7fbc161fc700 (LWP 20363)]
[New Thread 0x7fbc157fb700 (LWP 20364)]
Program received signal SIGABRT, Aborted.
0x00007fbc7dbe98a5 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007fbc7dbe98a5 in raise () from /lib64/libc.so.6
#1 0x00007fbc7dbeb085 in abort () from /lib64/libc.so.6
#2 0x00007fbc80d39371 in die2 (err=<value optimized out>,
what=0x7fbc80eb39e4 "pthread_create")
at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:79
#3 0x00007fbc80d3991e in thread_create (aiocb=0x7fbc824227b0)
at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:117
#4 spawn_thread (aiocb=0x7fbc824227b0)
at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:397
#5 qemu_paio_submit (aiocb=0x7fbc824227b0)
at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:408
#6 0x00007fbc80d39a9b in paio_submit (bs=<value optimized out>, fd=10,
sector_num=4196904, qiov=0x7fbc887ddc80, nb_sectors=8,
cb=<value optimized out>, opaque=0x7fbc872f9110, type=2)
at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:595
#7 0x00007fbc80d57153 in raw_aio_submit (bs=0x7fbc8216e9f0,
sector_num=4196904, qiov=0x7fbc887ddc80, nb_sectors=8,
cb=0x7fbc80d2cce0 <bdrv_co_io_em_complete>, opaque=<value optimized out>,
type=<value optimized out>)
at /usr/src/debug/qemu-kvm-0.12.1.2/block/raw-posix.c:448
#8 0x00007fbc80d57200 in raw_aio_writev (bs=<value optimized out>,
sector_num=<value optimized out>, qiov=<value optimized out>,
nb_sectors=<value optimized out>, cb=<value optimized out>,
opaque=<value optimized out>)
at /usr/src/debug/qemu-kvm-0.12.1.2/block/raw-posix.c:464
#9 0x00007fbc80d2dd43 in bdrv_co_io_em (bs=0x7fbc8216e9f0,
sector_num=4196904, nb_sectors=8, iov=0x7fbc887ddc80, is_write=true)
at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4109
#10 0x00007fbc80d2f206 in bdrv_co_do_writev (bs=0x7fbc8216e9f0,
sector_num=4196904, nb_sectors=8, qiov=0x7fbc887ddc80,
flags=<value optimized out>)
at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2298
#11 0x00007fbc80d2f206 in bdrv_co_do_writev (bs=0x7fbc8216e010,
sector_num=4196904, nb_sectors=8, qiov=0x7fbc887ddc80,
flags=<value optimized out>)
at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2298
#12 0x00007fbc80d2f2c1 in bdrv_co_do_rw (opaque=0x7fbc8245e5d0)
---Type <return> to continue, or q <return> to quit---
at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3962
#13 0x00007fbc80d3889b in coroutine_trampoline (i0=<value optimized out>,
i1=<value optimized out>)
at /usr/src/debug/qemu-kvm-0.12.1.2/coroutine-ucontext.c:129
#14 0x00007fbc7dbfab70 in ?? () from /lib64/libc.so.6
#15 0x00007fff173c9310 in ?? ()
#16 0x0000000000000000 in ?? ()
hmm, I think I found the problem: When I check /proc/20307/task/, I found there are 20 threads. So I remember that I have configured "max_processes = 20" in /etc/libvirt/qemu.conf. After comment this configuration and restart libvirt, qemu-kvm can start. I think I should close this BZ as NOTABUG. I'm not sure what max_processes is for... Jiri, can you check this bug? You added it in commit 87e78b2 (qemu: Support for overriding NPROC limit, 2011-04-05). # If max_processes is set to a positive integer, libvirt will use # it to set the maximum number of processes that can be run by qemu # user. This can be used to override default value set by host OS. # The same applies to max_files which sets the limit on the maximum # number of opened files. The reason for adding it was that for big machines with lots of running qemu domains (possibly also on a lots of CPUs) the default system limit on the number of processes that any user can run was too low. One can use max_processes to increase this limit (which of course must be done prior to starting any domains). Setting max_processes to 20 is insane (except for testing that max_processes option actually influences the system limit). |
Description of problem: qemu-kvm core dump when start from libvirt. qemu log says "pthread_create failed: Resource temporarily unavailable". Version-Release number of selected component (if applicable): libvirt-0.10.2-23.el6.x86_64 qemu-kvm-0.12.1.2-2.398.el6.x86_64 How reproducible: always Steps to Reproduce: 1. # virsh dumpxml r6 <domain type='kvm'> <name>r6</name> <uuid>fbd9d4c5-ae9c-2e3c-0265-894fbe093d7a</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='rhel6.5.0'>hvm</type> <loader>/usr/share/seabios/bios.bin</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/var/lib/libvirt/images/kvm-rhel6.4.z-x86_64-raw.img'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0' model='piix3-uhci'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/r6.agent'/> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'/> </domain> 2. # virsh start r6 Domain r6 started The guest will shutdown after grub stage, that caused by qemu-kvm coredump. Actual results: coredump Expected results: no error Additional info: I also tried old version qemu-kvm-0.12.1.2-2.377.el6.x86_64, still coredump.