Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1000319

Summary: qemu-kvm pthread_create failed: Resource temporarily unavailable
Product: Red Hat Enterprise Linux 6 Reporter: Jincheng Miao <jmiao>
Component: qemu-kvmAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 6.5CC: acathrow, bsarathy, dyuan, gsun, jdenemar, jiahu, juzhang, mkenneth, mzhan, pbonzini, qzhang, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-26 04:05:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/libvirt/qemu/r6.log
none
ps output none

Description Jincheng Miao 2013-08-23 07:48:49 UTC 
Description of problem:
qemu-kvm core dump when start from libvirt. 
qemu log says "pthread_create failed: Resource temporarily unavailable".


Version-Release number of selected component (if applicable):
libvirt-0.10.2-23.el6.x86_64
qemu-kvm-0.12.1.2-2.398.el6.x86_64


How reproducible:
always

Steps to Reproduce:
1. 
# virsh dumpxml r6
<domain type='kvm'>
  <name>r6</name>
  <uuid>fbd9d4c5-ae9c-2e3c-0265-894fbe093d7a</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
    <loader>/usr/share/seabios/bios.bin</loader>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/kvm-rhel6.4.z-x86_64-raw.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/r6.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>


2. 
# virsh start r6
Domain r6 started

The guest will shutdown after grub stage, that caused by qemu-kvm coredump.


Actual results:
coredump

Expected results:
no error

Additional info:
I also tried old version qemu-kvm-0.12.1.2-2.377.el6.x86_64, still coredump.
Comment 1 Jincheng Miao 2013-08-23 08:00:20 UTC 
Created attachment 789494 [details]
/var/log/libvirt/qemu/r6.log
Comment 3 Jincheng Miao 2013-08-23 08:03:06 UTC 
the coredump file is too large, and the backtrace is:

(gdb) t a a bt

Thread 20 (Thread 15216):
#0  0x00007f3001f6cec3 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 19 (Thread 15215):
#0  0x00007f3001f6cec3 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 18 (Thread 15214):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 17 (Thread 15213):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 16 (Thread 15202):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 15 (Thread 15212):
#0  0x00007f3001f6cec3 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 14 (Thread 15211):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 13 (Thread 15210):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 12 (Thread 15209):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 11 (Thread 15208):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 10 (Thread 15207):
#0  0x00007f3001f6cec3 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 9 (Thread 15206):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 8 (Thread 15205):
#0  0x00007f3001f6cec3 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 7 (Thread 15204):
#0  0x00007f3001f6cec3 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 6 (Thread 15201):
#0  0x00007f3001f6cec3 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 5 (Thread 15200):
#0  0x00007f2fff5d2084 in ?? ()
#1  0x0000000000000200 in ?? ()
#2  0x0000000000002000 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 4 (Thread 15203):
#0  0x00007f3001f6cec3 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 3 (Thread 15198):
#0  0x00007f2fff5d1a47 in ?? ()
#1  0x00007f30026517fa in kvm_run (env=0x7f30049c2720)
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1015
#2  0x00007f3002651cb9 in kvm_cpu_exec (env=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1744
#3  0x00007f3002652b9d in kvm_main_loop_cpu (_env=0x7f30049c2720)
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2005
#4  ap_main_loop (_env=0x7f30049c2720)
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2061
#5  0x00007f3001f65851 in ?? ()
#6  0x00007f2ff9bb1700 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 15199):
#0  0x00007f2fff5d0253 in ?? ()
#1  0x00007f2faaa5fbc0 in ?? ()
#2  0x00007f300000000a in ?? ()
#3  0x0000000000000014 in ?? ()
#4  0x00007f2fa40008f8 in ?? ()
#5  0xffffbe989e3eadf7 in ?? ()
#6  0x00007f2fffd5a9f4 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 15196):
#0  0x00007f2fff5238a5 in ?? ()
#1  0x00007f2fff525085 in ?? ()
#2  0x00007f3006749f28 in ?? ()
#3  0x00007f2fff87e860 in ?? ()
#4  0x00007f300674a080 in ?? ()
#5  0x0000000000001e60 in ?? ()
#6  0x0000000000000002 in ?? ()
#7  0x00007f300674a200 in ?? ()
#8  0x0000000000000001 in ?? ()
#9  0x00007f2fff5f0dcb in ?? ()
#10 0x0000003000000028 in ?? ()
#11 0x00007f3006749ff0 in ?? ()
#12 0x00007f3006749f20 in ?? ()
#13 0x00007f2fff648be0 in ?? ()
#14 0x0000000000000003 in ?? ()
#15 0x00007f30023948d9 in _dl_allocate_tls () from /lib64/ld-linux-x86-64.so.2
#16 0x0000000000000000 in ?? ()
Comment 4 Paolo Bonzini 2013-08-23 08:28:39 UTC 
Can you attach "ps -eLf" output?
Comment 5 Jincheng Miao 2013-08-26 02:13:11 UTC 
Hello Paolo,

there is the output of "ps -eLf".

I remember I touch the hugepage, "sysctl vm.nr_hugepages=100".

is that the reason?
Comment 6 Jincheng Miao 2013-08-26 02:13:50 UTC 
Created attachment 790212 [details]
ps output
Comment 7 Jincheng Miao 2013-08-26 03:51:56 UTC 
I capture this abort from gdb:

(gdb) c
Continuing.
[New Thread 0x7fbc74caa700 (LWP 20348)]
[New Thread 0x7fbc279fc700 (LWP 20349)]
[New Thread 0x7fbc26ffb700 (LWP 20350)]
[New Thread 0x7fbc265fa700 (LWP 20351)]
[New Thread 0x7fbc25bf9700 (LWP 20352)]
[New Thread 0x7fbc251f8700 (LWP 20353)]
[New Thread 0x7fbc1ffff700 (LWP 20354)]
[New Thread 0x7fbc1f5fe700 (LWP 20355)]
[New Thread 0x7fbc1ebfd700 (LWP 20356)]
[New Thread 0x7fbc1e1fc700 (LWP 20357)]
[New Thread 0x7fbc1d7fb700 (LWP 20358)]
[New Thread 0x7fbc1cdfa700 (LWP 20359)]
[New Thread 0x7fbc17fff700 (LWP 20360)]
[New Thread 0x7fbc175fe700 (LWP 20361)]
[New Thread 0x7fbc16bfd700 (LWP 20362)]
[New Thread 0x7fbc161fc700 (LWP 20363)]
[New Thread 0x7fbc157fb700 (LWP 20364)]

Program received signal SIGABRT, Aborted.
0x00007fbc7dbe98a5 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007fbc7dbe98a5 in raise () from /lib64/libc.so.6
#1  0x00007fbc7dbeb085 in abort () from /lib64/libc.so.6
#2  0x00007fbc80d39371 in die2 (err=<value optimized out>, 
    what=0x7fbc80eb39e4 "pthread_create")
    at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:79
#3  0x00007fbc80d3991e in thread_create (aiocb=0x7fbc824227b0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:117
#4  spawn_thread (aiocb=0x7fbc824227b0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:397
#5  qemu_paio_submit (aiocb=0x7fbc824227b0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:408
#6  0x00007fbc80d39a9b in paio_submit (bs=<value optimized out>, fd=10, 
    sector_num=4196904, qiov=0x7fbc887ddc80, nb_sectors=8, 
    cb=<value optimized out>, opaque=0x7fbc872f9110, type=2)
    at /usr/src/debug/qemu-kvm-0.12.1.2/posix-aio-compat.c:595
#7  0x00007fbc80d57153 in raw_aio_submit (bs=0x7fbc8216e9f0, 
    sector_num=4196904, qiov=0x7fbc887ddc80, nb_sectors=8, 
    cb=0x7fbc80d2cce0 <bdrv_co_io_em_complete>, opaque=<value optimized out>, 
    type=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/raw-posix.c:448
#8  0x00007fbc80d57200 in raw_aio_writev (bs=<value optimized out>, 
    sector_num=<value optimized out>, qiov=<value optimized out>, 
    nb_sectors=<value optimized out>, cb=<value optimized out>, 
    opaque=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/raw-posix.c:464
#9  0x00007fbc80d2dd43 in bdrv_co_io_em (bs=0x7fbc8216e9f0, 
    sector_num=4196904, nb_sectors=8, iov=0x7fbc887ddc80, is_write=true)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4109
#10 0x00007fbc80d2f206 in bdrv_co_do_writev (bs=0x7fbc8216e9f0, 
    sector_num=4196904, nb_sectors=8, qiov=0x7fbc887ddc80, 
    flags=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2298
#11 0x00007fbc80d2f206 in bdrv_co_do_writev (bs=0x7fbc8216e010, 
    sector_num=4196904, nb_sectors=8, qiov=0x7fbc887ddc80, 
    flags=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2298
#12 0x00007fbc80d2f2c1 in bdrv_co_do_rw (opaque=0x7fbc8245e5d0)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3962
#13 0x00007fbc80d3889b in coroutine_trampoline (i0=<value optimized out>, 
    i1=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/coroutine-ucontext.c:129
#14 0x00007fbc7dbfab70 in ?? () from /lib64/libc.so.6
#15 0x00007fff173c9310 in ?? ()
#16 0x0000000000000000 in ?? ()
Comment 8 Jincheng Miao 2013-08-26 04:05:28 UTC 
hmm, I think I found the problem:

When I check /proc/20307/task/, I found there are 20 threads. So I remember that 
I have configured "max_processes = 20" in /etc/libvirt/qemu.conf. 

After comment this configuration and restart libvirt, qemu-kvm can start.

I think I should close this BZ as NOTABUG.
Comment 9 Paolo Bonzini 2013-08-29 15:11:47 UTC 
I'm not sure what max_processes is for...  Jiri, can you check this bug?  You added it in commit 87e78b2 (qemu: Support for overriding NPROC limit, 2011-04-05).
Comment 10 Jiri Denemark 2013-08-29 19:25:59 UTC 
# If max_processes is set to a positive integer, libvirt will use
# it to set the maximum number of processes that can be run by qemu
# user. This can be used to override default value set by host OS.
# The same applies to max_files which sets the limit on the maximum
# number of opened files.

The reason for adding it was that for big machines with lots of running qemu domains (possibly also on a lots of CPUs) the default system limit on the number of processes that any user can run was too low. One can use max_processes to increase this limit (which of course must be done prior to starting any domains). Setting max_processes to 20 is insane (except for testing that max_processes option actually influences the system limit).