Calling ulimit from the shell parent process is one approach; another is to have libvirt itself use setrlimit() from <sys/resource.h> at runtime.  However, if libvirt increases the limit at runtime, this doesn't affect any child processes already spawned prior to the increase; it matters if libvirt needs to pass a higher-value fd to a child qemu than the child's ulimit that was inherited from an earlier point under the smaller limit.

Perhaps we need to add a line to libvirtd.conf that lists the ulimit to be set at libvirt startup, prior to creating any qemu children.

The limit cannot be increased in runtime since qemu-kvm processes which are already running when we decide to increase the limit will still be running with the old limit and checked against it so they will not be allowed to clone() until number of processes owned by qemu user gets back under the old limit.

IMHO, from libvirt's point of view this should be a documentation issue.

For vdsm, the best solution is to provide and install a file under /etc/security/limits.d/ directory which would set a limit for user qemu (or whatever user qemu-kvm is usued to run as).

Something like the following (you can consult /etc/security/limits.conf for the syntax):

/etc/security/limits.d/99-vdsm.conf:
qemu   soft   nproc   30000

(In reply to comment #2)
> The limit cannot be increased in runtime since qemu-kvm processes which are
> already running when we decide to increase the limit will still be running with
> the old limit and checked against it so they will not be allowed to clone()
> until number of processes owned by qemu user gets back under the old limit.
> 
> IMHO, from libvirt's point of view this should be a documentation issue.
> 
> For vdsm, the best solution is to provide and install a file under
> /etc/security/limits.d/ directory which would set a limit for user qemu (or
> whatever user qemu-kvm is usued to run as).
> 
> Something like the following (you can consult /etc/security/limits.conf for the
> syntax):
> 
> /etc/security/limits.d/99-vdsm.conf:
> qemu   soft   nproc   30000

works - as suggested, created 99-vdsm.conf under /etc/security/limits.d/ with qemu user, restarted vdsmd\libvirtd, and managed to create 190 guests (no qemu unexpected deaths).
also, login to system as qemu user, ulimit -u showed 30000.

> works - as suggested, created 99-vdsm.conf under /etc/security/limits.d/ with
> qemu user, restarted vdsmd\libvirtd, and managed to create 190 guests (no qemu
> unexpected deaths).

I'm struggling to see *how* this worked.  AFAICT, the only thing in Linux which ever loads the rules set in /etc/security/limits.d is the PAM limits module. Nothing in libvirt uses PAM when spawning QEMU processes, so I don't see how the QEMU process will ever see the raised limit.

> also, login to system as qemu user, ulimit -u showed 30000.

This is not a good confirmation of behaviour of QEMU spawned from libvirt. When you login as 'qemu', the login program is using PAM, and thus loading the limits.

What is actually happening is that QEMU is inheriting the ulimit settings from libvirtd, and the settings libvirt has are inherited from the process which started libvirt.

If 'init' started libvirtd at system boot, then (on my Fedora 14 system) it gets  nprocesses=30732

If the admin started libvirtd from a login shell with 'service libvirtd start', then it gets the nprocesses=1024.


If we want to reliably control the limits that QEMU sees, we need to manually call setrlimit() when spawning QEMU.

BTW, to see what QEMU is actually running with you need to look in /proc

  PID=`pgrep qemu`
  grep process /proc/$PID/limits

(In reply to comment #6)
> BTW, to see what QEMU is actually running with you need to look in /proc
> 
>   PID=`pgrep qemu`
>   grep process /proc/$PID/limits

well, you are correct, as you anticipated, it has only 1024. 
probably running 185 gusts is not the real reason for qemu to consume its limits. 

[root@rhev-i32c-01 core]# cat /proc/21678/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             515177               515177               processes
Max open files            1024                 1024                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       515177               515177               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2011-April/msg00245.html

http://post-office.corp.redhat.com/archives/rhvirt-patches/2011-April/msg00121.html

Hi Eric , 
Would you please help check the steps to verify this bug . If it is correct ,
I'll move the bug status to VERIFIED . 

Thanks
Vivian 


tested with libvirt-0.8.7-16.el6.x86_64 

with libvirt-0.8.7-15.el6.x86_64

in /etc/libvirt/qemu.conf file there isn't the 
# If max_processes is set to a positive integer, libvirt will use it to set
# maximum number of processes that can be run by qemu user. This can be used to
# override default value set by host OS.
#
# max_processes = 0


But with libvirt-0.8.7-16.el6.x86_64 we got the "max_processes" option  

[Steps to check up the function]
1. set max_processes = 1 
2. virsh start guest (get a hang of course)
   # cat /proc/60741/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             1                    1                    processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       8247730              8247730              signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us   

3.  set max_process = 5
4.  virsh start guest (could start up guest)

5. leave the max_processes = 0 commented , and restart libvirtd , 
the default 1024 max processes will be set , and I got 257 guest started , and
worked well on my testing machine . 

So we can manually set the ulimit depending on # , and make the qemu-kvm guest
run successfully .

(In reply to comment #12)
> Hi Eric , 
> Would you please help check the steps to verify this bug . If it is correct ,
> I'll move the bug status to VERIFIED . 

> 1. set max_processes = 1 
> 2. virsh start guest (get a hang of course)
>    # cat /proc/60741/limits 

> Max processes             1                    1                    processes 

> 
> 3.  set max_process = 5
> 4.  virsh start guest (could start up guest)
> 
> 5. leave the max_processes = 0 commented , and restart libvirtd , 
> the default 1024 max processes will be set , and I got 257 guest started , and
> worked well on my testing machine . 

Looks like a good test to me.

According to comment #12 and comment #13 . Set this bug status to VERIFIED .

tested with libvirt-0.8.7-18.el6.x86_64
In /etc/libvirt/qemu.conf file , there is the  set max_processes = 1 . Remove the comment # , and restart libvirtd , it could make effort on qemu process ulimit number . 

So keep the VERIFIED status

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0596.html