Bug 1372589

little correction:

Each connection from RBD image to OSD needs 1 fd and 2 threads for example if you have 200 OSDs:
  - This means we will need 200 fds and 400 threads for 1 RBD image.

  max_files = fds (open fds)
  max_processes = threads

 - Lets consider that user has not more than 500 RBD images in these 200 OSDs
 
  max_files = 500(RBD images) * 200(fds) = 100000
  max_processes = 500(RBD images) * 400 (threads) =  200000

But as number of files limit is per process basis and number of process limit is system wide user.

So we hit mostly FD limits not the max_processes limit.


- Same is given in man page of getrlimit function , means number of files limit is per process and processes limit is for system wide user.
   # man 2 getrlimit

   RLIMIT_NOFILE
              Specifies a value one greater than the maximum file descriptor number that can be opened by this process.  Attempts (open(2), pipe(2), dup(2), etc.)  to exceed  this  limit  yield  the  error
              EMFILE.  (Historically, this limit was named RLIMIT_OFILE on BSD.)

       RLIMIT_NPROC
              The  maximum  number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process.  Upon encountering this limit, fork(2) fails with the
              error EAGAIN.  This limit is not enforced for processes that have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability.

- For now we can start from here ---

/etc/libvirt/qemu.conf
  max_files = 131072
  max_processes = 65536

For more information please check below given article:

Ceph - VM hangs when transfering large amounts of data to RBD disk
https://access.redhat.com/solutions/1602683

This seems as Ceph requirement and its related issue in QEMU. Please re-assign if the evaluation of the group assignment is wrong.

*** Bug 1389503 has been marked as a duplicate of this bug. ***

Thanks for adjusting this!  max_files seems fine.

I'm a little unclear about what max_processes means, exactly.  Is it the maximum number of threads per user?  Sorry to be pedantic, I may be overthinking this, just trying to get a picture of what is being tuned.  How does this max_processes relate to kernel.pid-max change here:

https://bugzilla.redhat.com/show_bug.cgi?id=1389502#c8

(In reply to Ben England from comment #9)
> Thanks for adjusting this!  max_files seems fine.
> 
> I'm a little unclear about what max_processes means, exactly.  Is it the
> maximum number of threads per user?  Sorry to be pedantic, I may be
> overthinking this, just trying to get a picture of what is being tuned.  How
> does this max_processes relate to kernel.pid-max change here:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1389502#c8

hi Ben, yes max_processes in qemu.conf seems to set the maximum number of processes (counting threads too) for the user which libvirtd uses to launch qemu instances.

The pid-max systctl is global, not per-user and I've basically multiplied all three (max_files too) by a factor of 32, yet I guess the real goal for the bugs is to make them customizable.

verified on openstack-tripleo-heat-templates-6.0.0-3.el7ost.noarch

1) the heat templates have been merged to tripleo-heat-templates
2) the configuration on /etc/libvirt/qemu.conf has been set properly

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245

The problem reoccurred on RHOSP 11, see comment https://bugzilla.redhat.com/show_bug.cgi?id=1430002#c10.  Does /etc/security/limits.d/20-nproc.conf need to bump up the process limit from 4096 to a higher value?  If so I will re-open bz.

I verified that the qemu user does not have privs to create enough threads for librados with 1000 OSDs.   This problem is going away with RHCS 3 with async messenger, which requires way fewer threads.

The program here:

http://perf1.perf.lab.eng.bos.redhat.com/bengland/public/openstack/
thread-create.c

tests thread creation limits, and indeed it fails because of limits on the qemu account placed by /etc/security/limits.d/20-nproc.conf , but if you raise this limit or change it with ulimit, then the problem goes away.  So I guess that's the workaround.  I had to change qemu account in /etc/passwd to allow a shell to do this.

# su - qemu
Last login: Tue Jun 13 20:43:25 UTC 2017 on pts/0
-bash-4.2$ /tmp/thread-create 4096
thread count: 4096
cat /proc/358010/limits
Limit                     Soft Limit           Hard Limit           Units  
...   
Max processes             4096                 1030485              processes 
...
x: 0
fatal: Error creating thread
errno 11: Resource temporarily unavailable

-bash-4.2$ tail /etc/security/limits.d/20-nproc.conf 
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     4096
root       soft    nproc     unlimited