Hide Forgot
Request for adding deployment option to director to bump up the max_files and max_processes in /etc/libvirt/qemu.conf We are seeing that the current default limit deployed in RHEL7 is 1024 that is not enough for deployment with Ceph cluster. When we have a Ceph cluster, each librbd needs 1 fd and 2 threads(read and write), so in worst case each RBD is talking to each OSD in cluster. For a medium size cluster it is 200 OSDs: Each OSD needs 1 fd and 2 threads: - This means we will need 200 fds and 400 threads for 1 RBD image. max_files = fds (open fds) max_processes = threads - Lets consider that user has not more than 500 RBD images in these 200 OSDs max_files = 500(RBD images) * 200(fds) = 100000 max_processes = 400(threads) * 200(fds) = 80000 Yet this is the worst scenario, and we think it would make sense to bump the value for OSP director to max_files = 131072 max_processes = 65536: /etc/libvirt/qemu.conf max_files = 131072 max_processes = 65536 Thank you, Robin Cernin
little correction: Each connection from RBD image to OSD needs 1 fd and 2 threads for example if you have 200 OSDs: - This means we will need 200 fds and 400 threads for 1 RBD image. max_files = fds (open fds) max_processes = threads - Lets consider that user has not more than 500 RBD images in these 200 OSDs max_files = 500(RBD images) * 200(fds) = 100000 max_processes = 500(RBD images) * 400 (threads) = 200000 But as number of files limit is per process basis and number of process limit is system wide user. So we hit mostly FD limits not the max_processes limit. - Same is given in man page of getrlimit function , means number of files limit is per process and processes limit is for system wide user. # man 2 getrlimit RLIMIT_NOFILE Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.) RLIMIT_NPROC The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN. This limit is not enforced for processes that have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability. - For now we can start from here --- /etc/libvirt/qemu.conf max_files = 131072 max_processes = 65536
For more information please check below given article: Ceph - VM hangs when transfering large amounts of data to RBD disk https://access.redhat.com/solutions/1602683
This seems as Ceph requirement and its related issue in QEMU. Please re-assign if the evaluation of the group assignment is wrong.
*** Bug 1389503 has been marked as a duplicate of this bug. ***
Thanks for adjusting this! max_files seems fine. I'm a little unclear about what max_processes means, exactly. Is it the maximum number of threads per user? Sorry to be pedantic, I may be overthinking this, just trying to get a picture of what is being tuned. How does this max_processes relate to kernel.pid-max change here: https://bugzilla.redhat.com/show_bug.cgi?id=1389502#c8
(In reply to Ben England from comment #9) > Thanks for adjusting this! max_files seems fine. > > I'm a little unclear about what max_processes means, exactly. Is it the > maximum number of threads per user? Sorry to be pedantic, I may be > overthinking this, just trying to get a picture of what is being tuned. How > does this max_processes relate to kernel.pid-max change here: > > https://bugzilla.redhat.com/show_bug.cgi?id=1389502#c8 hi Ben, yes max_processes in qemu.conf seems to set the maximum number of processes (counting threads too) for the user which libvirtd uses to launch qemu instances. The pid-max systctl is global, not per-user and I've basically multiplied all three (max_files too) by a factor of 32, yet I guess the real goal for the bugs is to make them customizable.
verified on openstack-tripleo-heat-templates-6.0.0-3.el7ost.noarch 1) the heat templates have been merged to tripleo-heat-templates 2) the configuration on /etc/libvirt/qemu.conf has been set properly
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1245
The problem reoccurred on RHOSP 11, see comment https://bugzilla.redhat.com/show_bug.cgi?id=1430002#c10. Does /etc/security/limits.d/20-nproc.conf need to bump up the process limit from 4096 to a higher value? If so I will re-open bz.
I verified that the qemu user does not have privs to create enough threads for librados with 1000 OSDs. This problem is going away with RHCS 3 with async messenger, which requires way fewer threads. The program here: http://perf1.perf.lab.eng.bos.redhat.com/bengland/public/openstack/ thread-create.c tests thread creation limits, and indeed it fails because of limits on the qemu account placed by /etc/security/limits.d/20-nproc.conf , but if you raise this limit or change it with ulimit, then the problem goes away. So I guess that's the workaround. I had to change qemu account in /etc/passwd to allow a shell to do this. # su - qemu Last login: Tue Jun 13 20:43:25 UTC 2017 on pts/0 -bash-4.2$ /tmp/thread-create 4096 thread count: 4096 cat /proc/358010/limits Limit Soft Limit Hard Limit Units ... Max processes 4096 1030485 processes ... x: 0 fatal: Error creating thread errno 11: Resource temporarily unavailable -bash-4.2$ tail /etc/security/limits.d/20-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning. * soft nproc 4096 root soft nproc unlimited