Bug 1459891 - [Docs] Director should increase kernel.thread-max on ceph backed compute nodes
[Docs] Director should increase kernel.thread-max on ceph backed compute nodes
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation (Show other bugs)
11.0 (Ocata)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: RHOS Documentation Team
RHOS Documentation Team
: Documentation
Depends On:
  Show dependency treegraph
Reported: 2017-06-08 09:10 EDT by Tomas Rusnak
Modified: 2018-03-05 22:07 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Tomas Rusnak 2017-06-08 09:10:43 EDT
Description of problem:

In large scale deployments the kernel.pid_max parameter is set to higher value then default - 1048576. This was solved in BZ138950. The parameter is saying how many numerical pid numbers can be assigned by kernel. I don't think, this will solve the problem completely, as there is different parameter which limits number of concurrent running threads on system. 
/proc/sys/kernel/threads-max is actually the maximum number of elements contained in the data structure task_struct. Which is the data structure that contains the list of processes/threads.
So if we just raise kernel.pid_max and we let kernel.threads-max at default value 46031, we should touch this limit in large scales.

Version-Release number of selected component (if applicable):
Ocata, Newton, Pike

How reproducible:
I haven't system in such big scale available

Steps to Reproduce:
1. # cat /proc/sys/kernel/pid_max
2. # cat /proc/sys/kernel/threads-max

Actual results:

Expected results:

Additional info:
Comment 2 Ben England 2017-07-19 14:46:31 EDT

Good points, it got me thinking harder about this.

We should document thread-related kernel parameter requirements for RHOSP 10 and 11, which depend on the older RHCS 2. 

In RHCS 3.0, Ceph is switching to a new "async messenger", replacing the "simple messenger" component that caused the massive consumption of threads per OSD and per instance (in librados).   So this should not be as much of an issue at that point.  RHCS 3.0 is supposed to be the release used with RHOSP 12 (Pike) and will definitely be used for RHOSP 13 (Queens).

However, for RHOSP 10 and 11, which integrate with RHCS 2, this will still be an issue.  I think threads_max does not have to be as big as pid_max, but with simple messenger you still need on the order of 2 threads/OSD x (guests + OSDs). 

I saw librados pthread_create failing to create a thread recently (RHOSP 11), even with the higher pid_max.  


I used this program to investigate what was going on - it just does pthread_create N times to see how many threads can be run at the same time.


and looked at just kernel.threads-max, kernel.pid_max, and vm.max_map_count

When I started with an untuned RHEL7.3 kernel on a 256-GB host:

[root@c04-h01-6048r ~]# sysctl -a | grep threads-max
kernel.threads-max = 2061221
[root@c04-h01-6048r ~]# sysctl -a | grep pid_max
kernel.pid_max = 57344
[root@c04-h01-6048r ~]# sysctl -a | grep vm.max_map_count   
vm.max_map_count = 65530

[root@c04-h01-6048r ~]# ./thread-create 200000
thread count: 200000
fatal: Error creating thread
errno 12 with thrd=56824: Cannot allocate memory

[root@c04-h01-6048r ~]# sysctl -w kernel.pid_max=1048576
kernel.pid_max = 1048576
[root@c04-h01-6048r ~]# sysctl -w vm.max_map_count=400000
vm.max_map_count = 400000

[root@c04-h01-6048r ~]# ./thread-create 200000
thread count: 200000
fatal: Error creating thread
errno 12 with thrd=199989: Cannot allocate memory

[root@c04-h01-6048r ~]# sysctl -w vm.max_map_count=500000
vm.max_map_count = 500000

[root@c04-h01-6048r ~]# ./thread-create 200000
thread count: 200000

So this vm.max_map_count limits how many threads you can create!  Was not obvious to me at first.   Found this in a discussion of JVM thread creation.


Note that on a RHEL7.3 kernel with 256 GB RAM, the thread-max default for this appears to be:

[root@c04-h01-6048r ~]# sysctl -a | grep threads-max
kernel.threads-max = 2061221

So kernel.threads-max was likely not the problem in this case.
Comment 3 Lucy Bopf 2017-07-26 22:20:19 EDT
Clearing target release pending docs triage.
Comment 4 Ben England 2018-02-06 11:31:29 EST
So my conclusion above was that vm.max_map_count had to be increased to >> 2x the total number of threads used by Ceph OSDs or RADOS clients, and before RHCS 3.0, this is quite high for a large cluster.  To calculate:

number of processes using librados (RBD clients, OSDs, RGWs, Cephfs clients) x number of OSDs x 2.  For example, in an RHHI cluster with 36 OSDs/host, and 50 guests with Cinder volumes, and 1000 OSDs in the cluster:

50 guests/host x 1000 OSD connections/guest x 2 threads/OSD = 100000
36 OSDs/host x 1000 OSD connections/OSD x 2 threads/connection = 72000

The default value on RHEL7.4 is 65530.  So Ceph would not be able to connect up the cluster.

This problem does go away in RHCS 3.0 but there is still a huge problem with RHOSP 12 and RHOSP 11 support.  At a minimum this needs to be documented.  Can we get this fix into RHOSP12.z?

Note You need to log in before you can comment on or make changes to this bug.