2172314 – [Workload-DFG] RHEL 9 (cgroups v2) - the pid limits ARE enforced as compared to RHEL8 (cgroup v1) / [6.0]: rgws crashed when 'rgw_thread_pool_size' is set to 2048

Bug 2172314 - [Workload-DFG] RHEL 9 (cgroups v2) - the pid limits ARE enforced as compared to RHEL8 (cgroup v1) / [6.0]: rgws crashed when 'rgw_thread_pool_size' is set to 2048

Summary: [Workload-DFG] RHEL 9 (cgroups v2) - the pid limits ARE enforced as compared ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	Cephadm
Sub Component:
Version:	5.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	5.3z2
Assignee:	Adam King
QA Contact:	Mohit Bisht
Docs Contact:	Akash Raj
URL:
Whiteboard:
Depends On:	2165644
Blocks:	2185621
TreeView+	depends on / blocked

Reported:	2023-02-21 22:54 UTC by Vikhyat Umrao
Modified:	2023-05-19 07:10 UTC (History)
CC List:	14 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	.PID limits are now set to unlimited for all Ceph containers deployed in `cephadm` Previously, in `cephadm`, explicit PID limits were not provided to deployed containers. A default PID limit was being provided by the container engine. Due to this, certain daemons, specifically the Ceph Object Gateway with `rgw_thread_pool_size` set to a higher than default value, were hitting the PID limit and crashing when trying to create additional threads. With this fix, the PID limits are now set to unlimited for all Ceph containers deployed in `cephadm`. Daemon crashes related to hitting the PID limit no longer occur and the processes in the container spawns as many threads as required with no issues.
Clone Of:	2165644
Environment:
Last Closed:	2023-04-11 20:07:59 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHCEPH-6169	0	None	None	None	2023-02-21 22:55:31 UTC
Red Hat Product Errata	RHBA-2023:1732	0	None	None	None	2023-04-11 20:08:56 UTC

Comment 7 Preethi 2023-04-03 07:24:47 UTC

Below for TCMU and iscsi

[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# podman ps -a
CONTAINER ID  IMAGE                                                                                                                         COMMAND               CREATED         STATUS             PORTS       NAMES
b52f980494a0  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n client.crash.c...  55 minutes ago  Up 55 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-crash-ceph-mobisht-5-3z2-ygfgnd-node2
08fb24db9ac9  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n mgr.ceph-mobis...  55 minutes ago  Up 55 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-mgr-ceph-mobisht-5-3z2-ygfgnd-node2-eelstg
8b0b4d58c3e9  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n mon.ceph-mobis...  55 minutes ago  Up 55 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-mon-ceph-mobisht-5-3z2-ygfgnd-node2
1260752b93fb  registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.10                                                              --no-collector.ti...  55 minutes ago  Up 55 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-node-exporter-ceph-mobisht-5-3z2-ygfgnd-node2
c6004728c09d  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.0 -f --set...  53 minutes ago  Up 53 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-0
bc97ed4f0a3b  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.11 -f --se...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-11
cb30d6cb8de8  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.14 -f --se...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-14
2b25b573f38c  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.3 -f --set...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-3
4830b22d76a4  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.6 -f --set...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-6
4c97d09f3c3d  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.9 -f --set...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-9
c256aa0015fd  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n mds.cephfs.cep...  50 minutes ago  Up 50 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-mds-cephfs-ceph-mobisht-5-3z2-ygfgnd-node2-cenfzo
48ae35464719  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n client.rgw.rgw...  45 minutes ago  Up 45 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-rgw-rgw-1-ceph-mobisht-5-3z2-ygfgnd-node2-nilcez
abb662dd2fea  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d                        7 minutes ago   Up 7 minutes ago               ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-iscsi-iscsi1-ceph-mobisht-5-3z2-ygfgnd-node2-pelmkz-tcmu
12a25e2d5699  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d                        7 minutes ago   Up 7 minutes ago               ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-iscsi-iscsi1-ceph-mobisht-5-3z2-ygfgnd-node2-pelmkz
[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# 
[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# ceph version
ceph version 16.2.10-160.el8cp (6977980612de1db28e41e0a90ff779627cde7a8c) pacific (stable)
[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# 



[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# podman inspect abb662dd2fea | grep -i limit
                    "--pids-limit=-1",
               "PidsLimit": 0,
               "Ulimits": [
                         "Name": "RLIMIT_NOFILE",
                         "Name": "RLIMIT_NPROC",

[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# podman inspect 12a25e2d5699 | grep -i limit
                    "--pids-limit=-1",
               "PidsLimit": 0,
               "Ulimits": [
                         "Name": "RLIMIT_NOFILE",
                         "Name": "RLIMIT_NPROC",

Comment 11 errata-xmlrpc 2023-04-11 20:07:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.3 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:1732

Note You need to log in before you can comment on or make changes to this bug.