Bug 2172314

Summary:	[Workload-DFG] RHEL 9 (cgroups v2) - the pid limits ARE enforced as compared to RHEL8 (cgroup v1) / [6.0]: rgws crashed when 'rgw_thread_pool_size' is set to 2048
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Vikhyat Umrao <vumrao>
Component:	Cephadm	Assignee:	Adam King <adking>
Status:	CLOSED ERRATA	QA Contact:	Mohit Bisht <mobisht>
Severity:	urgent	Docs Contact:	Akash Raj <akraj>
Priority:	unspecified
Version:	5.3	CC:	adking, akraj, ceph-eng-bugs, cephqe-warriors, mobisht, pdhange, pnataraj, racpatel, saraut, skoduri, smanjara, tserlin, vimishra, vumrao
Target Milestone:	---	Keywords:	Regression, Scale, TestBlocker
Target Release:	5.3z2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	.PID limits are now set to unlimited for all Ceph containers deployed in `cephadm` Previously, in `cephadm`, explicit PID limits were not provided to deployed containers. A default PID limit was being provided by the container engine. Due to this, certain daemons, specifically the Ceph Object Gateway with `rgw_thread_pool_size` set to a higher than default value, were hitting the PID limit and crashing when trying to create additional threads. With this fix, the PID limits are now set to unlimited for all Ceph containers deployed in `cephadm`. Daemon crashes related to hitting the PID limit no longer occur and the processes in the container spawns as many threads as required with no issues.	Story Points:	---
Clone Of:	2165644	Environment:
Last Closed:	2023-04-11 20:07:59 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2165644
Bug Blocks:	2185621

Comment 7 Preethi 2023-04-03 07:24:47 UTC

Below for TCMU and iscsi

[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# podman ps -a
CONTAINER ID  IMAGE                                                                                                                         COMMAND               CREATED         STATUS             PORTS       NAMES
b52f980494a0  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n client.crash.c...  55 minutes ago  Up 55 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-crash-ceph-mobisht-5-3z2-ygfgnd-node2
08fb24db9ac9  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n mgr.ceph-mobis...  55 minutes ago  Up 55 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-mgr-ceph-mobisht-5-3z2-ygfgnd-node2-eelstg
8b0b4d58c3e9  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n mon.ceph-mobis...  55 minutes ago  Up 55 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-mon-ceph-mobisht-5-3z2-ygfgnd-node2
1260752b93fb  registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.10                                                              --no-collector.ti...  55 minutes ago  Up 55 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-node-exporter-ceph-mobisht-5-3z2-ygfgnd-node2
c6004728c09d  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.0 -f --set...  53 minutes ago  Up 53 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-0
bc97ed4f0a3b  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.11 -f --se...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-11
cb30d6cb8de8  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.14 -f --se...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-14
2b25b573f38c  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.3 -f --set...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-3
4830b22d76a4  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.6 -f --set...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-6
4c97d09f3c3d  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n osd.9 -f --set...  52 minutes ago  Up 52 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-osd-9
c256aa0015fd  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n mds.cephfs.cep...  50 minutes ago  Up 50 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-mds-cephfs-ceph-mobisht-5-3z2-ygfgnd-node2-cenfzo
48ae35464719  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d  -n client.rgw.rgw...  45 minutes ago  Up 45 minutes ago              ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-rgw-rgw-1-ceph-mobisht-5-3z2-ygfgnd-node2-nilcez
abb662dd2fea  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d                        7 minutes ago   Up 7 minutes ago               ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-iscsi-iscsi1-ceph-mobisht-5-3z2-ygfgnd-node2-pelmkz-tcmu
12a25e2d5699  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2433c9a2159075f977252dc27a5ed51999269434412ec6b00b94ff16e1172e9d                        7 minutes ago   Up 7 minutes ago               ceph-5debfbd4-d1e8-11ed-9b30-fa163e2f7940-iscsi-iscsi1-ceph-mobisht-5-3z2-ygfgnd-node2-pelmkz
[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# 
[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# ceph version
ceph version 16.2.10-160.el8cp (6977980612de1db28e41e0a90ff779627cde7a8c) pacific (stable)
[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# 



[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# podman inspect abb662dd2fea | grep -i limit
                    "--pids-limit=-1",
               "PidsLimit": 0,
               "Ulimits": [
                         "Name": "RLIMIT_NOFILE",
                         "Name": "RLIMIT_NPROC",

[root@ceph-mobisht-5-3z2-ygfgnd-node2 ~]# podman inspect 12a25e2d5699 | grep -i limit
                    "--pids-limit=-1",
               "PidsLimit": 0,
               "Ulimits": [
                         "Name": "RLIMIT_NOFILE",
                         "Name": "RLIMIT_NPROC",

Comment 11 errata-xmlrpc 2023-04-11 20:07:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.3 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:1732