Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1970597

Summary:	Setting environment variable TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES for all the ceph daemons
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Vikhyat Umrao <vumrao>
Component:	Cephadm	Assignee:	Sebastian Wagner <sewagner>
Status:	CLOSED ERRATA	QA Contact:	Preethi <pnataraj>
Severity:	high	Docs Contact:	Karen Norteman <knortema>
Priority:	high
Version:	5.0	CC:	adking, gabrioux, jdurgin, jefbrown, nojha, pnataraj, tserlin, twilkins, vereddy
Target Milestone:	---	Keywords:	Performance, Regression
Target Release:	5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-16.2.0-84.el8cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-08-30 08:31:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vikhyat Umrao 2021-06-10 18:32:16 UTC

Description of problem:
Setting environment variable TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES for all the ceph daemons


Version-Release number of selected component (if applicable):
RHCS 5

Comment 1 Veera Raghava Reddy 2021-06-14 12:36:31 UTC

Hi Vikhyat, Is this required for 5.0?

Comment 2 Vikhyat Umrao 2021-06-14 12:40:23 UTC

(In reply to Veera Raghava Reddy from comment #1)
> Hi Vikhyat, Is this required for 5.0?

If time permits I think we should take it in 5.0 @jdurgin what do you think?

Comment 3 Vikhyat Umrao 2021-06-14 15:38:10 UTC

(In reply to Vikhyat Umrao from comment #2)
> (In reply to Veera Raghava Reddy from comment #1)
> > Hi Vikhyat, Is this required for 5.0?
> 
> If time permits I think we should take it in 5.0 @jdurgin what do
> you think?

PR - https://github.com/ceph/ceph/pull/41805 is merged maybe we can take it. This is a feature regression as compared to 4.x and the workaround needs direct modification in unit files which is not clean for production clusters?

Comment 4 Josh Durgin 2021-06-14 18:08:36 UTC

Yes, (In reply to Vikhyat Umrao from comment #2)
> (In reply to Veera Raghava Reddy from comment #1)
> > Hi Vikhyat, Is this required for 5.0?
> 
> If time permits I think we should take it in 5.0 @jdurgin what do
> you think?

Yes, this has significant impact to performance, so it's a blocker for 5.0 imo.

Comment 5 Vikhyat Umrao 2021-06-14 18:55:42 UTC

(In reply to Josh Durgin from comment #4)
> Yes, (In reply to Vikhyat Umrao from comment #2)
> > (In reply to Veera Raghava Reddy from comment #1)
> > > Hi Vikhyat, Is this required for 5.0?
> > 
> > If time permits I think we should take it in 5.0 @jdurgin what do
> > you think?
> 
> Yes, this has significant impact to performance, so it's a blocker for 5.0
> imo.

Thank you, Josh.

Comment 6 Veera Raghava Reddy 2021-06-15 04:06:00 UTC

Will this take care of 4.x cluster upgraded to 5.0?

Comment 7 Vikhyat Umrao 2021-06-15 09:47:19 UTC

(In reply to Veera Raghava Reddy from comment #6)
> Will this take care of 4.x cluster upgraded to 5.0?

I think it should be taken care of as the upgrades will be handled via ceph-ansible and in 4.2.z2 we are fixing it for containers. @gabrioux is my understanding correct?

Comment 9 Sebastian Wagner 2021-06-17 08:49:15 UTC

pacific PR: https://github.com/ceph/ceph/pull/41913

Comment 10 Yaniv Kaul 2021-06-23 12:50:57 UTC

(In reply to Sebastian Wagner from comment #9)
> pacific PR: https://github.com/ceph/ceph/pull/41913

That looks like a fairly large change... What's the prospect for it?

Comment 11 Adam King 2021-06-25 13:24:27 UTC

(In reply to Yaniv Kaul from comment #10)
> (In reply to Sebastian Wagner from comment #9)
> > pacific PR: https://github.com/ceph/ceph/pull/41913
> 
> That looks like a fairly large change... What's the prospect for it?

The cephadm team is currently doing backports to pacific in large batches rather than individual changes for efficiency. Most of those changes in that PR are unrelated to this BZ. We would only be bringing https://github.com/ceph/ceph/pull/41913/commits/777a9a45aed0d243c52788c7df32d87f0e1be723 downstream which is just a few lines setting TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES to a good default.

Comment 17 Preethi 2021-06-28 10:40:36 UTC

@Vikhyat, The issue is fixed in latest ceph version 16.2.0-84. Below is the snippet of verification. Moving to verified state.

[cephuser@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp ~]$ sudo cephadm version
Using recent ceph image registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:43445a111804f0974a87476fbc42d70144103ce1016f88f9a7ea1e9c69bf6701
ceph version 16.2.0-84.el8cp (548c78dcee1fb77a3a3b1a8d3c747a9a5f60b2e1) pacific (stable)
[cephuser@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp ~]$  ps -ef | grep ceph-osd
root       44944   44934  0 02:09 ?        00:00:00 /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        44957   44944  0 02:09 ?        00:01:03 /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root       45915   45904  0 02:09 ?        00:00:00 /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        45925   45915  0 02:09 ?        00:00:57 /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root       46895   46884  0 02:09 ?        00:00:00 /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        46930   46895  0 02:09 ?        00:01:05 /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root       48094   48082  0 02:09 ?        00:00:00 /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        48104   48094  0 02:09 ?        00:01:03 /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
cephuser  109236  108973  0 06:33 pts/0    00:00:00 grep --color=auto ceph-osd
[cephuser@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp ~]$ cat /proc/44944/environ
cat: /proc/44944/environ: Permission denied
[cephuser@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp ~]$ sudo su
[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp cephuser]# cat /proc/44944/environ
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociCEPH_DEVEL=falseOSD_FLAVOR=defaultNODE_NAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expCEPH_POINT_RELEASE=CEPH_REF=pacificCEPH_VERSION=pacificCONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:43445a111804f0974a87476fbc42d70144103ce1016f88f9a7ea1e9c69bf6701CEPH_USE_RANDOM_NONCE=1TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728HOME=/rootHOSTNAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp cephuser]# 



[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp cephuser]# ps -ef | grep ceph-mon
root       21238   21226  0 02:01 ?        00:00:00 /dev/init -- /usr/bin/ceph-mon -n mon.ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug  --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true
167        21248   21238  1 02:01 ?        00:03:20 /usr/bin/ceph-mon -n mon.ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug  --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true
root      110284  109241  0 06:37 pts/0    00:00:00 grep --color=auto ceph-mon
[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp cephuser]# cat /proc/21238/environ
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociCONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-5.0-rhel-8-containers-candidate-69997-20210626035550CEPH_REF=pacificCEPH_POINT_RELEASE=CEPH_DEVEL=falseCEPH_USE_RANDOM_NONCE=1TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728OSD_FLAVOR=defaultCEPH_VERSION=pacificNODE_NAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expHOME=/rootHOSTNAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp cephuser]#

Comment 18 Preethi 2021-06-28 12:10:50 UTC

Please ignore my previous comments, as the output for verification has PIDs for /dev/init instead if /usr/bin

MON -

[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# ps -ef | grep ceph-mon
root       21238   21226  0 02:01 ?        00:00:00 /dev/init -- /usr/bin/ceph-mon -n mon.ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug  --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true
167        21248   21238  1 02:01 ?        00:04:20 /usr/bin/ceph-mon -n mon.ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug  --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true
root      123512  121879  0 08:01 pts/0    00:00:00 grep --color=auto ceph-mon
[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# cat /proc/21248/environ 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociCONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-5.0-rhel-8-containers-candidate-69997-20210626035550CEPH_REF=pacificCEPH_POINT_RELEASE=CEPH_DEVEL=falseCEPH_USE_RANDOM_NONCE=1TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728OSD_FLAVOR=defaultCEPH_VERSION=pacificNODE_NAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expHOME=/rootHOSTNAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# ps -ef | grep ceph-osd
root       44944   44934  0 02:09 ?        00:00:00 /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        44957   44944  0 02:09 ?        00:01:24 /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root       45915   45904  0 02:09 ?        00:00:00 /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        45925   45915  0 02:09 ?        00:01:15 /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root       46895   46884  0 02:09 ?        00:00:00 /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        46930   46895  0 02:09 ?        00:01:26 /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root       48094   48082  0 02:09 ?        00:00:00 /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        48104   48094  0 02:09 ?        00:01:23 /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root      123609  121879  0 08:02 pts/0    00:00:00 grep --color=auto ceph-osd
[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# cat /proc/44957/environ 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociCEPH_DEVEL=falseOSD_FLAVOR=defaultNODE_NAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expCEPH_POINT_RELEASE=CEPH_REF=pacificCEPH_VERSION=pacificCONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:43445a111804f0974a87476fbc42d70144103ce1016f88f9a7ea1e9c69bf6701CEPH_USE_RANDOM_NONCE=1TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728HOME=/rootHOSTNAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]#


[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# cat /proc/45925/environ 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociOSD_FLAVOR=defaultCEPH_VERSION=pacificCEPH_POINT_RELEASE=NODE_NAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expTCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:43445a111804f0974a87476fbc42d70144103ce1016f88f9a7ea1e9c69bf6701CEPH_DEVEL=falseCEPH_REF=pacificCEPH_USE_RANDOM_NONCE=1HOME=/rootHOSTNAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# cat /proc/46930/environ 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociCEPH_REF=pacificCEPH_VERSION=pacificCONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:43445a111804f0974a87476fbc42d70144103ce1016f88f9a7ea1e9c69bf6701CEPH_USE_RANDOM_NONCE=1TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728OSD_FLAVOR=defaultCEPH_POINT_RELEASE=CEPH_DEVEL=falseNODE_NAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expHOME=/rootHOSTNAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# cat /proc/48104/environ 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociOSD_FLAVOR=defaultTCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728CEPH_VERSION=pacificCEPH_DEVEL=falseNODE_NAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expCEPH_USE_RANDOM_NONCE=1CEPH_POINT_RELEASE=CEPH_REF=pacificCONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:43445a111804f0974a87476fbc42d70144103ce1016f88f9a7ea1e9c69bf6701HOME=/rootHOSTNAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]#

MGR-
[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# ps -ef | grep ceph-mgr
root       21575   21564  0 02:01 ?        00:00:00 /dev/init -- /usr/bin/ceph-mgr -n mgr.ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp.qbceoj -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        21586   21575  1 02:01 ?        00:05:46 /usr/bin/ceph-mgr -n mgr.ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp.qbceoj -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root      124733  121879  0 08:07 pts/0    00:00:00 grep --color=auto ceph-mgr
[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# cat /proc/21586/environ 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociCEPH_VERSION=pacificCEPH_USE_RANDOM_NONCE=1TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728CEPH_REF=pacificOSD_FLAVOR=defaultNODE_NAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expCEPH_POINT_RELEASE=CEPH_DEVEL=falseCONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-5.0-rhel-8-containers-candidate-69997-20210626035550HOME=/var/lib/cephHOSTNAME=ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-expGLIBCPP_FORCE_NEW=1GLIBCXX_FORCE_NEW=1[root@ceph-testing-1624859654614-node1-installer-mon-mgr-osd-node-exp /]# 

RGW-
root@ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg /]# ps -ef | grep radosgw
root       33623   33612  0 02:11 ?        00:00:00 /dev/init -- /usr/bin/radosgw -n client.rgw.myrgw.ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg.yrbhfs -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        33633   33623  0 02:11 ?        00:00:39 /usr/bin/radosgw -n client.rgw.myrgw.ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg.yrbhfs -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root       66568   66552  0 08:09 pts/0    00:00:00 grep --color=auto radosgw
[root@ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg /]# cat /proc/33633/environ 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociCEPH_VERSION=pacificCEPH_POINT_RELEASE=CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:43445a111804f0974a87476fbc42d70144103ce1016f88f9a7ea1e9c69bf6701NODE_NAME=ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rgCEPH_REF=pacificOSD_FLAVOR=defaultCEPH_DEVEL=falseCEPH_USE_RANDOM_NONCE=1TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728HOME=/rootHOSTNAME=ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg[root@ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg /]# 


MDS-
[root@ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg /]# ps -ef | grep ceph-mds
root       32312   32301  0 02:10 ?        00:00:00 /dev/init -- /usr/bin/ceph-mds -n mds.cephfs.ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg.itkikv -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
167        32323   32312  0 02:10 ?        00:00:05 /usr/bin/ceph-mds -n mds.cephfs.ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg.itkikv -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug 
root       66589   66552  0 08:10 pts/0    00:00:00 grep --color=auto ceph-mds
[root@ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg /]# cat /proc/32323/environ 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binTERM=xtermcontainer=ociCEPH_REF=pacificOSD_FLAVOR=defaultCEPH_POINT_RELEASE=CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:43445a111804f0974a87476fbc42d70144103ce1016f88f9a7ea1e9c69bf6701NODE_NAME=ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rgCEPH_VERSION=pacificCEPH_DEVEL=falseCEPH_USE_RANDOM_NONCE=1TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728HOME=/rootHOSTNAME=ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg[root@ceph-testing-1624859654614-node3-mon-osd-node-exporter-crash-rg /]#

Comment 20 errata-xmlrpc 2021-08-30 08:31:13 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294

Comment 21 Red Hat Bugzilla 2023-09-15 01:09:42 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days