Bug 2101424

Summary: [CEE/SD][ceph-mgr][RFE] ceph-mgr crashing with the error "RuntimeError: uid/gid not found"
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Geo Jose <gjose>
Component: CephadmAssignee: Adam King <adking>
Status: CLOSED WONTFIX QA Contact: Manasa <mgowri>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 5.1CC: bhubbard, cephqe-warriors, epuertat, jolmomar, saraut
Target Milestone: ---Keywords: FutureFeature
Target Release: 7.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-05 15:14:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Geo Jose 2022-06-27 13:01:29 UTC 
Description of problem:
 - ceph-mgr crashing with the error "RuntimeError: uid/gid not found" when there is low/full space on /var/tmp.  

The following improvement is needed:

1. The code needs to handle the above exception.
2. "ceph health detail" should report the warnings prior to stop working mgr / Alert when disk space is low.
3. What should be our recommendation[1] for the size of "/var/tmp"(in case of separate mount point) and "/tmp".

Version-Release number of selected component (if applicable):
 - Ceph version 16.2.7-112

How reproducible:
Yes

Steps to Reproduce:
1. Install RHEL with separate /var/tmp directory and install RHCS 5.
2. Find the active mgr and fill the directory /var/tmp
3. Execute "ceph health detail".
4. Run Apache Benchmark: ab -c100 -n1000 "https://<dashboard_url>:<dashboard_port>" 
5. Access the dashboard and check the active mgr logs.

Actual results:
 - Dashboard is not accessing.

Expected results:
 - Mgr/cherrypy/code should handle the exception in case of high disk usage.  

Additional info:

[1] Documentation Improvement: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/hardware_guide/minimum-hardware-recommendations-for-containerized-ceph_hw