Bug 2189983

Summary: fd leak in httpd reload
Product: Red Hat Enterprise Linux 9 Reporter: daniel.bauman
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: NEW --- QA Contact: rhel-cs-infra-services-qe <rhel-cs-infra-services-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.1CC: luhliari
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description daniel.bauman 2023-04-26 16:43:05 UTC
Description of problem:
Reload of httpd leaks a file descriptor.
Repeated reloads will eventually shut down server when file descriptor limit is reached.

Version-Release number of selected component (if applicable):
RHEL 9.1
httpd-core-2.4.53-7.el9_1.5.x86_64
httpd-2.4.53-7.el9_1.5.x86_64

How reproducible:
- Every time on RHEL 9.1 ubi image and on RHEL 9.1 VMs
- Happens with no modifications to httpd conf after install.
- Also happens if mpm conf is updated to use mpm_worker instead of mpm_event

Steps to Reproduce:
# cat /var/run/httpd/httpd.pid
35
# ls -l /proc/35/fd | wc -l
13
# systemctl reload httpd
# ls -l /proc/35/fd | wc -l
14
# systemctl reload httpd
# ls -l /proc/35/fd | wc -l
15
# while true; do systemctl reload httpd || break; ls -l /proc/35/fd | wc -l; done
16
17
...
1022
httpd.service is not active, cannot reload.
# tail /var/log/httpd/error_log
...
[Wed Apr 26 16:32:55.824207 2023] [lbmethod_heartbeat:notice] [pid 35:tid 35] AH02282: No slotmem from mod_heartmonitor
[Wed Apr 26 16:32:55.826628 2023] [core:error] [pid 35:tid 35] (24)Too many open files: AH00099: could not create /etc/httpd/run/httpd.pid.jp5DtD
[Wed Apr 26 16:32:55.826710 2023] [core:error] [pid 35:tid 35] AH00100: httpd: could not log pid to file /etc/httpd/run/httpd.pid

Actual results:
Number of open files grows with each reload. Eventually a reload brings httpd down. 

Expected results:
Number of open files does not grow with each reload. httpd stays up.

Additional info:
Increasing fd ulimit can make it take longer to crash.
Cannot reproduce on apache/bullseye httpd:2.4.53 image.