Bug 1410883

Summary: httpd reload causes leak of semaphores
Product: [Fedora] Fedora Reporter: Sam Wright <sam.wright>
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 25CC: carl, jkaluza, jorton, luhliari, pahan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: httpd-2.4.25-3.fc25 httpd-2.4.25-8.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1527688 (view as bug list) Environment:
Last Closed: 2017-06-09 18:59:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sam Wright 2017-01-06 17:24:54 UTC
Description of problem:
A service reload of httpd 2.4.25 will leak exactly one semaphore on every reload.

Version-Release number of selected component (if applicable):
httpd-2.4.23-4.fc25.x86_64
httpd-2.4.25-1.fc25.x86_64

How reproducible:
100% reproducible

Steps to Reproduce:
1. Install and start a fresh copy of httpd-2.4.25
2. Count the number of semaphores used (ipcs -s | grep apache | wc -l)
3. Reload httpd a few times (for i in {1..10}; do systemctl reload apache; done)
4. Recount the number of semaphores used (ipcs -s | grep apache | wc -l)

Actual results:
[sam@fc25 ~] $ ipcs -s |grep apache
0x00000000 163840     apache     600        1         
0x00000000 196609     apache     600        1         
0x00000000 229378     apache     600        1         
0x00000000 262147     apache     600        1         
0x00000000 294916     apache     600        1         
0x00000000 327685     apache     600        1         
[sam@fc25 ~] $ for i in {1..10}; do sudo service httpd reload; done
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
[sam@fc25 ~] $ ipcs -s |grep apache
0x00000000 2129920    apache     600        1         
0x00000000 2162689    apache     600        1         
0x00000000 2195458    apache     600        1         
0x00000000 2228227    apache     600        1         
0x00000000 2260996    apache     600        1         
0x00000000 327685     apache     600        1         
0x00000000 524294     apache     600        1         
0x00000000 720903     apache     600        1         
0x00000000 917512     apache     600        1         
0x00000000 1114121    apache     600        1         
0x00000000 1310730    apache     600        1         
0x00000000 1507339    apache     600        1         
0x00000000 1703948    apache     600        1         
0x00000000 1900557    apache     600        1         
0x00000000 2097166    apache     600        1   

Expected results:
Semaphore count after reloads should remain equal.

Additional info:
A service stop will clean up all the used semaphores.

Comment 1 Carl George 2017-01-06 18:43:45 UTC
After some research, I was able to isolate this down to the proxy hcheck module.  Commenting out `LoadModule proxy_hcheck_module modules/mod_proxy_hcheck.so` in `/etc/httpd/conf.modules.d/00-proxy.conf` makes the issue goes away.  According to the changelog, that module was first enabled in 2.4.23-1.

Comment 2 Joe Orton 2017-01-09 08:19:02 UTC
Thanks for narrowing that down, Carl.  It looks like a long-standing bug in mod_watchdog in fact, but it's triggered by loading mod_proxy_hcheck.

Comment 4 Joe Orton 2017-01-20 20:32:36 UTC
Package: httpd-2.4.25-3.fc26
Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=835366

Comment 5 Joe Orton 2017-01-20 20:57:58 UTC
Package: httpd-2.4.25-3.fc25
Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=835373

Comment 6 Joe Orton 2017-01-20 21:07:23 UTC
Seems to be fixed in -3 if you want to try that.

Comment 7 Sam Wright 2017-01-20 21:18:49 UTC
Seems to work for me.

Comment 8 Fedora Update System 2017-05-26 15:56:30 UTC
httpd-2.4.25-8.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7ef5411216

Comment 9 Fedora Update System 2017-06-09 18:59:06 UTC
httpd-2.4.25-8.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.