Bug 970136

Summary: No space left on device error after several restarts
Product: Red Hat Enterprise Linux 7 Reporter: Lukas "krteknet" Novy <lnovy>
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED NOTABUG QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: ebenes, jorton, lnovy
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-03 15:39:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lukas "krteknet" Novy 2013-06-03 14:07:20 UTC 
Description of problem:
Restarting httpd several times in VM leads to No space left on device message in error log and httpd refuses to start

Version-Release number of selected component (if applicable):
httpd-2.4.4-5.el7

How reproducible:
Always (after couple of tries)

Steps to Reproduce:
1. Schedule /CoreOs/httpd/Sanity/bz874479-Support-SSLCompression-on-in several times in one recipeset
2. Watch it fails after about 4th try

Actual results:
error_log reads:
[Mon Jun 03 15:36:30.127818 2013] [core:notice] [pid 10499] SELinux policy enabled; httpd running as context unconfined_u:system_r:unconfined
[Mon Jun 03 15:36:30.128428 2013] [suexec:notice] [pid 10499] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jun 03 15:36:31.001252 2013] [ssl:notice] [pid 10499] AH01886: SSL FIPS mode disabled
[Mon Jun 03 15:36:31.057056 2013] [auth_digest:notice] [pid 10500] AH01757: generating secret for digest authentication ...
[Mon Jun 03 15:36:31.057517 2013] [core:emerg] [pid 10500] (28)No space left on device: AH00023: Couldn't create the authdigest-opaque mutex
[Mon Jun 03 15:36:31.057551 2013] [auth_digest:error] [pid 10500] (28)No space left on device: AH01760: failed to create lock (opaque_lock) -
[Mon Jun 03 15:36:31.057620 2013] [:emerg] [pid 10500] AH00020: Configuration Failed, exiting


Expected results:
httpd runs

Additional info:
Simple reboot fixes issue.
There's plenty of space on all volumes.
I'm investigating now if I can reproduce this more simply.
Comment 1 Lukas "krteknet" Novy 2013-06-03 14:11:22 UTC 
Simple

# while true; do killall -9 httpd; httpd; sleep 1; done

can cause this.
Comment 3 Joe Orton 2013-06-03 14:26:24 UTC 
Terminating the httpd parent with SIGKILL is expected to have this effect, SysV semaphores and shm segments cannot be cleaned up reliably if the parent is terminated "with extreme prejudice"...

However this should not be seen during normal operation.  We have been adjusting systemd shutdown handling so this could be a side-effect.
Comment 7 Joe Orton 2013-06-03 14:50:33 UTC 
Put the env vars in /etc/sysconfig/httpd
Comment 8 Lukas "krteknet" Novy 2013-06-03 15:30:59 UTC 
If this is intended behaviour, feel free to close this bug, my issue is solved.
Comment 9 Joe Orton 2013-06-03 15:39:44 UTC 
Thanks Lukas.  Yes, done.