Bug 855111
| Summary: | Cache rebuild goes wild and fills disk | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Petr Tuma <petr.tuma> |
| Component: | squid | Assignee: | Michal Luscon <mluscon> |
| Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rawhide | CC: | henrik, jonathansteffan, mluscon, stadtkind2, thozza |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-06-27 11:18:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Petr Tuma
2012-09-06 18:02:55 UTC
This is known to happen when swap.state gets corrupted for some reason. But swap.state corruption is not easily seen unless the machine as such crashes, so this is a quite unexpected. I've scanned the logs several months back and see no unclean shutdown. Is it possible to manually detect whether swap.state is corrupted, so that I can perhaps track down when the corruption happens (sooner than when I see the disk getting filled to maximum capacity) ? I see the same thing with F17, systemd doesn't seem to honor squids *long* shutdown time (up to 30 seconds) and after every reboot I get warnings like: 2012/09/30 01:57:51 kid1| WARNING: Disk space over limit: 5278968.00 KB > 209715 2 KB in /var/log/squid/cache.log. Stopping squid, removing /var/spool/squid/swap.state and then restarting squid again helped but that's just a workaround. I confirm the observation above. Another pass through the logs suggests that while clean shutdown was always initiated, it sometimes did not run to completion (the log says "Preparing for shutdown" but other common shutdown messages do not follow). What is strange is that default systemd shutdown timeout for the squid service should be 90 seconds, well above the 30 seconds mentioned above. I'm now trying with "TimeoutSec=0" added to /lib/systemd/system/squid.service to see if that helps. For a large Squid cache 90 seconds may not be enough. The 30 second mentioned is how log Squid waits for existing connections to finish before it shuts down for real. After that is the actual shutdown procedure which can take quite some time if either the cache is large or your system have swapped out significant parts of Squid. Summary of the problem: Squid shutdown easily takes more than one minute, even if it is not particularly active. Systemd does not wait for Squid to shut down properly. When killed, Squid can leave its cache files in an inconsistent state, which cause it to go wild and use up all disk space on some of the next starts. Current solution that seems to work for me: Wait for proper Squid shutdown in Systemd. Change the ExecStop line and add the next two lines in /lib/systemd/system/squid.service: ExecStop=/bin/sh -c "/usr/sbin/squid -k shutdown && while kill -0 $MAINPID ; do sleep 1 ; done" PIDFile=/run/squid.pid TimeoutSec=300 What I do not like on the current solution: The active wait for the Squid process to disappear. I could not find any better way to do this in Systemd, but I could easily be missing something. Squid itself apparently does not offer any way of shutting down synchronously (if it did, it could simply be used in ExecStop instead of the asynchronous shutdown command). Finally, there is a SQUID_SHUTDOWN_TIMEOUT variable in /etc/sysconfig/squid, which is apparently not used anywhere. Perhaps we could do something like TimeoutSec=$SQUID_SHUTDOWN_TIMEOUT in the Systemd service file, but I'm not sure if Systemd expands shell variables in other than the Exec lines. This bug is still present in the current Squid package in Fedora 18 (ExecStop command in /lib/systemd/system/squid.service kills Squid rather than shutting it down properly). This message is a reminder that Fedora 18 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 18. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '18'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 18's end of life. Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 18 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior to Fedora 18's end of life. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 18 changed to end-of-life (EOL) status on 2014-01-14. Fedora 18 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. Resolved in the current rawhide version of squid -> timeout option is now set to zero(unlimited) as it can't be predicted in advance. |