Bug 1176318

Summary: "systemctl reload squid.service" fails due to missing PID file directory
Product: [Fedora] Fedora Reporter: Michael Chapman <redhat-bugzilla>
Component: squidAssignee: Pavel Šimerda (pavlix) <psimerda>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: bojan, frank, henrik, jonathansteffan, kevin, marcosfrm, mluscon, paul, psimerda, redhat-bugzilla, steve.traylen, thozza, trevor
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: squid-3.4.12-2.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-31 21:56:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michael Chapman 2014-12-20 12:08:44 UTC
This problem was encountered with squid-3.4.7-2.fc21.x86_64.

The ExecReload action in squid.service fails:

# systemctl reload squid.service 
Job for squid.service failed. See "systemctl status squid.service" and "journalctl -xe" for details.

There's nothing useful in the journal, but Squid's cache.log indicates that the problem is that the PID file does not exist:

2014/12/20 22:38:41 kid1| /var/run/squid/squid.pid: (2) No such file or directory
2014/12/20 22:38:41 kid1| WARNING: Could not write pid file

This in turn is because the /var/run/squid/ directory doesn't exist.

One simple solution is to create this directory with a tmpfiles.d fragment, e.g.:

d /run/squid 0755 squid squid -

The PID written in that file is *not* Squid's parent process -- it's actually the first child of that parent process -- so it can't be used in a MainPID directive in squid.service. This does not seem to cause any problems though.

To make this more robust, however, I would recommend that Squid be run in "no daemon" mode. This way systemd can simply signal Squid directly. I am using the following changes to the unit file:

--- /usr/lib/systemd/system/squid.service	2014-09-12 20:58:56.000000000 +1000
+++ /etc/systemd/system/squid.service	2014-12-20 22:52:29.579380834 +1100
@@ -3,16 +3,14 @@
 After=syslog.target network.target nss-lookup.target
 
 [Service]
-Type=forking
+Type=simple
 User=squid
 Group=squid
 LimitNOFILE=16384
 EnvironmentFile=/etc/sysconfig/squid
 ExecStartPre=/usr/libexec/squid/cache_swap.sh
-ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF
-ExecReload=/usr/sbin/squid $SQUID_OPTS -k reconfigure -f $SQUID_CONF
-ExecStop=/usr/sbin/squid -k shutdown -f $SQUID_CONF
-TimeoutSec=0
+ExecStart=/usr/sbin/squid $SQUID_OPTS -N -f $SQUID_CONF
+ExecReload=/bin/kill -HUP $MAINPID
 
 [Install]
 WantedBy=multi-user.target

Note that Squid still writes out a PID file, even in no-daemon mode, which means manual "squid -k <command>" invocations by the sysadmin still work. So whether or not the systemd unit is changed, the tmpfiles.d fragment is still useful.

Comment 1 Bojan Smojver 2014-12-21 19:53:52 UTC
This also interferes with log rotation. Squid logging stops, because logrotate moves the log file from under squid and squid process is never properly signalled.

Comment 2 Bojan Smojver 2014-12-21 21:20:03 UTC
See https://bugzilla.redhat.com/show_bug.cgi?id=1145235#c26 for something similar, but that also support TPROXY mode of operation.

Comment 3 Steve Traylen 2015-01-05 15:36:59 UTC
The tmpfile.d example above should be

d /var/run/squid 0755 squid squid -

This is my preferred solution since it's the smallest change but also the
multi-cpu and ${process_number} configuration does not work.

Steve.

Comment 4 Steve Traylen 2015-01-05 16:01:45 UTC
(In reply to Steve Traylen from comment #3)
> The tmpfile.d example above should be
> 
> d /var/run/squid 0755 squid squid -
> 
> This is my preferred solution since it's the smallest change but also the
> multi-cpu and ${process_number} configuration does not work.
> 
> Steve.

Ignore this last comment, I now realize that /var/run -> /run.

Comment 5 Pavel Šimerda (pavlix) 2015-01-05 16:04:18 UTC
(In reply to Michael Chapman from comment #0)
>  [Service]
> -Type=forking
> +Type=simple
>  User=squid
>  Group=squid
>  LimitNOFILE=16384
>  EnvironmentFile=/etc/sysconfig/squid
>  ExecStartPre=/usr/libexec/squid/cache_swap.sh
> -ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF
> -ExecReload=/usr/sbin/squid $SQUID_OPTS -k reconfigure -f $SQUID_CONF
> -ExecStop=/usr/sbin/squid -k shutdown -f $SQUID_CONF
> -TimeoutSec=0
> +ExecStart=/usr/sbin/squid $SQUID_OPTS -N -f $SQUID_CONF
> +ExecReload=/bin/kill -HUP $MAINPID

We already switched once to no-daemon mode and had to switch back because it breaks the multi-cpu setup.

Comment 6 Pavel Šimerda (pavlix) 2015-01-05 16:10:30 UTC
(In reply to Michael Chapman from comment #0)
> One simple solution is to create this directory with a tmpfiles.d fragment,
> e.g.:
> 
> d /run/squid 0755 squid squid -

This looks good. So if we avoid the `-N` option  to squid to keep all the features, do we need any changes to the service file?

Comment 7 Bojan Smojver 2015-01-05 20:22:52 UTC
(In reply to Pavel Šimerda (pavlix) from comment #6)
> (In reply to Michael Chapman from comment #0)
> > One simple solution is to create this directory with a tmpfiles.d fragment,
> > e.g.:
> > 
> > d /run/squid 0755 squid squid -
> 
> This looks good. So if we avoid the `-N` option  to squid to keep all the
> features, do we need any changes to the service file?

Yes, for the other bug. See:

https://bugzilla.redhat.com/show_bug.cgi?id=1145235#c26

Essentially, running squid under squid account breaks tproxy unless capabilities are given to squid binary. So, it should be run as root, but the capabilities should be dropped to what I mentioned in the other bug.

Comment 8 Bojan Smojver 2015-01-05 20:31:36 UTC
After dropping -N option, I have:
-------------
[Unit]
Description=Squid caching proxy
After=syslog.target network.target nss-lookup.target

[Service]
Type=forking
LimitNOFILE=16384
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW CAP_DAC_READ_SEARCH CAP_SETUID CAP_SETGID CAP_DAC_OVERRIDE
EnvironmentFile=/etc/sysconfig/squid
ExecStartPre=/usr/libexec/squid/cache_swap.sh
ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF
ExecReload=/usr/sbin/squid $SQUID_OPTS -k reconfigure -f $SQUID_CONF
ExecStop=/usr/sbin/squid -k shutdown -f $SQUID_CONF
TimeoutSec=0

[Install]
WantedBy=multi-user.target
-------------

Comment 9 Michael Chapman 2015-01-06 14:42:02 UTC
Yes, that config should work well enough.

It's a pity that Squid does a double-fork, putting the *child* process's PID in the PID file. That's the process that needs to receive various signals to reload config, rotate logs, etc. Without a PIDFile directive, systemd assumes the top process is the process' "main PID"... but sending any signal to that top process (with, say, "systemctl kill") just causes Squid to complete exit. With a PIDFile directive, on the other hand, systemd rightly warns that it will "likely not notice when it exits".

But that is a completely different Squid bug, and it can be worked around by using "squid -k <command>" instead.

The tmpfiles.d snippet is sufficient to resolve this bug report.

Comment 10 Pavel Šimerda (pavlix) 2015-01-06 14:56:21 UTC
(In reply to Michael Chapman from comment #9)
> The tmpfiles.d snippet is sufficient to resolve this bug report.

Good. We're not going to do any magic in Fedora anyway, it's best to report anything upstream unless it's somehow specific to Fedora or it's an urgent fix.

Comment 11 Bojan Smojver 2015-01-06 19:59:35 UTC
(In reply to Pavel Šimerda (pavlix) from comment #10)
> (In reply to Michael Chapman from comment #9)
> > The tmpfiles.d snippet is sufficient to resolve this bug report.
> 
> Good. We're not going to do any magic in Fedora anyway, it's best to report
> anything upstream unless it's somehow specific to Fedora or it's an urgent
> fix.

Could you also fix bug #1145235? This has been broken since the release of F-21. The service file I posted here is the fix for it.

Comment 12 Marcos Mello 2015-02-05 12:47:07 UTC
Also logrotate postrotate script is not working because this. Please push a fix soon.

Comment 13 Marcos Mello 2015-02-05 18:43:51 UTC
(In reply to Michael Chapman from comment #9)
> 
> It's a pity that Squid does a double-fork, putting the *child* process's PID
> in the PID file. That's the process that needs to receive various signals to
> reload config, rotate logs, etc. Without a PIDFile directive, systemd
> assumes the top process is the process' "main PID"... but sending any signal
> to that top process (with, say, "systemctl kill") just causes Squid to
> complete exit. With a PIDFile directive, on the other hand, systemd rightly
> warns that it will "likely not notice when it exits".
> 
> But that is a completely different Squid bug, and it can be worked around by
> using "squid -k <command>" instead.

https://bugzilla.redhat.com/show_bug.cgi?id=913262
http://bugs.squid-cache.org/show_bug.cgi?id=3826

Comment 14 Kevin Fenzi 2015-02-26 21:42:17 UTC
This also makes things nicely break (silently) when you try and enable workers (smp). It will happily start your workers, but since it puts the .ipc sockets in /var/run/squid/ and that doesn't exist, they cannot communicate and never end up listening on the socket. ;(

Comment 15 Pavel Šimerda (pavlix) 2015-03-23 18:15:36 UTC
No such file or directory doesn't always mean the problem is in the absance of the actual file, hence the change of summary.

Comment 16 Fedora Update System 2015-03-24 09:39:08 UTC
squid-3.4.12-2.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/squid-3.4.12-2.fc21

Comment 17 Fedora Update System 2015-03-26 22:09:05 UTC
Package squid-3.4.12-2.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing squid-3.4.12-2.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-4739/squid-3.4.12-2.fc21
then log in and leave karma (feedback).

Comment 18 Fedora Update System 2015-03-31 21:56:26 UTC
squid-3.4.12-2.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.