Bug 1145235 - TPROXY support is broken when upgrading from Fedora 20
Summary: TPROXY support is broken when upgrading from Fedora 20
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: squid   
(Show other bugs)
Version: 21
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Pavel Šimerda (pavlix)
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords: Regression
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-22 15:28 UTC by Trever Adams
Modified: 2015-03-31 21:56 UTC (History)
10 users (show)

Fixed In Version: squid-3.4.12-2.fc21
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-03-31 21:56:22 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

Description Trever Adams 2014-09-22 15:28:30 UTC
Description of problem:
squid: Bungled /etc/squid/squid.conf line 143: http_port 3129 tproxy
squid: 2014/09/22 09:26:00| FATAL: http(s)_port: TPROXY support in the system does not work.
squid: FATAL: Bungled /etc/squid/squid.conf line 143: http_port 3129 tproxy

This was working perfectly prior to upgrading from F20 at about 7am. I am not sure what has changed, if it is a Squid compilation problem or bug, glibc (don't think so) or kernel compilation change (possibly).

Version-Release number of selected component (if applicable):
squid-3.4.7-2.fc21.x86_64

How reproducible:
Every time

Steps to Reproduce:
http://wiki.squid-cache.org/Features/Tproxy4

Follow those directions and see that things are broken.

Actual results:
The errors under description

Expected results:
No change from perfectly functioning behavior under F20.

Comment 1 Fedora Admin XMLRPC Client 2014-09-30 12:13:04 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 2 Bojan Smojver 2014-11-16 11:04:21 UTC
So, is this still a problem in F21?

Comment 3 Trever Adams 2014-11-17 14:29:45 UTC
I don't know. I had to downgrade because gcc/glibc in F21 breaks running and compilation of FreeSWITCH. I have most everything else running F21, but the machine that did TPROXY/Squid could not stay at F21.

Comment 4 Bojan Smojver 2014-11-18 03:19:39 UTC
I looked at build logs:

https://kojipkgs.fedoraproject.org//packages/squid/3.4.7/2.fc21/data/logs/x86_64/build.log

Somehow, ip6_tables.h was not found (or not usable, I guess). That is part of kernel-headers on my F-20 machine. Now, whether that could have thrown the build off tracks, I don't know.

Let me check the kernel packages for F-21 to see whether they are different... No, same thing - 3.17.3 for F-21 has that header in the same package. This version of squid was build against 3.16.1-301.fc21 kernel which does have that header. Weird... Probably irrelevant anyway.

Anyhow, maybe the package maintainer can bump up to the most recent squid (3.4.9) and rebuild.

Pavel?

Comment 5 Henrik Nordström 2014-11-18 16:56:25 UTC
3.4.9 is being built in rawhide now. Please give it a try when ready.

Comment 6 Bojan Smojver 2014-11-24 07:11:44 UTC
(In reply to Henrik Nordström from comment #5)
> 3.4.9 is being built in rawhide now. Please give it a try when ready.

Installed this package on F-21 (i686). This is what I got in logs:
----------------
Nov 24 18:00:48 <host> squid[3838]: 2014/11/24 18:00:48| FATAL: http(s)_port: TPROXY support in the system does not work.
Nov 24 18:00:48 <host> squid[3838]: Bungled /etc/squid/squid.conf line 59: http_port 3128 tproxy
----------------

Config line (the rest is default):
----------------
http_port 3128 tproxy
----------------

So, no dice. Still broken.

However, if I run squid -d 10 -N, I get:
----------------
2014/11/24 18:06:23| Accepting TPROXY intercepted HTTP Socket connections at local=[::]:3128 remote=[::] FD 10 flags=25
----------------

So, this appears to be some kind of systemd problem (i.e. when started as a service, it won't start). Not sure what yet...

Comment 7 Bojan Smojver 2014-11-24 07:29:02 UTC
Maybe it is the fact that new service file wants to run squid as non-root. I will try that next.

Comment 8 Bojan Smojver 2014-11-24 07:46:19 UTC
(In reply to Bojan Smojver from comment #7)
> Maybe it is the fact that new service file wants to run squid as non-root. I
> will try that next.

Yeah, that's it. Commenting User and Group lines in service file makes it run again.

Comment 10 Henrik Nordström 2014-11-24 22:10:42 UTC
In general it's a good change to not run Squid as root.

If I am not mistaken TPROXY requires CAP_NETADMIN, but needs to be verified.

Does it help to add

CapabilityBoundingSet=CAP_NET_ADMIN,CAP_NET_BIND_SERVICE,CAP_NET_BROADCAST,CAP_NET_RAW

to the systemd service definition?


This too is a little too broad for Squid in general, but may be required in different configurations

  - TPROXY -> CAP_NET_ADMIN, to bind to any address
  - Acceleration -> CAP_NET_BIND_SERVICE to bind to port 80/443
  - Multicast HTCP/ICP -> CAP_NET_BROADCAST, to accept Multicast
  - Pinger -> CAP_NET_RAW (i think), to send ICMP.

Comment 11 Bojan Smojver 2014-11-24 22:25:31 UTC
(In reply to Henrik Nordström from comment #10)
> In general it's a good change to not run Squid as root.

Squid will switch to squid user on it's listeing socket anyway, so only the lead process will run as root. But, I'll check about capabilities.

Comment 12 Bojan Smojver 2014-11-24 22:30:32 UTC
(In reply to Henrik Nordström from comment #10)
 
> Does it help to add
> 
> CapabilityBoundingSet=CAP_NET_ADMIN,CAP_NET_BIND_SERVICE,CAP_NET_BROADCAST,
> CAP_NET_RAW

Tried with commas and spaces, as per http://0pointer.de/blog/projects/security.html. No dice.

Comment 13 Henrik Nordström 2014-11-24 22:32:06 UTC
No, Squid only fully drops root privileges if you also use chroot_dir.

Note: Maybe CapabilitiesBoundingSet do not work here and you need to use Capabilities option instead, or combination of CapabilitiesBoundingSet and setting the needed capabilities as ei on the squid executable, not 100% sure how Systemd & capabilities interact. A quick google gave this systemd discussion on the topic: http://comments.gmane.org/gmane.comp.sysutils.systemd.devel/19344

Comment 14 Bojan Smojver 2014-11-24 22:50:14 UTC
This works, without changes to service file:

setcap cap_net_admin,cap_net_bind_service,cap_net_broadcast,cap_net_raw+ep /usr/sbin/squid

So, I guess you need to do that when you're building the package.

Comment 15 Bojan Smojver 2014-11-24 22:52:34 UTC
(In reply to Henrik Nordström from comment #13)

> Note: Maybe CapabilitiesBoundingSet do not work here and you need to use
> Capabilities option instead, or combination of CapabilitiesBoundingSet and
> setting the needed capabilities as ei on the squid executable, not 100% sure
> how Systemd & capabilities interact. A quick google gave this systemd
> discussion on the topic:
> http://comments.gmane.org/gmane.comp.sysutils.systemd.devel/19344

BTW, did not work. Not without changing capabilities of squid binary anyway.

Comment 16 Bojan Smojver 2014-11-25 05:57:09 UTC
Note, also getting this when in use:
-----------
type=AVC msg=audit(1416894906.383:473): avc:  denied  { net_raw } for  pid=883 comm="squid" capability=13  scontext=system_u:system_r:squid_t:s0 tcontext=system_u:system_r:squid_t:s0 tclass=capability permissive=1
-----------

Ergo, requires:
-----------
require {
	type squid_t;
	class capability net_raw;
}

#============= squid_t ==============
allow squid_t self:capability net_raw;
-----------

Comment 17 Henrik Nordström 2014-11-25 20:11:19 UTC
From what I understand for security the binary should NOT have p capability set, those should be set by systemd. But it likely needs e and maybe i sets defined. systemd setting alloved capability boundaries (p), binary defining what it wants to have (e for Squid itself, and i for allowing inheriting to pinger etc).

Binaries with p capability set is similar to setuid in terms security, and may allow local user privilege escalation.

Comment 18 Bojan Smojver 2014-11-25 20:20:02 UTC
(In reply to Henrik Nordström from comment #17)
> From what I understand for security the binary should NOT have p capability
> set, those should be set by systemd. But it likely needs e and maybe i sets
> defined. systemd setting alloved capability boundaries (p), binary defining
> what it wants to have (e for Squid itself, and i for allowing inheriting to
> pinger etc).
> 
> Binaries with p capability set is similar to setuid in terms security, and
> may allow local user privilege escalation.

Honestly, I just cut/pasted an example off the 'net, which made my squid work. Of course, you can work out how you want to fix this. It is your package. I'm reporting what made it run for me.

Comment 19 Bojan Smojver 2014-11-25 20:31:42 UTC
Just for reference, doing 'e' does not work. Neither does 'ei'.

P, according to the manual, stands for permitted and it is a limiting superset above which the thread cannot assume capabilities. It can drop some from that set, which is irreversible.

Comment 20 Henrik Nordström 2014-11-25 20:40:23 UTC
Giving e without p on the binary requires the systemd service to properly setup the p part, which from what I understand is done by using CapabilityBoundingSet= in the systemd service definition.

when done this way the binary can only get the capabilities when executed from a context where those capabilities are explicit allowed, not any other context such as a normal (unprivileged) user shell.

Comment 21 Bojan Smojver 2014-11-25 20:45:12 UTC
(In reply to Henrik Nordström from comment #20)
> Giving e without p on the binary requires the systemd service to properly
> setup the p part, which from what I understand is done by using
> CapabilityBoundingSet= in the systemd service definition.
> 
> when done this way the binary can only get the capabilities when executed
> from a context where those capabilities are explicit allowed, not any other
> context such as a normal (unprivileged) user shell.

Just tried that. Did not work.

Comment 22 Henrik Nordström 2014-11-26 08:43:58 UTC
Ok, looks like we need to bring someone more familiar with systemd to help with this.

Comment 23 Bojan Smojver 2014-11-26 23:07:53 UTC
(In reply to Henrik Nordström from comment #20)
> Giving e without p on the binary requires the systemd service to properly
> setup the p part, which from what I understand is done by using
> CapabilityBoundingSet= in the systemd service definition.

Giving the binary on disk e does nothing (i.e. getcap returns empty set). The e, according to man 7 capabilities, is there (as a bit) to enable the p set on execve().

As for CapabilityBoundingSet, it cannot give any more than there is. This setting can only reduce and existing set of capabilities.

Comment 24 Bojan Smojver 2014-12-02 20:45:36 UTC
I guess another way to fix this is to revert to running squid as root (which should give it all capabilities), but then specify CapabilityBoundingSet in the service file, which should then clip the full set to just what is required/desired.

Comment 25 Bojan Smojver 2014-12-11 21:32:33 UTC
Can we have a decision and a fix here?

I'd personally go with my last suggestion, because no special capabilities would be given to squid binary on disk unless it is run as a systemd service (i.e. as root). And, the set of those capabilities would be limited to exactly what is required and no more.

Comment 26 Bojan Smojver 2014-12-21 21:19:11 UTC
This worked for me:
------------------
[Unit]
Description=Squid caching proxy
After=syslog.target network.target nss-lookup.target

[Service]
Type=simple
LimitNOFILE=16384
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW CAP_DAC_READ_SEARCH CAP_SETUID CAP_SETGID CAP_DAC_OVERRIDE
EnvironmentFile=/etc/sysconfig/squid
ExecStartPre=/usr/libexec/squid/cache_swap.sh
ExecStart=/usr/sbin/squid $SQUID_OPTS -N -f $SQUID_CONF
ExecReload=/usr/sbin/squid $SQUID_OPTS -k reconfigure -f $SQUID_CONF
ExecStop=/usr/sbin/squid -k shutdown -f $SQUID_CONF
TimeoutSec=0

[Install]
WantedBy=multi-user.target
------------------

Note the -N option and type simple there (i.e. not forking any more). This is in relation to proposal in bug #1176318.

It requires this as well in /etc/tmpfiles.d/squid.conf:
------------------
d /run/squid 0755 squid squid -
------------------

Explanation of capabilities in the set beyond what you specified:

- CAP_DAC_READ_SEARCH: for /usr/libexec/squid/cache_swap.sh to work
- CAP_SETUID CAP_SETGID: to switch to squid:squid UID/GID when running
- CAP_DAC_OVERRIDE: to create /run/squid/squid.pid file

Without those, I haven't had much luck starting squid with capabilities limited to a particular set.

Comment 27 Marcos Mello 2015-03-02 17:09:48 UTC
(In reply to Bojan Smojver from comment #25)
> Can we have a decision and a fix here?
> 
> I'd personally go with my last suggestion, because no special capabilities
> would be given to squid binary on disk unless it is run as a systemd service
> (i.e. as root). And, the set of those capabilities would be limited to
> exactly what is required and no more.

Please. It is pretty sad squid being broken out-of-box in F21 for such a time as this.

Regarding Type=simple (implied default if not specified, btw), if we go this way, can also cleanup other options a bit.

ExecReload can be:

ExecReload=/usr/bin/kill -HUP $MAINPID

ExecStop can be droped (systemd sends SIGTERM by default) and KillMode=process added to make it signal only the main process and leave squid's kids alone.

The "-f $SQUID_CONF" is unnecessary, no? squid has it hardcoded to /etc/squid/squid.conf anyway.

[Service]
Type=simple
LimitNOFILE=16384
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW CAP_DAC_READ_SEARCH CAP_SETUID CAP_SETGID CAP_DAC_OVERRIDE
EnvironmentFile=/etc/sysconfig/squid
ExecStartPre=/usr/libexec/squid/cache_swap.sh
ExecStart=/usr/sbin/squid -N $SQUID_OPTS
ExecReload=/usr/bin/kill -HUP $MAINPID
KillMode=process
TimeoutSec=0

Very close to the service file is being distributed by upstream squid in version 3.5.0.4 and up:

http://bazaar.launchpad.net/~squid/squid/trunk/view/head:/tools/systemd/squid.service


And then cleanup /etc/sysconfig/squid: remove SQUID_SHUTDOWN_TIMEOUT (SysV leftover...) and SQUID_CONF.

Comment 28 Marcos Mello 2015-03-02 20:00:24 UTC
(In reply to Marcos Mello from comment #27)
> Regarding Type=simple (implied default if not specified, btw), if we go this
> way...
> 

Unfortunately squid -N + Type=simple breaks SMP mode. Until it is fixed we are stuck with Type=forking.

http://pkgs.fedoraproject.org/cgit/squid.git/commit/squid.service?id=431df598c689b970cd700ddcab72dd1835a6db52

https://bugzilla.redhat.com/show_bug.cgi?id=1038160 (can RH open this bug to public?)

Comment 29 Marcos Mello 2015-03-20 16:11:57 UTC
(In reply to Bojan Smojver from comment #9)
> This commit should be reverted:
> 
> http://pkgs.fedoraproject.org/cgit/squid.git/commit/squid.
> service?id=1cf2c264ec5d92e1e5b45a21f6f67d03a4856646

Dear Squid maintainers, can you please revert this commit until a better solution exists?

Squid is *broken* the way it is now! For months!

Comment 30 Pavel Šimerda (pavlix) 2015-03-23 13:52:24 UTC
How does the reference commit relate to TPROXY support?

Comment 31 Marcos Mello 2015-03-23 14:29:24 UTC
(In reply to Pavel Šimerda (pavlix) from comment #30)
> How does the reference commit relate to TPROXY support?

Have you read the bug report? Squid needs root for TPROXY.

As already discussed, we can reduce capability set using systemd's CapabilityBoundingSet (see #c26), but first let's have a working Squid again at least and then evaluate the cap stuff.

Comment 32 Pavel Šimerda (pavlix) 2015-03-23 18:01:40 UTC
> (In reply to Pavel Šimerda (pavlix) from comment #30)
> > How does the reference commit relate to TPROXY support?
> 
> Have you read the bug report?

How constructive. See below.

(In reply to Marcos Mello from comment #28)
> (In reply to Marcos Mello from comment #27)
> > Regarding Type=simple (implied default if not specified, btw), if we go this
> > way...
> > 
> 
> Unfortunately squid -N + Type=simple breaks SMP mode. Until it is fixed we
> are stuck with Type=forking.
> 
> http://pkgs.fedoraproject.org/cgit/squid.git/commit/squid.
> service?id=431df598c689b970cd700ddcab72dd1835a6db52
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1038160 (can RH open this bug to
> public?)

I actually meant the above commit (the latest in the discussion). We tried Type=simple in the past but stepped back because of the squid's multiprocessing model. I can confirm that we're not changing this.

Will issue an update without User/Group shortly.

Comment 33 Marcos Mello 2015-03-23 18:54:20 UTC
Thank you. While you are there, remove "syslog.target" from "After=". It is not needed anymore since systemd 202. We only support F20 (208) an up, so no problem -- and RHEL 7 has 208 too.

http://thread.gmane.org/gmane.linux.suse.opensuse.packaging/11792

Comment 34 Bojan Smojver 2015-03-23 21:05:52 UTC
I see you reverted squid's PID in /run/squid directory as well. Isn't the fix to actually provide /etc/tmpfiles.d/squid.conf, like in comment #26?

Never mind the simple thing, that's no good. I can tell you from personal experience of a few months now that CapabilityBoundingSet works when squid is run as root (I run it in TPROXY mode).

Comment 35 Fedora Update System 2015-03-24 09:39:04 UTC
squid-3.4.12-2.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/squid-3.4.12-2.fc21

Comment 36 Pavel Šimerda (pavlix) 2015-03-24 09:50:15 UTC
(In reply to Marcos Mello from comment #33)
> Thank you. While you are there, remove "syslog.target" from "After=". It is
> not needed anymore since systemd 202. We only support F20 (208) an up, so no
> problem -- and RHEL 7 has 208 too.
> 
> http://thread.gmane.org/gmane.linux.suse.opensuse.packaging/11792

Right, syslog.target is definitely redundant. What about network.target? That one doesn't guarantee anything at all in general. While with /etc/init.d/network it works like network-online.target, with NetworkManager it's effectively useless.

(In reply to Bojan Smojver from comment #34)
> I see you reverted squid's PID in /run/squid directory as well. Isn't the
> fix to actually provide /etc/tmpfiles.d/squid.conf, like in comment #26?

Do we need the directory when we don't run as a user?

Comment 37 Bojan Smojver 2015-03-24 10:44:13 UTC
(In reply to Pavel Šimerda (pavlix) from comment #36)

> Do we need the directory when we don't run as a user?

Yeah, good point. Not sure whether limited capabilities will affect this - probably not. So, I guess leave it as is.

Comment 38 Marcos Mello 2015-03-24 11:14:26 UTC
(In reply to Pavel Šimerda (pavlix) from comment #36)
> (In reply to Marcos Mello from comment #33)
> > Thank you. While you are there, remove "syslog.target" from "After=". It is
> > not needed anymore since systemd 202. We only support F20 (208) an up, so no
> > problem -- and RHEL 7 has 208 too.
> > 
> > http://thread.gmane.org/gmane.linux.suse.opensuse.packaging/11792
> 
> Right, syslog.target is definitely redundant. What about network.target?
> That one doesn't guarantee anything at all in general. While with
> /etc/init.d/network it works like network-online.target, with NetworkManager
> it's effectively useless.
> 

systemd documentation says it is useful on shutdown.

Comment 39 Pavel Šimerda (pavlix) 2015-03-24 11:24:22 UTC
(In reply to Marcos Mello from comment #38)
> systemd documentation says it is useful on shutdown.

True.

Comment 40 Fedora Update System 2015-03-26 22:08:59 UTC
Package squid-3.4.12-2.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing squid-3.4.12-2.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-4739/squid-3.4.12-2.fc21
then log in and leave karma (feedback).

Comment 41 Marcos Mello 2015-03-30 15:14:25 UTC
There is a problem with the update:

https://admin.fedoraproject.org/updates/FEDORA-2015-4739/squid-3.4.12-2.fc21

(lack of a F22 build?)

Comment 42 Pavel Šimerda (pavlix) 2015-03-31 18:34:44 UTC
(In reply to Marcos Mello from comment #41)
> There is a problem with the update:
> 
> https://admin.fedoraproject.org/updates/FEDORA-2015-4739/squid-3.4.12-2.fc21
> 
> (lack of a F22 build?)

Fedora updates are AFAIK done one branch at a time. But you're right that a F22 update would be good to have.

Comment 43 Fedora Update System 2015-03-31 21:56:22 UTC
squid-3.4.12-2.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.