Bug 1142311
| Summary: | RHEL 7 base docker image, ping does not work. | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Chris Evich <cevich> | |
| Component: | iputils | Assignee: | Jan Synacek <jsynacek> | |
| Status: | CLOSED ERRATA | QA Contact: | Robin Hack <rhack> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | high | |||
| Version: | 7.0 | CC: | alsadi, cgoern, dwalsh, eparis, fulminemizzega, jeder, jgreguske, jhenner, jkrieger, jperrin, jscotka, jsynacek, jumanjiman, lfarkas, maurizio.antillon, msvistun, onagano, rhack, scollier, sct, sghosh, tiwillia | |
| Target Milestone: | rc | Keywords: | Reopened, ZStream | |
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Previously, the ping program was installed with its capabilities in the effective set as well as the permitted set. Because of the effective set and the way that docker restricts program capabilities, ping could not be used when invoked from a docker image. With this update, only the permitted capabilities set is used when installing ping, which can now be used from within a docker image as intended.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1228606 1350019 (view as bug list) | Environment: | ||
| Last Closed: | 2015-11-19 09:41:37 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1133060, 1154205, 1186007, 1211379, 1228606 | |||
this also affects the centos7 base image, even with _all_ capabilities. example: $ docker run --cap-add all --rm -it centos:centos7 bash [root@254c75830b1d /]# useradd user [root@254c75830b1d /]# su - user [user@254c75830b1d ~]$ ping -c1 google.com ping: icmp open socket: Operation not permitted what if we make ping setuid? $ cat Dockerfile FROM centos:centos7 RUN chmod u+s /usr/bin/ping RUN useradd user USER user $ docker build --rm -t ping . $ docker run --rm -it ping bash [user@f8249b2ef089 /]$ ping -c1 google.com PING google.com (216.58.216.128) 56(84) bytes of data. 64 bytes from sea15s01-in-f0.1e100.net (216.58.216.128): icmp_seq=1 ttl=52 time=6.48 ms --- google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 6.489/6.489/6.489/0.000 ms but if we drop setuid from the container, ping breaks again. (thus proving that "all" is a valid capability to add or drop) $ docker run --cap-drop all --rm -it ping bash [user@ec3248d7bd96 /]$ ping -c1 google.com ping: icmp open socket: Operation not permitted running centos:centos7 ping is already installed. I note that for some reason (aka a bug) this version of iputils does not have the file capabilities assigned to the binaries. Which is why you show the need or making them setuid. yum reinstall iputils fixes the files. something about the centos7 base image is wrong/broken. The rhel7 base image does not have ping installed by default. Installing ping does seem to install the binary with the proper file caps.... I'm not sure how to handle this bug. RHEL seems ok. CentOS base image seems broken... Regarding the RHEL7 base image, Dan's comment lead in the right direction: the container that is running ping needs a set of capabilities enabled: net_raw and net_admin. so please do verify that these steps solve the problem: 1. host# docker run --cap-add net_raw --cap-add net_admin --rm -t -i registry.access.redhat.com/rhel7 bash 2. container# yum install -y iputils 3. container# ping 127.0.0.1 PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.032 ms 64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.141 ms (In reply to Eric Paris from comment #14) > running centos:centos7 ping is already installed. I note that for some > reason (aka a bug) this version of iputils does not have the file > capabilities assigned to the binaries. Which is why you show the need or > making them setuid. > > yum reinstall iputils fixes the files. > > something about the centos7 base image is wrong/broken. > > The rhel7 base image does not have ping installed by default. Installing > ping does seem to install the binary with the proper file caps.... > > I'm not sure how to handle this bug. RHEL seems ok. CentOS base image > seems broken... i've opened https://github.com/CentOS/sig-cloud-instance-images/issues/9 for the centos7 bug fwiw `yum reinstall iputils` does not fix it for me: https://github.com/CentOS/sig-cloud-instance-images/issues/9#issuecomment-71869080 (In reply to Christoph Görn from comment #15) > 1. host# docker run --cap-add net_raw --cap-add net_admin --rm -t -i > registry.access.redhat.com/rhel7 bash > 2. container# yum install -y iputils > 3. container# ping 127.0.0.1 Confirmed for RHEL7:0-23 w/ docker-1.4.1-28.el7.x86_64 Suggestion: If this isn't going to work "out of the box" for RHEL (w/o specifying caps), I recommend documenting the need somewhere so it's easy to find. Maybe add a parenthetical "ping" example to existing --cap-add docs, so it's easy to find when searching. Alternatively, a ping look-a-like specificaly for docker than doesn't require caps would certainly be widely appreciated I'm sure. i.e. the busybox ping works just fine w/o caps. I don't think we want to document every use case which is going to blocked by security constraints, the list will get rather long. lol, no probably not, you're right. Even the list of common tools is likely rather large :( Well, here's hoping that this BZ at least gets indexed by google and saves somebody some time fighting with ping :D Ok after digging into this a little further the problem is that ping can be used for net_raw and net_admin, We install it with both capabilities and it fails on the net_admin since this is not allowed by policy. If it runs with just net_raw it works fine. setcap cap_net_raw+ep /usr/bin/ping # ping 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_seq=1 ttl=55 time=16.5 ms 64 bytes from 4.2.2.2: icmp_seq=2 ttl=55 time=15.3 ms We probably need to open a bugzilla on iputils to be able to handle this situation. The "File capabilities" and "Transformation of capabilities during execve()" sections of man (7) capabilities are probably illuminating. If you need to allow it both net_raw and admin in the file cap but only permit 1 or the other, you are going to have to remote the +ep portion of the filecap, but add code to put them into the effective set (and gracefully handle error).... (In reply to Daniel Walsh from comment #20) > Ok after digging into this a little further the problem is that ping can be > used for net_raw and net_admin, We install it with both capabilities and it > fails on the net_admin since this is not allowed by policy. If it runs with > just net_raw it works fine. It's the execve() of /usr/bin/ping in the first place that is failing # strace ping -h execve("/usr/bin/ping", ["ping", "-h"], [/* 12 vars */]) = -1 EPERM (Operation not permitted) due to it attempting to raise capabilities that are not available. Within the container gets it to work even though ping does not have cap_net_admin # setcap cap_net_raw,cap_net_admin+e /usr/bin/ping # ping 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_seq=1 ttl=55 time=17.0 ms 64 bytes from 4.2.2.2: icmp_seq=2 ttl=55 time=24.1 ms I think we need +p, not +e, so that the caps are still permitted even though they are not immediately raised. And indeed, with setcap cap_net_raw,cap_net_admin+p, things seem to work perfectly for me, both as root and as non-root within a container: caps are raised as needed and fail gracefully when not available. # mkdir -p /opt/ping # cp /usr/bin/ping /opt/ping/ # setcap cap_net_raw,cap_net_admin+p /opt/ping/ping # docker run -ti --rm -v /opt/ping:/opt/ping:z rhel7/rhel-tools bash [root@29b49db23971 /]# /opt/ping/ping -c1 10.3.1.1 PING 10.3.1.1 (10.3.1.1) 56(84) bytes of data. 64 bytes from 10.3.1.1: icmp_seq=1 ttl=62 time=0.358 ms ... and re-raising caps once dropped --- [root@29b49db23971 /]# adduser testme [root@29b49db23971 /]# su - testme [testme@29b49db23971 ~]$ /opt/ping/ping -c1 10.3.1.1 PING 10.3.1.1 (10.3.1.1) 56(84) bytes of data. 64 bytes from 10.3.1.1: icmp_seq=1 ttl=62 time=0.351 ms ... Also "-m <mark>" requires cap_net_admin, which is unavailable within the container, but this fails gracefully as desired: [root@29b49db23971 /]# /opt/ping/ping -m 123 -c1 10.3.1.1 PING 10.3.1.1 (10.3.1.1) 56(84) bytes of data. Warning: Failed to set mark 123 64 bytes from 10.3.1.1: icmp_seq=1 ttl=62 time=0.327 ms so we all now agree, the bug is in the spec file. it shouldn't include the e, only the p.
%attr(0755,root,root) %caps(cap_net_raw=ep cap_net_admin=ep) %{_bindir}/ping
This is likely true of the other binaries (ping6, arp, et al) but you'd need to make sure the code is doing a capset and failing gracefully for those. (we've proved it does it for ping....)
Be nice to grep all our spec files to see where there might be other issues lurking. (In reply to Daniel Walsh from comment #20) > Ok after digging into this a little further the problem is that ping can be > used for net_raw and net_admin, We install it with both capabilities and it > fails on the net_admin since this is not allowed by policy. What policy? Why is it not allowed? I tested ping and arping and yes, they seem to work without +e. Clockdiff doesn't though (I'm not sure how often is this utility used), and I didn't test ninfod. So, there will be some binaries that have +ep, and some that have only +p. And the *real* problem still escapes me. (In reply to Jan Synacek from comment #28) > (In reply to Daniel Walsh from comment #20) > > Ok after digging into this a little further the problem is that ping can be > > used for net_raw and net_admin, We install it with both capabilities and it > > fails on the net_admin since this is not allowed by policy. > > What policy? Why is it not allowed? Containers run as svirt_lxc_net_t by default. selinux policy restricts the capabilities allowed within that context, both to enforce isolation between containers, and between container and host. Networking for an unprivileged container is supposed to be managed by the host; we prevent containers from administering their own interfaces. > So, there will be some binaries that have +ep, and some that have only +p. > And the *real* problem still escapes me. The underlying problem is that we need to handle the capability error gracefully. If you have cap_net_admin+ep, simply execve()ing /usr/bin/ping in a container requests that cap_net_admin be raised immediately, which is not allowed. So the execve() itself gets EPERM; you simply cannot exec ping. So we want to have +p only, which lets us handle this failure gracefully. +p allows the binary to request cap_net_admin, but doesn't immediately raise it, so the execve() succeeds. And when the binary *does* request to enable the capability, then yes, that fails due to the same policy; but ping handles that failure gracefully. Thank you for the explanation! So +e should be removed from all binaries which call capset(). As there is no need for the kernel to automatically add those capabilities on execve(). The application is doing that step themselves. So if clockdiff doesn't work without +e, we'd need to check the code to see if clockdiff is using capset() correctly to add whatever it needs (cap_net_raw) into effective at runtime, instead of requiring the kernel to do it at execve() time... This is not only an SELinux issue. Docker removes capabilities net_admin from the container. Even with SELinux disabled this problem will happen. This probably blows up on an Ubunto machine also. Nice now how do we get this into RHEL7.2 Do we have to wait until this ships or can we put this into the RHEL7 base image early? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2087.html this is still not work with a 7.2 host and 7.2 guest with iputils-20121221-7.el7.x86_64 ping 4.2.2.2 ping: icmp open socket: Operation not permitted omg it seems that exactly the same version exists without this fix!!! ie after: rpm -e iputils yum install iputils everything works even if they are the same version! it's a nightmare! Hi Levente, Hmm, the patch that fixed the spec file incremented the package version from 6 to 7: https://git.centos.org/blobdiff/!!!!rpms!iputils.git/b47e8a940e38a0d23b9f0039050289d10970901d/SPECS!iputils.spec Specifically here: https://git.centos.org/blob/!!!!rpms!iputils.git/b47e8a940e38a0d23b9f0039050289d10970901d/SPECS!iputils.spec#L158 It looks fine to me, I wonder if you can try what I did to test: Pulled a CentOS 7 image from here: http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1601.qcow2 Installed/started docker, and then: # docker run -it centos ping -c 5 google.com Unable to find image 'centos:latest' locally Trying to pull repository docker.io/library/centos ... latest: Pulling from library/centos 47d44cb6f252: Pull complete 838c1c5c4f83: Pull complete 5764f0a31317: Pull complete 60e65a8e4030: Pull complete library/centos:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Digest: sha256:8072bc7c66c3d5b633c3fddfc2bf12d5b4c2623f7004d9eed6aae70e0e99fbd7 Status: Downloaded newer image for docker.io/centos:latest PING google.com (173.194.204.102) 56(84) bytes of data. 64 bytes from qb-in-f102.1e100.net (173.194.204.102): icmp_seq=1 ttl=40 time=35.0 ms 64 bytes from qb-in-f102.1e100.net (173.194.204.102): icmp_seq=2 ttl=40 time=23.6 ms 64 bytes from qb-in-f102.1e100.net (173.194.204.102): icmp_seq=3 ttl=40 time=20.9 ms 64 bytes from qb-in-f102.1e100.net (173.194.204.102): icmp_seq=4 ttl=40 time=21.2 ms 64 bytes from qb-in-f102.1e100.net (173.194.204.102): icmp_seq=5 ttl=40 time=22.1 ms --- google.com ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 20.942/24.633/35.087/5.314 ms [root@dhcp23-177 ~]# docker run -it centos rpm -q iputils iputils-20121221-7.el7.x86_64 what i did: docker pull centos:7 # = centos/centos7 = centos/latest docker run -it centos bash [root@618445c44c4f /]# rpm -qi iputils Name : iputils Version : 20121221 Release : 7.el7 Architecture: x86_64 Install Date: Wed Dec 23 18:09:24 2015 Group : System Environment/Daemons Size : 368577 License : BSD and GPLv2+ Signature : RSA/SHA256, Wed Nov 25 14:43:41 2015, Key ID 24c6a8a7f4a80eb5 Source RPM : iputils-20121221-7.el7.src.rpm Build Date : Fri Nov 20 19:12:19 2015 Build Host : worker1.bsys.centos.org Relocations : (not relocatable) Packager : CentOS BuildSystem <http://bugs.centos.org> Vendor : CentOS URL : http://www.skbuff.net/iputils Summary : Network monitoring tools including ping Description : The iputils package contains basic utilities for monitoring a network, including ping. The ping command sends a series of ICMP protocol ECHO_REQUEST packets to a specified network host to discover whether the target machine is alive and receiving network traffic. now try the same thing on a real fully updated centos7 system: # rpm -qi iputils Name : iputils Version : 20121221 Release : 7.el7 Architecture: x86_64 Install Date: Thu 17 Dec 2015 12:37:59 PM CET Group : System Environment/Daemons Size : 368577 License : BSD and GPLv2+ Signature : RSA/SHA256, Wed 25 Nov 2015 03:43:41 PM CET, Key ID 24c6a8a7f4a80eb5 Source RPM : iputils-20121221-7.el7.src.rpm Build Date : Fri 20 Nov 2015 08:12:19 PM CET Build Host : worker1.bsys.centos.org Relocations : (not relocatable) Packager : CentOS BuildSystem <http://bugs.centos.org> Vendor : CentOS URL : http://www.skbuff.net/iputils Summary : Network monitoring tools including ping Description : The iputils package contains basic utilities for monitoring a network, including ping. The ping command sends a series of ICMP protocol ECHO_REQUEST packets to a specified network host to discover whether the target machine is alive and receiving network traffic. to help compare them these are the difference: Signature : RSA/SHA256, Wed Nov 25 14:43:41 2015, Key ID 24c6a8a7f4a80eb5 Build Date : Fri Nov 20 19:12:19 2015 Signature : RSA/SHA256, Wed 25 Nov 2015 03:43:41 PM CET, Key ID 24c6a8a7f4a80eb5 Build Date : Fri 20 Nov 2015 08:12:19 PM CET These aren't differences in the package. These are your own local timezone differences. UTC -> CET conversions. ok here is a full test (what else can i say): # docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE docker.io/centos 7 60e65a8e4030 6 weeks ago 196.6 MB docker.io/centos centos7 60e65a8e4030 6 weeks ago 196.6 MB docker.io/centos latest 60e65a8e4030 6 weeks ago 196.6 MB docker.io/centos 6 3bbbf0aca359 3 months ago 190.6 MB [✔-0 lfarkas@fox:~] docker run -it 60e65a8e4030 bash [root@f35839ff0da5 /]# adduser test [root@f35839ff0da5 /]# su - test [test@f35839ff0da5 ~]$ ping google.com ping: icmp open socket: Operation not permitted [test@f35839ff0da5 ~]$ logout [root@f35839ff0da5 /]# rpm -q iputils iputils-20121221-7.el7.x86_64 [root@f35839ff0da5 /]# rpm -e iputils [root@f35839ff0da5 /]# yum install iputils Loaded plugins: fastestmirror, ovl base | 3.6 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/4): base/7/x86_64/group_gz | 155 kB 00:00:00 (2/4): extras/7/x86_64/primary_db | 90 kB 00:00:00 (3/4): updates/7/x86_64/primary_db | 2.3 MB 00:00:00 (4/4): base/7/x86_64/primary_db | 5.3 MB 00:00:01 Determining fastest mirrors * base: ftp.freepark.org * extras: ftp.freepark.org * updates: ftp.freepark.org Resolving Dependencies --> Running transaction check ---> Package iputils.x86_64 0:20121221-7.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================ Package Arch Version Repository Size ============================================================================================================================================================================ Installing: iputils x86_64 20121221-7.el7 base 152 k Transaction Summary ============================================================================================================================================================================ Install 1 Package Total download size: 152 k Installed size: 360 k Is this ok [y/d/N]: y Downloading packages: warning: /var/cache/yum/x86_64/7/base/packages/iputils-20121221-7.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY Public key for iputils-20121221-7.el7.x86_64.rpm is not installed iputils-20121221-7.el7.x86_64.rpm | 152 kB 00:00:00 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 Importing GPG key 0xF4A80EB5: Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security>" Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5 Package : centos-release-7-2.1511.el7.centos.2.10.x86_64 (@CentOS) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 Is this ok [y/N]: y Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. Installing : iputils-20121221-7.el7.x86_64 1/1 Verifying : iputils-20121221-7.el7.x86_64 1/1 Installed: iputils.x86_64 0:20121221-7.el7 Complete! [root@f35839ff0da5 /]# su - test Last login: Wed Feb 10 22:04:04 UTC 2016 on console [test@f35839ff0da5 ~]$ ping google.com PING google.com (216.58.209.174) 56(84) bytes of data. 64 bytes from bud02s21-in-f174.1e100.net (216.58.209.174): icmp_seq=1 ttl=58 time=2.07 ms 64 bytes from bud02s21-in-f174.1e100.net (216.58.209.174): icmp_seq=2 ttl=58 time=2.11 ms 64 bytes from bud02s21-in-f174.1e100.net (216.58.209.174): icmp_seq=3 ttl=58 time=2.18 ms ^C --- google.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 2.074/2.123/2.185/0.070 ms Hi Levente, thanks -- there appears to be a problem in how the base centos image was created. There's another centos image called the "centos tools image" which does not exhibit this behavior. I'll reply on the mailing list thread. https://lists.centos.org/pipermail/centos-devel/2016-February/014335.html [root@dhcp23-177 ~]# docker run -it centos rpm -V iputils ........P /usr/bin/ping ........P /usr/bin/ping6 ........P /usr/sbin/arping ........P /usr/sbin/clockdiff Those were the 4 files touched by the patch that fixed the issue in this bugzilla. [root@dhcp23-177 ~]# docker run -it centos/tools rpm -V iputils [root@dhcp23-177 ~]# And your reproducer looks better when using the tools image. root@dhcp23-177 ~]# docker run -it centos/tools bash [root@af28656bfae7 /]# adduser test [root@af28656bfae7 /]# su - test [test@af28656bfae7 ~]$ ping google.com PING google.com (216.58.217.142) 56(84) bytes of data. 64 bytes from iad23s43-in-f14.1e100.net (216.58.217.142): icmp_seq=1 ttl=52 time=12.2 ms ^C --- google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms I just encountered this issue using a fedora 24 base image. Should I file a new bug for it? There are different conditions that can lead to these symptoms. I'd suggest using the 'clone this bug' link (at the top) to make a new bug, then change all the details to match your environment. Thanks! |
Description of problem: On a RHEL 7 host (registered and subscribed), I can use Yum to install additional packages from 'docker run ...' or in a Docker file. If I install the 'iputils' package (and any dependencies), the basic 'ping' command does not work. However, if I use the 'busybox' image from the public Docker index, it's ping command works perfectly fine (on same RHEL 7 host). Version-Release number of selected component (if applicable): registry.access.redhat.com/rhel7:0-21 docker-1.2.0-13.el7.x86_64.rpm Client version: 1.2.0 Client API version: 1.15 Go version (client): go1.3.1 Git commit (client): dc45aa1/1.2.0 OS/Arch (client): linux/amd64 Server version: 1.2.0 Server API version: 1.15 Go version (server): go1.3.1 Git commit (server): dc45aa1/1.2.0 How reproducible: Trivial Steps to Reproduce: 1. docker run -i -t <image> bash 2. (in container) yum install -y iputils 3. (in container) ping 127.0.0.1 Actual results: bash: /usr/bin/ping: Operation not permitted Expected results: PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.032 ms 64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.141 ms 64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.044 ms Additional info: Daniel J Walsh wrote: > Open a bugzilla on this for docker. I believe the problem is something > to do with dropping capabilities. > > If I was a betting man I would guess > > CAP_SETPCAP is not allowed and ping is needs this access to modify its > capabilities.