Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionPavel Tikhomirov
2015-07-16 13:30:15 UTC
Description of problem:
Docker inter-container communication stopped working after updating docker from docker 1.6.0 to 1.6.2. Iptables-services package prevents docker to properly configure ICC iptables rules with its default REJECT rule.
Version-Release number of selected component (if applicable):
docker-1.6.2-14.el7.centos.x86_64
iptables-services-1.4.21-13.el7.x86_64
docker-1.6.0-11.0.1.el7.centos.x86_64 - for comparison
How reproducible:
100%
Steps to Reproduce:
1. yum install docker -y
2. systemctl start docker
3. docker run -d --name test ubuntu sleep 1000
4. ip=$(docker inspect -f '{{.NetworkSettings.IPAddress}}' test)
5. docker run -it ubuntu ping -c 1 $ip
6. iptables -L -v
Actual results:
docker run -it ubuntu ping -c 1 $ip
PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data.
> From 172.17.42.1 icmp_seq=1 Destination Host Prohibited
--- 172.17.0.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
iptables -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
2 168 ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED
0 0 ACCEPT icmp -- any any anywhere anywhere
0 0 ACCEPT all -- lo any anywhere anywhere
0 0 ACCEPT tcp -- any any anywhere anywhere state NEW tcp dpt:ssh
0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1 84 DOCKER all -- any docker0 anywhere anywhere
0 0 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- docker0 !docker0 anywhere anywhere
> 1 84 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
> 0 0 ACCEPT all -- docker0 docker0 anywhere anywhere
Chain OUTPUT (policy ACCEPT 3 packets, 280 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Expected results(1.6.0):
docker run -it ubuntu ping -c 1 $ip
PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data.
> 64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=0.302 ms
--- 172.17.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.302/0.302/0.302/0.000 ms
iptables -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
2844 4001K ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED
0 0 ACCEPT icmp -- any any anywhere anywhere
0 0 ACCEPT all -- lo any anywhere anywhere
0 0 ACCEPT tcp -- any any anywhere anywhere state NEW tcp dpt:ssh
0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
2 168 DOCKER all -- any docker0 anywhere anywhere
1 84 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- docker0 !docker0 anywhere anywhere
> 1 84 ACCEPT all -- docker0 docker0 anywhere anywhere
> 0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 1498 packets, 79255 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Additional info:
Default reject rule for all connections is added to FORWARD chain here:
[root@localhost ~]# cat /etc/sysconfig/iptables
# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
> -A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
This config comes from iptables-services package:
[root@localhost ~]# rpm -qf /etc/sysconfig/iptables
iptables-services-1.4.21-13.el7.x86_64
This config file is immutable and was there for a long time:
[root@localhost ~]# rpm -V iptables-services
[root@localhost ~]# rpm -qi iptables-services
Name : iptables-services
Version : 1.4.21
Release : 13.el7
Architecture: x86_64
Install Date: Fri Jun 26 20:20:08 2015
Group : System Environment/Base
Size : 23727
License : GPLv2
Signature : RSA/SHA256, Fri Jul 4 06:08:37 2014, Key ID 24c6a8a7f4a80eb5
Source RPM : iptables-1.4.21-13.el7.src.rpm
Build Date : Tue Jun 10 09:02:48 2014
Build Host : worker1.bsys.centos.org
Relocations : (not relocatable)
Packager : CentOS BuildSystem <http://bugs.centos.org>
Vendor : CentOS
URL : http://www.netfilter.org/
Summary : iptables and ip6tables services for iptables
Description :
iptables services for IPv4 and IPv6
In docker-1.7.0 it will be default behavior to append but not to insert icc(inter-container communication) rule. And this patch from 1.7.0 is ported to docker-1.6.2-14.el7.centos.x86_64:
git describe --contains 90a8e45604f42d60d58b4cefa37a5e5d3112b64a
v1.7.0-rc1~2^2~278^2
commit 90a8e45604f42d60d58b4cefa37a5e5d3112b64a
Author: Gosuke Miyashita <gosukenator>
Date: Sat Mar 21 01:52:05 2015 +0900
Append icc related iptables rules, not INSERT
Signed-off-by: Gosuke Miyashita <gosukenator>
Daniel, no, with docker 1.7 it will be the same(as it will also have this patch) - docker ICC rule will be appended after iptables-service reject rule(table filter, chain FORWARD), and there will be no ICC between dockers.
Comment 10Jan Pazdziora (Red Hat)
2015-07-17 13:49:42 UTC
My take would probably be that if people want do run docker with ICC, they need to edit that default /etc/sysconfig/iptables as provided by iptables-services and replace the
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
with something like
-A FORWARD ! -o docker0 -j REJECT --reject-with icmp-host-prohibited
Because by default, forwarding is disabled and after all, ICC is forwarding.
Of course, another possibility is to put the default docker rules to something like DOCKERDEFAULT, have that inserted to FORWARD, and then append the ICC enable/disable rules to DOCKERDEFAULT, to make sure we don't clash with anything else added to FORWARD.
Comment 11Pavel Tikhomirov
2015-07-17 14:03:12 UTC
> Is this the same problem.
Not exactly as docker-1.6.2-14.el7.centos.x86_64 sets these rules in DOCKER chain for --link'ed docker containers and they work normally:
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- !docker0 docker0 anywhere 172.17.0.1 tcp dpt:http
> 0 0 ACCEPT tcp -- docker0 docker0 172.17.0.2 172.17.0.1 tcp dpt:https
> 0 0 ACCEPT tcp -- docker0 docker0 172.17.0.1 172.17.0.2 tcp spt:https
> 0 0 ACCEPT tcp -- docker0 docker0 172.17.0.2 172.17.0.1 tcp dpt:http
> 0 0 ACCEPT tcp -- docker0 docker0 172.17.0.1 172.17.0.2 tcp spt:http
my guess is that docker 1.7.0 do not do that.
Description of problem: Docker inter-container communication stopped working after updating docker from docker 1.6.0 to 1.6.2. Iptables-services package prevents docker to properly configure ICC iptables rules with its default REJECT rule. Version-Release number of selected component (if applicable): docker-1.6.2-14.el7.centos.x86_64 iptables-services-1.4.21-13.el7.x86_64 docker-1.6.0-11.0.1.el7.centos.x86_64 - for comparison How reproducible: 100% Steps to Reproduce: 1. yum install docker -y 2. systemctl start docker 3. docker run -d --name test ubuntu sleep 1000 4. ip=$(docker inspect -f '{{.NetworkSettings.IPAddress}}' test) 5. docker run -it ubuntu ping -c 1 $ip 6. iptables -L -v Actual results: docker run -it ubuntu ping -c 1 $ip PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data. > From 172.17.42.1 icmp_seq=1 Destination Host Prohibited --- 172.17.0.1 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms iptables -L -v Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 2 168 ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED 0 0 ACCEPT icmp -- any any anywhere anywhere 0 0 ACCEPT all -- lo any anywhere anywhere 0 0 ACCEPT tcp -- any any anywhere anywhere state NEW tcp dpt:ssh 0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 1 84 DOCKER all -- any docker0 anywhere anywhere 0 0 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- docker0 !docker0 anywhere anywhere > 1 84 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited > 0 0 ACCEPT all -- docker0 docker0 anywhere anywhere Chain OUTPUT (policy ACCEPT 3 packets, 280 bytes) pkts bytes target prot opt in out source destination Chain DOCKER (1 references) pkts bytes target prot opt in out source destination Expected results(1.6.0): docker run -it ubuntu ping -c 1 $ip PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data. > 64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=0.302 ms --- 172.17.0.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.302/0.302/0.302/0.000 ms iptables -L -v Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 2844 4001K ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED 0 0 ACCEPT icmp -- any any anywhere anywhere 0 0 ACCEPT all -- lo any anywhere anywhere 0 0 ACCEPT tcp -- any any anywhere anywhere state NEW tcp dpt:ssh 0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 2 168 DOCKER all -- any docker0 anywhere anywhere 1 84 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- docker0 !docker0 anywhere anywhere > 1 84 ACCEPT all -- docker0 docker0 anywhere anywhere > 0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT 1498 packets, 79255 bytes) pkts bytes target prot opt in out source destination Chain DOCKER (1 references) pkts bytes target prot opt in out source destination Additional info: Default reject rule for all connections is added to FORWARD chain here: [root@localhost ~]# cat /etc/sysconfig/iptables # sample configuration for iptables service # you can edit this manually or use system-config-firewall # please do not ask us to add additional ports/services to this default configuration *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited > -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT This config comes from iptables-services package: [root@localhost ~]# rpm -qf /etc/sysconfig/iptables iptables-services-1.4.21-13.el7.x86_64 This config file is immutable and was there for a long time: [root@localhost ~]# rpm -V iptables-services [root@localhost ~]# rpm -qi iptables-services Name : iptables-services Version : 1.4.21 Release : 13.el7 Architecture: x86_64 Install Date: Fri Jun 26 20:20:08 2015 Group : System Environment/Base Size : 23727 License : GPLv2 Signature : RSA/SHA256, Fri Jul 4 06:08:37 2014, Key ID 24c6a8a7f4a80eb5 Source RPM : iptables-1.4.21-13.el7.src.rpm Build Date : Tue Jun 10 09:02:48 2014 Build Host : worker1.bsys.centos.org Relocations : (not relocatable) Packager : CentOS BuildSystem <http://bugs.centos.org> Vendor : CentOS URL : http://www.netfilter.org/ Summary : iptables and ip6tables services for iptables Description : iptables services for IPv4 and IPv6 In docker-1.7.0 it will be default behavior to append but not to insert icc(inter-container communication) rule. And this patch from 1.7.0 is ported to docker-1.6.2-14.el7.centos.x86_64: git describe --contains 90a8e45604f42d60d58b4cefa37a5e5d3112b64a v1.7.0-rc1~2^2~278^2 commit 90a8e45604f42d60d58b4cefa37a5e5d3112b64a Author: Gosuke Miyashita <gosukenator> Date: Sat Mar 21 01:52:05 2015 +0900 Append icc related iptables rules, not INSERT Signed-off-by: Gosuke Miyashita <gosukenator>