Bug 1245325

+++ This bug was initially created as a clone of Bug #1244124 +++

Description of problem:
Cannot link containers.


Version-Release number of selected component (if applicable):
docker-1.7.0-6.git74e7a7a.fc22.x86_64


How reproducible:
Always.


Steps to Reproduce:
1. Open 2 terminals.
2. In terminal 1, execute `docker run -it --rm --name db docker.io/postgres:9.2`
3. Wait until it says "LOG:  database system is ready to accept connections".
4. In terminal 2, execute `docker run -it --rm --link db:db postgres:9.2 bash -c 'psql -h $DB_PORT_5432_TCP_ADDR -p $DB_PORT_5432_TCP_PORT -U postgres'`


Actual results:
psql: could not connect to server: No route to host
    Is the server running on host "172.17.0.162" and accepting
    TCP/IP connections on port 5432?


Expected results:
Client connected to database.


Additional info:
Bug to postgres image: https://github.com/docker-library/postgres/issues/73

However, I tested with other images and none can be linked.

I tested with `setenforce 0` with same results.

You can try with `docker run -it --rm --link db:db docker.io/uzyexe/nmap db -p 5432` instead of step 4. It will say the port is "filtered", where it should be "open".

--- Additional comment from Yajo on 2015-07-17 04:37:03 EDT ---

Workaround is to downgrade docker. I know I will get back some bugs, but without linking I just cannot work:

$ sudo dnf downgrade docker-fish-completion docker
$ docker run -it --rm --name db docker.io/postgres:9.2

(new term)
$ docker run -it --rm --link db:db docker.io/uzyexe/nmap db -p 5432

Starting Nmap 6.47 ( http://nmap.org ) at 2015-07-17 08:33 UTC
Nmap scan report for db (172.17.0.1)
Host is up (0.000063s latency).
PORT     STATE SERVICE
5432/tcp open  postgresql
MAC Address: 02:42:AC:11:00:01 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds

$ rpm -q docker
docker-1.6.0-3.git9d26a07.fc22.x86_64

--- Additional comment from Kayvan Sylvan on 2015-07-17 10:25:27 EDT ---

Same problem here. This used to work flawlessly for me, and now it does not.

The simple figest source (with a flask application linking redis with a python app) which used to work is no longer working, with the following error:

web_1   |     raise ConnectionError(self._error_message(e))
web_1   | ConnectionError: Error 113 connecting to redis:6379. No route to host.

From my host, I can access the redis container, but it looks like forwarding of network traffic between containers is not working:

$ docker inspect figtest_redis_1  | grep 172
        "Gateway": "172.17.42.1",
        "IPAddress": "172.17.0.7",

$ nc 172.17.0.7 6379
ping
+PONG

--- Additional comment from Kayvan Sylvan on 2015-07-17 13:28:08 EDT ---

In one window:

$ docker run --rm -it --name=one docker.io/busybox /bin/sh
/ # nc -l -p 4444

On the host:

$ docker inspect one | grep 172
        "Gateway": "172.17.42.1",
        "IPAddress": "172.17.0.11",
$ nc 172.17.0.11 4444
hello

And the "hello" shows up in the container.

Now, on the host again:

$ docker run --rm -it --name=two --link one:one docker.io/busybox /bin/sh
/ # 

Now, inside the "two" container:

/ # ping one
PING one (172.17.0.11): 56 data bytes
64 bytes from 172.17.0.11: seq=0 ttl=64 time=0.160 ms
64 bytes from 172.17.0.11: seq=1 ttl=64 time=0.100 ms
^C
--- one ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.100/0.130/0.160 ms

Looks like I can ping the other container, but this:

/ # nc one 4444
nc: can't connect to remote host (172.17.0.11): No route to host
/ #

--- Additional comment from Kayvan Sylvan on 2015-07-17 21:36:00 EDT ---

Quick and dirty workaround:

# sudo service firewalld stop

It seems that the correct rules to enable forwarding packets that originate from or go to the docker0 interface are not being setup.

I hope this gives the pointer to the developers for how to fix this.

Thanks!

--- Additional comment from Kayvan Sylvan on 2015-07-19 18:52:15 EDT ---

And one more (better) workaround:

After bringing up the docker daemon (so the DOCKER) firewall chain is there, as root, run:

iptables -A DOCKER --source 0.0.0.0/0 --destination 172.17.0.0/16 \
  -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

iptables -A DOCKER --destination 0.0.0.0/0 --source 172.17.0.0/16 -j ACCEPT

Now the forwarding works as expected and linked containers work.

--- Additional comment from Yajo on 2015-07-20 04:44:15 EDT ---

@Kayvan I tried your workaround but I got hit by https://bugzilla.redhat.com/show_bug.cgi?id=1244659, so I'll stick with the downgraded docker for now.

--- Additional comment from Kayvan Sylvan on 2015-07-20 08:53:17 EDT ---

@Yajo which workaround? Fixing the DOCKER chain for iptables? I'll play around with it some more today and see if I can reproduce your issue.

--- Additional comment from Kayvan Sylvan on 2015-07-20 10:28:31 EDT ---

@Yajo The curl command in bug 1244659 works perfectly for me in the container, before and after applying the iptables workaround, so that is some other networking/routing bug.

--- Additional comment from Yajo on 2015-07-21 03:24:51 EDT ---

(In reply to Kayvan Sylvan from comment #7)
> @Yajo which workaround? Fixing the DOCKER chain for iptables? I'll play
> around with it some more today and see if I can reproduce your issue.

Disabling firewalld. I did not try to add the iptables rules because without firewalld it did not work too, so it had no sense to me.


(In reply to Kayvan Sylvan from comment #8)
> @Yajo The curl command in bug 1244659 works perfectly for me in the
> container, before and after applying the iptables workaround, so that is
> some other networking/routing bug.

That's why I reported it separately. But the workaround that works for me is the same: downgrade docker.

--- Additional comment from Kayvan Sylvan on 2015-07-21 05:22:27 EDT ---

@Yajo Disabling firewalld does not work too well, since docker-compose, for example seems to rely on it.

Can you try the other workaround instead? I'm curious if it will work for you.

--- Additional comment from Josef Skladanka on 2015-07-21 07:06:14 EDT ---

(In reply to Kayvan Sylvan from comment #10)

I tried the firewall rules proposed in comment #5, and it works for me - containers are now able to connect to each other.

--- Additional comment from Alec Benson on 2015-07-21 10:33:04 EDT ---

I've moved this over to firewalld so that they can take a look at this.

--- Additional comment from Daniel Walsh on 2015-07-21 10:39:06 EDT ---

Thomas I believe this problem is being caused by the firewalld patch you guys submitted to docker.  Could you check?

--- Additional comment from Daniel Walsh on 2015-07-21 14:06:18 EDT ---

https://github.com/docker/docker/issues/14396

Is covering this upstream.

https://github.com/Nightling states:

"
Just add a rule to the DOCKER chain in the filter table: -i docker0 -o docker0 -j ACCEPT

Here's an easier way to reproduce:
(Fedora Cloud does not ship firewalld as pointed out by @dustymabe.)

docker run -it --rm --name server busybox nc -lp 5000

docker run -it --rm --name client busybox nc \
$(docker inspect -f "{{ .NetworkSettings.IPAddress }}" server) 5000

The root cause is simply firewall rules not being added automatically.
I also tested this with the latest 1.8 build in the weekend.
"

--- Additional comment from Daniel Walsh on 2015-07-21 14:11:11 EDT ---

This rule would break 

docker -d --icc=false

But the default is --icc=true

iptables -A DOCKER -i docker0 -o docker0 -j ACCEPT