Bug 1033606 - Failed to connect to network from Docker container [NEEDINFO]
Summary: Failed to connect to network from Docker container
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: docker-io
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Marek Goldmann
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-11-22 13:38 UTC by Michal Fojtik
Modified: 2015-06-17 13:13 UTC (History)
11 users (show)

Fixed In Version: docker-io-0.7.0-14.fc20
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-12-03 01:20:13 UTC
Type: Bug
Embargoed:
andrei: needinfo?


Attachments (Terms of Use)
iptables output (20.13 KB, text/plain)
2013-11-26 09:27 UTC, Marek Goldmann
no flags Details

Description Michal Fojtik 2013-11-22 13:38:12 UTC
Description of problem:

Connecting to external network from Docker container fail due to firewalld. I guess you must have masquerade enabled, however this is not mentioned anywhere. I think docker-io should set the firewalld rules automatically, or tell users that they need to enable masquarade in firewalld.

Version-Release number of selected component (if applicable):

Name        : docker-io
Arch        : x86_64
Version     : 0.7
Release     : 0.17.rc6.fc20

Steps to Reproduce:
1. $ yum install docker-io
2. $ systemctl enable docker.service
3. $ systemctl start docker.service
4. $ docker pull mattdm/fedora
5. $ docker run -i -t mattdm/fedora:latest /bin/bash
6. $ ping google.com
ping: unknown host google.com

When I stop firewalld on host (systemctl stop firewalld) and then restart the docker (systemctl restart docker), the ping works like a charm.

Actual results:

Unable to connect outside the Docker container with firewalld enabled.

Expected results:

Docker should configure firewalld automatically (during install?), or inform users to do so manually.

Additional info:

Comment 1 Lokesh Mandvekar 2013-11-24 00:18:43 UTC
Josh,

looks like something similar to Bug 1026045 is happening again. is this a known issue?

Just verified it on 0.7-0.19.rc7.fc19

Comment 2 Josh Poimboeuf 2013-11-25 14:31:27 UTC
I installed docker-io-0.7-0.19.rc7.fc19.x86_64 but wasn't able to recreate.    Is it possible you ran an older version of docker without rebooting before trying this version?  Can you post the output of "iptables -Lv"?

Comment 3 Josh Poimboeuf 2013-11-25 14:32:34 UTC
correction: iptables -L -v

Comment 4 Marek Goldmann 2013-11-26 09:27:21 UTC
Created attachment 829120 [details]
iptables output

I can see this issue as well, with the upcoming 0.7.0-1 release, attaching my iptables output.

Comment 5 Marek Goldmann 2013-11-26 09:50:12 UTC
We have following rules executed:

The systemd service (before docker starts):

/usr/sbin/sysctl -w net.ipv4.ip_forward=1 net.ipv6.conf.all.forwarding=1

And docker runs this on its own:

lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-D FORWARD -i docker0 -o docker0 -j DROP]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-C FORWARD -i docker0 -o docker0 -j ACCEPT]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -D PREROUTING -j DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -D OUTPUT -j DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -F DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -X DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -N DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
lis 26 10:43:15 mistress docker[11602]: [DEBUG] [iptables]: /usr/sbin/iptables, [-t nat -A OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]

Comment 6 Marek Goldmann 2013-11-26 11:50:53 UTC
After boot I cannot connect to the Internet from container, but when I do:

    systemctl restart firewalld
    systemctl restart docker

then I can. Restarting docker service alone does not help. Adding hard requires to docker systemd file in the [Unit] section:

    Requires=firewalld.service

doesn't help either. BTW, we should add "Wants=firewalld.service" to make sure the firewalld service is started before Docker, if it's available.

I suspect that the issue is in the time which is needed to create all the required devices. If we add:

    ExecStartPre=/usr/bin/sleep 3

to the [Service] section makes it possible to connect to the Internet from the container. Even 1s is sufficient on my system.

Comment 7 Michal Fojtik 2013-11-26 14:44:40 UTC
FYI I was playing with this and I discover this (dunno if it helps):

[root@localhost redis-server]# docker run -i -t mattdm/fedora /bin/bash
bash-4.2# ping google.com
ping: unknown host google.com

[root@localhost redis-server]# systemctl stop firewalld
[root@localhost redis-server]# docker run -i -t mattdm/fedora /bin/bash
lxc-start: failed to attach 'vethkb9kR0' to the bridge 'docker0' : No such device
lxc-start: failed to create netdev
lxc-start: failed to create the network
lxc-start: failed to spawn '1db793c53b22a2fda433a9ce2ddbb9022cca6a0c33126389ac95ca7b176c947c'
[error] commands.go:2459 Error resize: Error: bad file descriptor

[root@localhost redis-server]# systemctl restart docker

[root@localhost redis-server]# docker run -i -t mattdm/fedora /bin/bash
bash-4.2# ping google.com
PING google.com (173.194.41.135) 56(84) bytes of data.

Comment 8 Michal Fojtik 2013-11-26 14:51:46 UTC
Marek: http://fpaste.org/56877/38547748/

Comment 9 Josh Poimboeuf 2013-11-26 15:11:38 UTC
This isn't right:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            

There should be more docker-related rules there.  Is there a unit file that creates the docker0 device before docker starts?  If so, remove it so that docker can create it and set up its iptables rules.

Comment 10 Marek Goldmann 2013-11-26 15:19:56 UTC
(In reply to Josh Poimboeuf from comment #9)

> There should be more docker-related rules there.  Is there a unit file that
> creates the docker0 device before docker starts?  If so, remove it so that
> docker can create it and set up its iptables rules.

This is what I have:

$ systemctl list-units -a | grep docker
sys-devices-virtual-net-docker0.device                                                                         loaded    active   plugged   /sys/devices/virtual/net/docker0
sys-subsystem-net-devices-docker0.device                                                                       loaded    active   plugged   /sys/subsystem/net/devices/docker0
docker.service                                                                                                 loaded    inactive dead      Docker container management daemon

And indeed, the docker0 interface is up, even when we stop the docker service.

Comment 11 Josh Poimboeuf 2013-11-26 15:41:32 UTC
Ok.  Removing those unit files and rebooting should fix the issue.

Comment 12 Marek Goldmann 2013-11-26 16:45:21 UTC
(In reply to Josh Poimboeuf from comment #11)
> Ok.  Removing those unit files and rebooting should fix the issue.

Josh, above are devices. These cannot be listed with the "systemctl list-unit-files" command. I think the real issue is that docker leaves the docker0 interface running, even after we stop the service. WDYT?

Comment 13 Josh Poimboeuf 2013-11-26 17:06:46 UTC
> Josh, above are devices. These cannot be listed with the "systemctl
> list-unit-files" command. I think the real issue is that docker leaves the
> docker0 interface running, even after we stop the service. WDYT?

Sorry, I misunderstood.  It should be ok for docker to leave the docker0 bridge device after it exits.  In fact it's probably necessary so that already running containers won't lose their network if the docker daemon has to restart.

When docker starts up, it checks for the existence of docker0.  If it doesn't exist then it creates it and sets up the iptables rules appropriately.  So future starts of docker will re-use the same bridge device, which should work fine.

The debug trace you posted seemed to show that docker0 already existed.  So what I'm still confused about is how is the the docker0 device getting created to start with?  It looks like somebody created docker0 without setting up its needed iptables rules.

Comment 14 Marek Goldmann 2013-11-26 17:27:55 UTC
(In reply to Josh Poimboeuf from comment #13)

> Sorry, I misunderstood.  It should be ok for docker to leave the docker0
> bridge device after it exits.  In fact it's probably necessary so that
> already running containers won't lose their network if the docker daemon has
> to restart.
>
> When docker starts up, it checks for the existence of docker0.  If it
> doesn't exist then it creates it and sets up the iptables rules
> appropriately.  So future starts of docker will re-use the same bridge
> device, which should work fine.

OK, this makes sense.
 
> The debug trace you posted seemed to show that docker0 already existed.  So
> what I'm still confused about is how is the the docker0 device getting
> created to start with?  It looks like somebody created docker0 without
> setting up its needed iptables rules.

I wouldn't assume this, since the rules are executed no matter if the docker0 interface is started up or not:

https://github.com/dotcloud/docker/blob/v0.7.0/iptables/iptables.go#L105

This bridge was created by running the systemd service, no other tool created it.

Comment 15 Josh Poimboeuf 2013-11-27 02:25:37 UTC
> > The debug trace you posted seemed to show that docker0 already existed.  So
> > what I'm still confused about is how is the the docker0 device getting
> > created to start with?  It looks like somebody created docker0 without
> > setting up its needed iptables rules.
> 
> I wouldn't assume this, since the rules are executed no matter if the
> docker0 interface is started up or not:
> 
> https://github.com/dotcloud/docker/blob/v0.7.0/iptables/iptables.go#L105
> 
> This bridge was created by running the systemd service, no other tool
> created it.

Actually the rules that seem to be missing are the ones in the FORWARD table, which are created whenever the docker0 bridge is created:

https://github.com/dotcloud/docker/blob/v0.7.0/network.go#L180

Comment 16 Marek Goldmann 2013-11-27 11:24:07 UTC
Thanks Josh!

It looks like the iptables rules creation in docker is wrong. It assumes that the bridge interface is started every time. I created a patch which can be found here: https://github.com/goldmann/docker/commit/0ff9bc1be3ae044107732c605986a0af20220134

Additionally I prepared a scratch build for Fedora 20 (x86_64) with this patch already applied, please test the build and let me know if this fixes the issue:

http://koji.fedoraproject.org/koji/taskinfo?taskID=6231401

I'm going to open an upstream bug report for this.

P.S. I'm assigning this ticket to me.

Comment 17 Marek Goldmann 2013-11-27 11:40:02 UTC
Please use this scratch build:

https://koji.fedoraproject.org/koji/taskinfo?taskID=6231502

I've added this to the docker.service:

[Unit]
...
Requires=firewalld.service
After=firewalld.service

To make sure the iptables rules are executed after the firewalld is actually running (if available) to prevent seeing network connectivity issues after boot.

Comment 18 Marek Goldmann 2013-11-27 11:41:54 UTC
Upstream pull request opened: https://github.com/dotcloud/docker/pull/2907

Comment 19 Lokesh Mandvekar 2013-11-27 14:20:11 UTC
+1

Comment 20 Lokesh Mandvekar 2013-11-27 15:53:15 UTC
hmm, so it seems if i try a docker pull after immediately starting docker service, it'll complain /var/run/docker.sock no such file or directory, but it seems to work well after sometime. This could vary between machines though, also sometimes it'll work right away

Comment 21 Josh Poimboeuf 2013-11-28 04:55:36 UTC
(In reply to Marek Goldmann from comment #16)
> Thanks Josh!
> 
> It looks like the iptables rules creation in docker is wrong. It assumes
> that the bridge interface is started every time. I created a patch which can
> be found here:
> https://github.com/goldmann/docker/commit/
> 0ff9bc1be3ae044107732c605986a0af20220134

AFAICT, the FORWARD rules only need to be created once, at bridge creation time.  The bridge device and the FORWARD rules are never removed.  They can then be re-used if the docker daemon exits and restarts.

It seems like somebody is either a) creating the bridge without creating the rules or b) removing the rules without removing the bridge.  I still don't understand what's happening here.

That said, the patch itself looks fine to me.  And it might be a good idea anyway, to make sure the rules are always correct.

Comment 22 Lokesh Mandvekar 2013-11-28 06:37:59 UTC
i'm planning to push in Marek's pull request commit 0ff9bc1 to yum

Comment 23 Fedora Update System 2013-11-28 07:03:24 UTC
docker-io-0.7.0-6.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-6.fc20

Comment 24 Fedora Update System 2013-11-28 07:13:46 UTC
docker-io-0.7.0-6.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-6.fc19

Comment 25 Fedora Update System 2013-11-28 09:12:48 UTC
docker-io-0.7.0-9.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-9.fc20

Comment 26 Fedora Update System 2013-11-28 09:24:03 UTC
docker-io-0.7.0-9.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-9.fc19

Comment 27 Fedora Update System 2013-11-28 09:42:18 UTC
docker-io-0.7.0-9.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-9.el6

Comment 28 Fedora Update System 2013-11-28 18:28:27 UTC
docker-io-0.7.0-10.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-10.fc20

Comment 29 Fedora Update System 2013-11-28 18:36:32 UTC
docker-io-0.7.0-10.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-10.fc19

Comment 30 Fedora Update System 2013-11-28 18:44:16 UTC
docker-io-0.7.0-10.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-10.el6

Comment 31 Fedora Update System 2013-11-29 15:59:53 UTC
docker-io-0.7.0-10.fc20 has been pushed to the Fedora 20 testing repository.

Comment 32 Michal Fojtik 2013-11-29 22:12:25 UTC
After update and reboot, I can still reproduce this problem. Now is even worse, because stopping firewalld does not help:

[root@localhost ~]# docker run -i -t mattdm/fedora /bin/bash
bash-4.2# ping google.com
ping: unknown host google.com

[root@localhost ~]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Fri 2013-11-29 23:06:35 CET; 4min 41s ago

[root@localhost ~]# systemctl status docker
docker.service - Docker container management daemon
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: active (running) since Fri 2013-11-29 23:09:39 CET; 1min 56s ago
  Process: 2451 ExecStartPre=/usr/sbin/sysctl -w net.ipv4.ip_forward=1 net.ipv6.conf.all.forwarding=1 (code=exited, status=0/SUCCESS)
 Main PID: 2452 (docker)
   CGroup: /system.slice/docker.service
           ├─2452 /usr/bin/docker -d
           ├─2557 lxc-start -n a98a431f8b6c8666ed6a77265f48ee883b737059cd932677f37f390df1d7ef8d -f /var/lib/docker/containers/a98a431f8b6c8666ed6a77265f48ee883b737059cd932677f37f390df1d7ef8d/...
           └─2564 /bin/bash

Nov 29 23:10:47 localhost.localdomain docker[2452]: 2013/11/29 23:10:47 POST /v1.7/containers/create
Nov 29 23:10:47 localhost.localdomain docker[2452]: [/var/lib/docker|aa7b0df9] +job create()
Nov 29 23:10:48 localhost.localdomain docker[2452]: 0f3196cd4b7f343a3a487bd38860c92ad289fc66ffa672c9583a484844c07962
Nov 29 23:10:48 localhost.localdomain docker[2452]: [/var/lib/docker|aa7b0df9] -job create() = OK (0)
Nov 29 23:10:48 localhost.localdomain docker[2452]: 2013/11/29 23:10:48 POST /v1.7/containers/0f3196cd4b7f343a3a487bd38860c92ad289fc66ffa672c9583a484844c07962/attach?stderr=1&stdi...t=1&stream=1
Nov 29 23:10:48 localhost.localdomain docker[2452]: 2013/11/29 23:10:48 POST /v1.7/containers/0f3196cd4b7f343a3a487bd38860c92ad289fc66ffa672c9583a484844c07962/start
Nov 29 23:10:48 localhost.localdomain docker[2452]: [/var/lib/docker|aa7b0df9] +job start(0f3196cd4b7f343a3a487bd38860c92ad289fc66ffa672c9583a484844c07962)
Nov 29 23:10:48 localhost.localdomain docker[2452]: [/var/lib/docker|aa7b0df9] -job start(0f3196cd4b7f343a3a487bd38860c92ad289fc66ffa672c9583a484844c07962) = OK (0)
Nov 29 23:10:48 localhost.localdomain docker[2452]: 2013/11/29 23:10:48 POST /v1.7/containers/0f3196cd4b7f343a3a487bd38860c92ad289fc66ffa672c9583a484844c07962/resize?h=47&w=194
Nov 29 23:11:12 localhost.localdomain docker[2452]: 2013/11/29 23:11:12 GET /v1.7/containers/0f3196cd4b7f343a3a487bd38860c92ad289fc66ffa672c9583a484844c07962/json
Hint: Some lines were ellipsized, use -l to show in full.


Version:

Name        : docker-io
Arch        : x86_64
Version     : 0.7.0
Release     : 10.fc20

Comment 33 Lokesh Mandvekar 2013-11-30 16:43:37 UTC
Michal, can you change your Requires=firewalld.service to Wants=firewalld.service as in Bug 1036217 and check if it helps?

Comment 34 Michal Fojtik 2013-11-30 20:03:09 UTC
Lokesh: I tried, but unfortunatelly it did not help. However, the behavior changed a bit (note I do a full reboot after the change).

[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# docker run -i -t base/arch /bin/bash
lxc-start: failed to attach 'vethOjHimB' to the bridge 'docker0' : No such device
lxc-start: failed to create netdev
lxc-start: failed to create the network
lxc-start: failed to spawn '2331a2594cd703ca76f15bf382f0c2724b149c64c6529081cc767beb4c22868d'

After restarting Docker service:

[root@localhost ~]# systemctl restart docker
[root@localhost ~]# docker run -i -t base/arch /bin/bash
[root@d50801ccec40 /]# ping google.com
ping: unknown host google.com

So the result is still the same :-(

BUT, I found the workaround:

[root@localhost ~]# firewall-cmd --add-masquerade
success
[root@localhost ~]# docker run -i -t base/arch /bin/bash
[root@f3b88e508538 /]# ping google.com
PING google.com (173.194.35.70) 56(84) bytes of data.
64 bytes from 173.194.35.70: icmp_seq=1 ttl=55 time=14.5 ms

So adding a MASQUARADE in firewalld seems to fix this problem. Can we make this call in Docker service, or alternatively create a 'docker' zone in firewalld and enable MASQUARADE for this zone?

Comment 35 Lokesh Mandvekar 2013-11-30 22:19:57 UTC
I could add

ExecStartPost=firewall-cmd --add-masquerade  ... to the unit file, unless there's a cleaner solution

Comment 36 Lokesh Mandvekar 2013-12-01 01:17:34 UTC
So, this service file should work (works for me) even if firewalld isn't present on the system: 

[Unit]
Description=Docker container management daemon
Wants=firewalld.service
After=firewalld.service

[Service]
Type=simple
ExecStartPre=/usr/sbin/sysctl -w net.ipv4.ip_forward=1 net.ipv6.conf.all.forwarding=1
ExecStart=/usr/bin/docker -d
ExecStartPost=firewall-cmd --add-masquerade
Restart=on-failure

[Install]
WantedBy=multi-user.target


Comments?

Comment 37 Fedora Update System 2013-12-01 02:49:56 UTC
docker-io-0.7.0-10.el6 has been pushed to the Fedora EPEL 6 testing repository.

Comment 38 Fedora Update System 2013-12-01 19:33:33 UTC
docker-io-0.7.0-12.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-12.fc20

Comment 39 Fedora Update System 2013-12-02 02:45:18 UTC
docker-io-0.7.0-12.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-12.fc19

Comment 40 Fedora Update System 2013-12-02 04:59:06 UTC
docker-io-0.7.0-12.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-12.el6

Comment 41 Marek Goldmann 2013-12-02 09:29:42 UTC
(In reply to Lokesh Mandvekar from comment #35)
> I could add
> 
> ExecStartPost=firewall-cmd --add-masquerade  ... to the unit file, unless
> there's a cleaner solution

Please don't do this. This should be already covered in my patch.

Comment 42 Michal Fojtik 2013-12-02 12:11:46 UTC
Marek: Is the patch included in docker-io-0.7.0-12.el6?

Comment 43 Michal Fojtik 2013-12-02 13:20:32 UTC
After system update and reboot, the networking and routing to containers seems to work perfectly fine. The --add-masquarade is not longer needed. Lokesh, Marek thanks for help!

Comment 44 Lokesh Mandvekar 2013-12-02 13:52:50 UTC
(In reply to Marek Goldmann from comment #41)
> Please don't do this. This should be already covered in my patch.

Alright, pushing out another release without masquerade, thanks for the heads up

Comment 45 Fedora Update System 2013-12-02 16:00:11 UTC
docker-io-0.7.0-14.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-14.fc20

Comment 46 Fedora Update System 2013-12-02 16:16:58 UTC
docker-io-0.7.0-14.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-14.fc19

Comment 47 Fedora Update System 2013-12-02 16:28:23 UTC
docker-io-0.7.0-14.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/docker-io-0.7.0-14.el6

Comment 48 Fedora Update System 2013-12-03 01:20:13 UTC
docker-io-0.7.0-14.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 49 Fedora Update System 2013-12-03 10:38:36 UTC
docker-io-0.7.0-14.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 50 Fedora Update System 2013-12-14 02:47:24 UTC
docker-io-0.7.0-14.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 51 michael.faille 2014-02-13 10:08:36 UTC
I use Fedora 20 freshly updated and the problem persist for me.
Docker version : Docker version 0.7.6, build bc3b2ec/0.7.6

Steps to Reproduce:
1. $ yum install docker-io
2. $ systemctl enable docker.service
3. $ systemctl start docker.service
4. $ docker pull mattdm/fedora
5. $ docker run -i -t mattdm/fedora:latest /bin/bash
6. $ ping google.com
ping: unknown host google.com

(It also dindn't work with any image)

Watch could I do to quick fix ?

Thank you !

Comment 52 Marek Goldmann 2014-02-13 10:12:15 UTC
Please try again with the fedora:latest image (not mattdm/fedora:latest - it shouldn't be used). If it doesn't work please run:

yum update docker-io --enablerepo updates-testing

to get the 0.8.0 version and try again.

Comment 53 michael.faille 2014-02-13 10:17:55 UTC
Hum, finally, I have the problem when I run a container... and my default route change ! 

Before starting docker 
$ ip route
default via 192.168.1.1 dev wlp3s0  proto static  metric 1024 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1 
192.168.1.0/24 dev wlp3s0  proto kernel  scope link  src 192.168.1.100 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1

After starting docker
$ ip route
default via 192.168.0.1 dev vethyzDqJ5  proto static  metric 1024 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1 
192.168.0.0/24 dev vethyzDqJ5  proto kernel  scope link  src 192.168.0.50 
192.168.1.0/24 dev wlp3s0  proto kernel  scope link  src 192.168.1.100 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 


The problem persist with firewall activated  : 
sudo systemctl stop firewalld
# and start container

sudo sysctl -a | grep "\.forwarding
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.docker0.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.p3p1.forwarding = 1
net.ipv4.conf.virbr0.forwarding = 1
net.ipv4.conf.virbr0-nic.forwarding = 1
net.ipv4.conf.wlp3s0.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.docker0.forwarding = 1
net.ipv6.conf.lo.forwarding = 1
net.ipv6.conf.p3p1.forwarding = 1
net.ipv6.conf.virbr0.forwarding = 1
net.ipv6.conf.virbr0-nic.forwarding = 1
net.ipv6.conf.wlp3s0.forwarding = 1

Comment 54 Marek Goldmann 2014-02-13 10:23:59 UTC
Could you please provide output of the commands:

docker version
rpm -q docker-io
rpm -V docker-io

Comment 55 michael.faille 2014-02-13 10:38:23 UTC
I use Fedora 20 with last update.

sudo rpm -q docker-io
docker-io-0.7.6-4.fc20.x86_64

rpm -V docker-io # give me nothing

Comment 56 michael.faille 2014-02-13 10:42:31 UTC
Ho sorry, I just see your help to update docker.
Few min. plz.

Comment 57 michael.faille 2014-02-13 10:53:37 UTC
You have all info in this 20 sec video made with Gnome-Shell record tool : https://www.youtube.com/watch?v=MGKMGzOjrxE&feature=youtu.be

Comment 58 Josh Poimboeuf 2014-02-13 15:20:39 UTC
Can you post the output of "iptables -vL FORWARD"?

Comment 59 michael.faille 2014-02-13 17:26:17 UTC
# In the first terminal
sudo docker run -i -t fedora /bin/bash

# In the second
sudo iptables -vL FORWARD
[sudo] password for michael: 
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED

Comment 60 michael.faille 2014-02-13 17:33:29 UTC
When I ping with this on my docker container term with this output 
12 packets transmitted, 0 received, +10 errors, 100% packet loss, time 10996ms
pipe 3


And executing "sudo iptables -vL FORWARD" after :
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            
   29  2436 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED

It give packets count on the second rule

Comment 61 Josh Poimboeuf 2014-02-13 18:00:53 UTC
I have no idea why starting a container would affect the host's routing table.  Strange...

Comment 62 michael.faille 2014-02-13 20:10:37 UTC
I think I have this problem since I have make Docker start from boot, cause I'm sure it has worked before that.
sudo systemctl enable docker

Comment 63 michael.faille 2014-02-15 11:39:39 UTC
Weird..... I fix the problem, I know exact what command I done. ...But, I don't know why it work.

My recipe : 

#1. finding manual LXC lunch command
ps aux | grep docker
root     20606  0.0  0.0 154140  5296 pts/4    S+   06:01   0:00 vim /var/lib/docker/containers/abe17802ba8958c91295fb33831ce769ef393da4f5c7e70cee32713ec165ba85/config.lxc
root     22508  0.0  0.0 209328  4072 pts/0    S+   06:18   0:00 sudo strace -f -s 1000 -o out.trace docker -d
root     22514  0.0  0.0   4704   888 pts/0    D+   06:18   0:00 strace -f -s 1000 -o out.trace docker -d
root     22519  0.0  0.0      0     0 pts/0    Zl+  06:18   0:00 [docker] <defunct>
root     22620  0.0  0.0  19192  1228 ?        Ss   06:19   0:00 lxc-start -n 3be52160a2e3f7bbbf987726edc83823a938920e39992ee9cd1726f56ca7f99e -f /var/lib/docker/containers/3be52160a2e3f7bbbf987726edc83823a938920e39992ee9cd1726f56ca7f99e/config.lxc -- /.dockerinit -driver lxc -g 172.17.42.1 -i 172.17.0.2/16 -mtu 1500 -- /bin/sh -c apt-get install -y inotify-tools openssh-server
root     22631  0.0  0.0  10292  2376 ?        Dl   06:19   0:00 /.dockerinit -driver lxc -g 172.17.42.1 -i 172.17.0.2/16 -mtu 1500 -- /bin/sh -c apt-get install -y inotify-tools openssh-server
michael  23573  0.0  0.0 112688   968 pts/1    S+   06:25   0:00 grep --color=auto docker

#2. Shudown all docker/lxc container/process

#3. Run following LXC command to just enter in the shell + test surprisingly internet with success. 
sudo lxc-start -n 3be52160a2e3f7bbbf987726edc83823a938920e39992ee9cd1726f56ca7f99e -f /var/lib/docker/containers/3be52160a2e3f7bbbf987726edc83823a938920e39992ee9cd1726f56ca7f99e/config.lxc -- /.dockerinit -driver lxc -g 172.17.42.1 -i 172.17.0.2/16 -mtu 1500 -- /bin/sh

#4. Start run docker normally......

Comment 64 Michael Gauthier 2014-07-29 19:16:33 UTC
I had this problem in Fedora 20 using Docker 1.0.

If I stop firewalld and then restart docker I am able to access the Internet from containers.

Comment 65 Andrei Gherzan 2015-06-03 18:06:18 UTC
I can reproduce this issue again on Fedora 22. The workaround for it was to configure firewall-config as follows:
1. add the docker0 interface to default zone (it wasn't there by default, why?)
2. configure masquerade for default zone

Without (2) a restart or docker is needed after every boot.

Comment 66 Andrei Gherzan 2015-06-03 18:07:42 UTC
(In reply to Andrei Gherzan from comment #65)
> I can reproduce this issue again on Fedora 22. The workaround for it was to
> configure firewall-config as follows:
> 1. add the docker0 interface to default zone (it wasn't there by default,
> why?)
> 2. configure masquerade for default zone
> 
> Without (2) a restart or docker is needed after every boot.

on docker*

Comment 67 Peter Williams 2015-06-17 13:13:28 UTC
I've been having the same issues. For reference, here's the command-line method for implementing the configuration mentioned by Andrei in #65, which tentatively seems to fix things for me too:

sudo firewall-cmd --permanent --zone=public --add-interface=docker0
sudo firewall-cmd --permanent --zone=public --add-masquerade


Note You need to log in before you can comment on or make changes to this bug.