Bug 1279002

Summary: Docker 1.8.2 fails to set iptables rules
Product: [Fedora] Fedora Reporter: Vartan Simonian <vsimonianpress>
Component: dockerAssignee: smahajan <smahajan>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 23CC: adimania, admiller, dwalsh, ichavero, jcajka, jchaloup, lsm5, miminar, vbatts, vsimonianpress, yajo.sk8, zlogic
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
URL: https://github.com/docker/docker/pull/16038
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1279015 (view as bug list) Environment:
Last Closed: 2016-02-16 20:00:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1279015    
Attachments:
Description Flags
Output of journalctl -xe none

Description Vartan Simonian 2015-11-07 00:48:23 UTC 
Description of problem:

Docker 1.8.2 suffers from bugs in its bundled libnetwork related to network namespace handling. One effect is that Docker 1.8.2 fails to set iptables rules, which is reproducible on a fresh instance of Fedora 23 Server with the latest available Docker package.

Here's a user with the same problem, albeit using Docker's repositories and Fedora 22: https://github.com/docker/docker/issues/15948


Version-Release number of selected component (if applicable):

1.8.2 10.git28c300f.fc23


How reproducible:

Consistent.


Steps to Reproduce:

1. Install docker (sudo dnf install docker)
2. Start docker (sudo systemctl enable docker && sudo systemctl start docker)
3. Check firewalld logs (journalctl -u firewalld)


Actual results:

The following errors are logged to the firewalld service:

Nov 06 15:16:37 testserver /firewalld[841]: 2015-11-06 15:16:37 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
                                                                    
                                                                    Try `iptables -h' or 'iptables --help' for more information.
Nov 06 15:16:37 testserver /firewalld[841]: 2015-11-06 15:16:37 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
                                                                    
                                                                    Try `iptables -h' or 'iptables --help' for more information.
Nov 06 15:16:37 testserver /firewalld[841]: 2015-11-06 15:16:37 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
                                                                    
                                                                    Try `iptables -h' or 'iptables --help' for more information.
Nov 06 15:16:37 testserver /firewalld[841]: 2015-11-06 15:16:37 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 06 15:16:37 testserver /firewalld[841]: 2015-11-06 15:16:37 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 06 15:16:37 testserver /firewalld[841]: 2015-11-06 15:16:37 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.
Nov 06 15:16:37 testserver /firewalld[841]: 2015-11-06 15:16:37 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -X DOCKER' failed: iptables: No chain/target/match by that name.
Nov 06 15:16:37 testserver /firewalld[841]: 2015-11-06 15:16:37 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -C POSTROUTING -s 172.17.42.1/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 06 15:16:38 testserver /firewalld[841]: 2015-11-06 15:16:38 ERROR: COMMAND_FAILED: '/sbin/iptables -w -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.


Expected results:

Docker starts up and successfully creates iptables rules.


Additional info:

This bug has been fixed upstream in Docker PR #16038 ( https://github.com/docker/docker/pull/16038 ), but according to the developers, has only been released in 1.9, not 1.8: https://github.com/docker/docker/issues/15948#issuecomment-140828074

Effectively, without 1.9 available on Fedora 23, this bug will persist.
Comment 1 Daniel Walsh 2015-11-07 06:19:11 UTC 
Shishir could you check if this is fixed in docker-1.9?
Comment 3 Daniel Walsh 2015-11-07 06:21:51 UTC 
So reading the docker issue, it looks like this is fixed in docker-1.9 release.
Comment 4 Daniel Walsh 2015-12-01 19:27:32 UTC 
Fixed in docker-1.9
Comment 5 Yajo 2015-12-16 10:47:45 UTC 
Created attachment 1106365 [details]
Output of journalctl -xe

I cannot even start docker with 1.9!

> rpm -q docker
docker-1.9.1-4.git6ec29ef.fc23.x86_64

> systemctl start docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

> systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since mié 2015-12-16 11:43:02 CET; 5s ago
     Docs: http://docs.docker.com
  Process: 11426 ExecStart=/usr/bin/docker daemon $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 11426 (code=exited, status=1/FAILURE)

dic 16 11:43:02 hpjairo.stage7.com systemd[1]: Starting Docker Application Container Engine...
dic 16 11:43:02 hpjairo.stage7.com docker[11426]: time="2015-12-16T11:43:02.552443433+01:00" level=warning msg="Usage of loopback d...tion."
dic 16 11:43:02 hpjairo.stage7.com docker[11426]: time="2015-12-16T11:43:02.602763817+01:00" level=info msg="[graphdriver] using pr...per\""
dic 16 11:43:02 hpjairo.stage7.com docker[11426]: time="2015-12-16T11:43:02.737127228+01:00" level=info msg="Firewalld running: true"
dic 16 11:43:02 hpjairo.stage7.com docker[11426]: time="2015-12-16T11:43:02.975804736+01:00" level=fatal msg="Error starting daemon...oints"
dic 16 11:43:02 hpjairo.stage7.com systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
dic 16 11:43:02 hpjairo.stage7.com systemd[1]: Failed to start Docker Application Container Engine.
dic 16 11:43:02 hpjairo.stage7.com systemd[1]: docker.service: Unit entered failed state.
dic 16 11:43:02 hpjairo.stage7.com systemd[1]: docker.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.

> journalctl -xe
See attachment.
Comment 6 Yajo 2015-12-16 11:41:49 UTC 
Oh I got that fixed with http://stackoverflow.com/a/33604859/1468388
Comment 7 Lokesh Mandvekar 2016-02-16 20:00:24 UTC 
Closing this as docker 1.9 is already in stable.