Bug 1399398

Summary: Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: cannot create network docker0
Product: Red Hat Enterprise Linux 7 Reporter: Alex Jia <ajia>
Component: dockerAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED ERRATA QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.4CC: amurdaca, ddarrah, dwalsh, lsm5, lsu
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-17 20:43:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Jia 2016-11-29 01:16:16 UTC
Description of problem:
When upgrade docker from docker-1.12.3-8.el7 to docker-1.10.3-59.el7 by rpm command, we can't successfully start docker daemon, the failed reason is creating the same docker0 bridge conflicts, the docker0 bridge exists in current host, but docker will create it again after restarting/starting docker service.

Version-Release number of selected component (if applicable):

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 (Maipo)

# rpm -q docker kernel selinux-policy
docker-1.10.3-59.el7.x86_64
kernel-3.10.0-514.el7.x86_64
selinux-policy-3.13.1-102.el7_3.4.noarch


How reproducible:
always

Steps to Reproduce:
1. download docker-1.10.3-59 related RPMs
2. rpm -Uvh docker-*.rpm
3. systemctl restart docker or systemctl start docker

Actual results:
[root@dell-per630-02 docker-1.10.3-59.el7]# rpm -Uvh *.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:docker-rhel-push-plugin-2:1.10.3-################################# [  6%]
   2:docker-common-2:1.10.3-59.el7    ################################# [ 13%]
   3:container-selinux-2:1.10.3-59.el7################################# [ 19%]
libsemanage.semanage_direct_remove_key: Unable to remove module docker at priority 400. (No such file or directory).
   4:docker-2:1.10.3-59.el7           ################################# [ 25%]
   5:docker-logrotate-2:1.10.3-59.el7 ################################# [ 31%]
   6:docker-lvm-plugin-2:1.10.3-59.el7################################# [ 38%]
   7:docker-novolume-plugin-2:1.10.3-5################################# [ 44%]
   8:docker-unit-test-2:1.10.3-59.el7 ################################# [ 50%]
Cleaning up / removing...
   9:docker-logrotate-1:1.12.3-8.el7  ################################# [ 56%]
  10:docker-novolume-plugin-1:1.12.3-8################################# [ 63%]
  11:docker-lvm-plugin-1:1.12.3-8.el7 ################################# [ 69%]
  12:docker-1:1.12.3-8.el7            ################################# [ 75%]
  13:container-selinux-1:1.12.3-8.el7 ################################# [ 81%]
  14:docker-common-1:1.12.3-8.el7     ################################# [ 88%]
  15:docker-rhel-push-plugin-1:1.12.3-################################# [ 94%]
  16:docker-unit-test-1:1.12.3-8.el7  ################################# [100%]

[root@dell-per630-02 docker-1.10.3-59.el7]# systemctl start docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

[root@dell-per630-02 docker-1.10.3-59.el7]# systemctl status docker -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2016-11-29 09:13:43 CST; 2s ago
     Docs: http://docs.docker.com
  Process: 102866 ExecStart=/usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 102866 (code=exited, status=1/FAILURE)

Nov 29 09:13:42 dell-per630-02.rhts.eng.pek2.redhat.com systemd[1]: Starting Docker Application Container Engine...
Nov 29 09:13:42 dell-per630-02.rhts.eng.pek2.redhat.com docker-current[102866]: time="2016-11-29T09:13:42.554591873+08:00" level=info msg="[graphdriver] using prior storage driver \"btrfs\""
Nov 29 09:13:42 dell-per630-02.rhts.eng.pek2.redhat.com docker-current[102866]: time="2016-11-29T09:13:42.557137297+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 29 09:13:42 dell-per630-02.rhts.eng.pek2.redhat.com docker-current[102866]: time="2016-11-29T09:13:42.566000931+08:00" level=info msg="Firewalld running: true"
Nov 29 09:13:42 dell-per630-02.rhts.eng.pek2.redhat.com docker-current[102866]: time="2016-11-29T09:13:42.838988867+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Nov 29 09:13:43 dell-per630-02.rhts.eng.pek2.redhat.com docker-current[102866]: time="2016-11-29T09:13:43.268437143+08:00" level=fatal msg="Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: cannot create network docker0 (a7970ab944307186c8ecf489dc614804b50a948f2155a6748b229ba3b2255c39): conflicts with network 4a77788efab9f0cc993f0208ce3046d269ce2e0340c5cc750fbafd430cb07c76 (docker0): networks have same bridge name"
Nov 29 09:13:43 dell-per630-02.rhts.eng.pek2.redhat.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Nov 29 09:13:43 dell-per630-02.rhts.eng.pek2.redhat.com systemd[1]: Failed to start Docker Application Container Engine.
Nov 29 09:13:43 dell-per630-02.rhts.eng.pek2.redhat.com systemd[1]: Unit docker.service entered failed state.
Nov 29 09:13:43 dell-per630-02.rhts.eng.pek2.redhat.com systemd[1]: docker.service failed.




Expected results:

fix it.

Additional info:

There is no AVC denied in audit.log.

# ip link show docker0
13: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT 
    link/ether 02:42:18:f4:b4:69 brd ff:ff:ff:ff:ff:ff

Comment 1 Alex Jia 2016-11-29 01:52:47 UTC
BTW. the docker-latest-1.12.3-10.el7 works well.

[root@dell-per630-02 docker-1.10.3-59.el7]# systemctl start docker-latest

[root@dell-per630-02 docker-1.10.3-59.el7]# systemctl status docker-latest -l
● docker-latest.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker-latest.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-11-29 09:52:04 CST; 5s ago
     Docs: http://docs.docker.com
 Main PID: 104715 (dockerd-latest)
   Memory: 26.6M
   CGroup: /system.slice/docker-latest.service
           ├─104715 /usr/bin/dockerd-latest --add-runtime docker-runc=/usr/libexec/docker/docker-runc --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd -g /var/lib/docker-latest --selinux-enabled --log-driver=journald --add-registry registry.access.redhat.com
           └─104723 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc --runtime-args --systemd-cgroup=true

Nov 29 09:52:03 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: time="2016-11-29T09:52:03.719069954+08:00" level=info msg="[graphdriver] using prior storage driver \"btrfs\""
Nov 29 09:52:03 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: time="2016-11-29T09:52:03.757021128+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 29 09:52:03 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: time="2016-11-29T09:52:03.757672241+08:00" level=info msg="Loading containers: start."
Nov 29 09:52:03 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: .time="2016-11-29T09:52:03.768155204+08:00" level=info msg="Firewalld running: true"
Nov 29 09:52:04 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: time="2016-11-29T09:52:04.179914501+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Nov 29 09:52:04 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: time="2016-11-29T09:52:04.391127593+08:00" level=info msg="Loading containers: done."
Nov 29 09:52:04 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: time="2016-11-29T09:52:04.391176036+08:00" level=info msg="Daemon has completed initialization"
Nov 29 09:52:04 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: time="2016-11-29T09:52:04.391188987+08:00" level=info msg="Docker daemon" commit=9858ec7 graphdriver=btrfs version=1.12.3
Nov 29 09:52:04 dell-per630-02.rhts.eng.pek2.redhat.com dockerd-latest[104715]: time="2016-11-29T09:52:04.400442579+08:00" level=info msg="API listen on /var/run/docker.sock"
Nov 29 09:52:04 dell-per630-02.rhts.eng.pek2.redhat.com systemd[1]: Started Docker Application Container Engine.

Comment 6 Antonio Murdaca 2016-11-29 10:15:26 UTC
Unfortunately though, this is a known issue when skipping incremental upgrades. Meaning, you would first go to 1.11.z and then to 1.12.z in order to solve this issue. Unfortunately again, we won't ship any 1.11.z Docker in rhel afaict.

Relevant upstream issues:

https://github.com/docker/docker/issues/24155
https://github.com/docker/docker/issues/28363

Sadly, data structures in network code changed over versions and the only knowsn

Comment 7 Antonio Murdaca 2016-11-29 10:17:30 UTC
Sadly, data structures in network code changed over versions and the only know workaround is to:

rm -rf /var/lib/docker/network

The above will preserve any containers from 1.10.z while purging network data.

Could we document this as a known issue somewhere in the KnowledgeBase?

Comment 13 errata-xmlrpc 2017-01-17 20:43:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0116.html

Comment 14 Red Hat Bugzilla 2023-09-14 03:35:16 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days