RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1447536 - Installation failed due to docker.service restart failed
Summary: Installation failed due to docker.service restart failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 7.3
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-03 06:33 UTC by Wenkai Shi
Modified: 2019-03-06 02:41 UTC (History)
14 users (show)

Fixed In Version: docker-1.12.6-25.git62520c0.el7_3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-26 14:52:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1325 0 normal SHIPPED_LIVE docker bug fix and enhancement update 2017-05-26 18:14:24 UTC

Description Wenkai Shi 2017-05-03 06:33:50 UTC
Description of problem:
Installation failed due to docker.service restart failed

Version-Release number of selected component (if applicable):
openshift-ansible-3.6.51-1.git.0.18eb563.el7.noarch
docker-1.12.6-19.git92b10e4.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.install OCP 
2.
3.

Actual results:
# ansible-playbook -i hosts -v /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
...
RUNNING HANDLER [openshift_node_certificates : restart docker after updating ca trust] ***
Wednesday 03 May 2017  06:03:34 +0000 (0:00:01.872)       0:13:11.561 ********* 
fatal: [openshift.example.com]: FAILED! => {
    "changed": false, 
    "failed": true
}

MSG:

Unable to restart service docker: Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.


Expected results:
Installation succeed

Additional info:
# systemctl status docker -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2017-05-03 02:03:35 EDT; 6min ago
     Docs: http://docs.docker.com
  Process: 19059 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 19059 (code=exited, status=1/FAILURE)

May 03 02:03:35 openshift.example.com systemd[1]: Starting Docker Application Container Engine...
May 03 02:03:35 openshift.example.com dockerd-current[19059]: time="2017-05-03T02:03:35-04:00" level=fatal msg="unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: log-driver: (from flag: journald, from file: journald)\n"
May 03 02:03:35 openshift.example.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
May 03 02:03:35 openshift.example.com systemd[1]: Failed to start Docker Application Container Engine.
May 03 02:03:35 openshift.example.com systemd[1]: Unit docker.service entered failed state.
May 03 02:03:35 openshift.example.com systemd[1]: docker.service failed.

# cat /etc/docker/daemon.json 
{
    "log-driver": "journald",
    "signature-verification": false
}

# yum history info 3
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Transaction ID : 3
Begin time     : Wed May  3 01:49:07 2017
Begin rpmdb    : 389:a10a1a5093d4b61e6dc2f0a4ccb32b203ab3aae5
End time       :            01:49:54 2017 (47 seconds)
End rpmdb      : 399:773f0b093894c110d626cf831f759b57787005ad
User           : root <root>
Return-Code    : Success
Command Line   : -d 2 -y install docker
Transaction performed with:
    Installed     rpm-4.11.3-21.el7.x86_64                  @anaconda/7.3
    Installed     subscription-manager-1.17.15-1.el7.x86_64 @anaconda/7.3
    Installed     yum-3.4.3-150.el7.noarch                  @anaconda/7.3
Packages Altered:
    Dep-Install container-selinux-2:2.10-3.el7.noarch                     @rhel7-extra
    Install     docker-2:1.12.6-19.git92b10e4.el7.x86_64                  @rhel7-extra
    Dep-Install docker-client-2:1.12.6-19.git92b10e4.el7.x86_64           @rhel7-extra
    Dep-Install docker-common-2:1.12.6-19.git92b10e4.el7.x86_64           @rhel7-extra
    Dep-Install docker-rhel-push-plugin-2:1.12.6-19.git92b10e4.el7.x86_64 @rhel7-extra
    Dep-Install libseccomp-2.3.1-2.el7.x86_64                             @rhel7
    Dep-Install oci-register-machine-1:0-3.11.gitdd0daef.el7.x86_64       @rhel7-extra
    Dep-Install oci-systemd-hook-1:0.1.7-2.git2788078.el7.x86_64          @rhel7-extra
    Dep-Install skopeo-containers-1:0.1.19-1.el7.x86_64                   @rhel7-extra
    Dep-Install yajl-2.0.4-4.el7.x86_64                                   @rhel7
Scriptlet output:
   1 Cannot connect to the Docker daemon. Is the docker daemon running on this host?
   2 "docker stop" requires at least 1 argument(s).
   3 See 'docker stop --help'.
   4 
   5 Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
   6 
   7 Stop one or more running containers
history info

Comment 1 Gan Huang 2017-05-04 02:37:02 UTC
This is because openshift-ansible set "--log-driver=journald" by default in /etc/sysconfig/docker which is conflicted with the setting in /etc/docker/daemon.json in docker-1.12.6-19.

# grep "log-driver" /etc/docker/daemon.json /etc/sysconfig/docker
/etc/docker/daemon.json:    "log-driver": "journald",
/etc/sysconfig/docker:#OPTIONS=' --selinux-enabled  --log-driver=journald'

Removed the setting in /etc/sysconfig/docker, docker can be restarted.

Comment 2 Gan Huang 2017-05-04 02:39:04 UTC
Sorry, the output should be in this issue:

# grep "log-driver" /etc/docker/daemon.json /etc/sysconfig/docker
/etc/docker/daemon.json:    "log-driver": "journald",
/etc/sysconfig/docker:OPTIONS=' --selinux-enabled  --log-driver=journald'

Comment 3 Scott Dodson 2017-05-04 13:31:24 UTC
Jhon,

This looks like a file that's new to 1.12.6-19, is this a bug in docker? It'll be a big challenge for the installer to determine which release of docker within 1.12.6 needs the option in --log-driver=journald or not.

--
Scott

Comment 5 Lokesh Mandvekar 2017-05-06 02:39:44 UTC
(In reply to Scott Dodson from comment #3)
> Jhon,
> 
> This looks like a file that's new to 1.12.6-19, is this a bug in docker?
> It'll be a big challenge for the installer to determine which release of
> docker within 1.12.6 needs the option in --log-driver=journald or not.
> 
> --
> Scott

Couple of things..

1. Is it hard to include the release tag for docker in the installer?

2. I could move back the --log-driver option back to /etc/sysconfig/docker until openshift is in a safe enough position to use it from /etc/docker/daemon.json.

Let me know.

Comment 6 Scott Dodson 2017-05-08 13:05:13 UTC
Is /etc/docker/daemon.json honored by all versions of docker? We could switch to using that, but keep in mind that we can only migrate to that when someone runs the installer, so if someone where to install docker-1.12.6-16 and then upgrade to docker-1.12.6-19 without using the installer they'd break their environment.

Comment 7 Steve Milner 2017-05-08 13:34:21 UTC
(In reply to Scott Dodson from comment #6)
> Is /etc/docker/daemon.json honored by all versions of docker? We could
> switch to using that, but keep in mind that we can only migrate to that when
> someone runs the installer, so if someone where to install docker-1.12.6-16
> and then upgrade to docker-1.12.6-19 without using the installer they'd
> break their environment.

I believe it is available in all of 1.12.x.

Comment 10 Scott Dodson 2017-05-10 13:27:20 UTC
Lokesh,

Can you just make /etc/docker/daemon.json not include any options in the RPM? By including things there that get created by simply upgrading the package this is a regression.

We'll start moving the installer to using /etc/dokcer/daemon.json rather than command line args.

Comment 11 Lokesh Mandvekar 2017-05-10 13:32:34 UTC
(In reply to Scott Dodson from comment #10)
> Lokesh,
> 
> Can you just make /etc/docker/daemon.json not include any options in the
> RPM? By including things there that get created by simply upgrading the
> package this is a regression.
> 
> We'll start moving the installer to using /etc/dokcer/daemon.json rather
> than command line args.

Sure thing. Do you need this done for docker-latest as well? Or just docker?

Comment 14 Scott Dodson 2017-05-10 15:36:29 UTC
Just docker, we'll make sure that before we switch to docker-1.13 we've fully migrated to daemon.json so I don't think you need to do anything to docker-latest.

Comment 15 Ed Santiago 2017-05-10 19:53:15 UTC
docker-1.12.6-24.git62520c0.el7 ships an empty /etc/docker/daemon.json, resulting in:

    # systemctl status docker
    ● docker.service - Docker Application Container Engine
    ...
       Active: failed (Result: exit-code) since Wed 2017-05-10 15:47:33 EDT; 15s ago
    ...
    May 10 15:47:33 esm-rhel7-d12-3.localdomain dockerd-current[21473]: time="2017-05-10T15:47:33-04:00" level=fatal msg="unable to configure the Docker daemon with file /etc/docker/daemon.json: EOF\n"

Comment 16 Steve Milner 2017-05-10 20:30:31 UTC
daemon.json looks to be an empty file. My guess is it needs to be valid json. I believe using {} should be sufficient.

Comment 17 Ed Santiago 2017-05-11 15:53:35 UTC
(In reply to Steve Milner from comment #16)
> daemon.json looks to be an empty file. My guess is it needs to be valid
> json. I believe using {} should be sufficient.

Confirmed:

    # echo '{}' >| /etc/docker/daemon.json
    # systemctl start docker   -> all ok

Comment 19 Ed Santiago 2017-05-11 17:41:18 UTC
docker-1.12.6-25.git62520c0.el7 - confirmed, looks good.

I've filed bug 1450172 to address warning messages seen during upgrade.

Comment 21 Luwen Su 2017-05-15 09:29:34 UTC
Per comment19 move to verified.

Comment 23 errata-xmlrpc 2017-05-26 14:52:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1325


Note You need to log in before you can comment on or make changes to this bug.