RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1678096 - Docker service hang
Summary: Docker service hang
Keywords:
Status: CLOSED DUPLICATE of bug 1671861
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.6
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: weiwei jiang
URL:
Whiteboard:
: 1684646 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-18 06:02 UTC by Edwin Anthony Joseph
Modified: 2019-04-22 17:08 UTC (History)
10 users (show)

Fixed In Version: docker-1.13.1-94.gitb2f74b2.el7_6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
HPE Synergy Platform Openshift container platform 3.10 on Rhel 7.5 https://github.com/HewlettPackard/hpe-solutions-openshift/tree/master/synergy/entry-level-config/rhhi
Last Closed: 2019-03-06 20:24:43 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Docker Service restart hang (971 bytes, text/plain)
2019-02-18 06:02 UTC, Edwin Anthony Joseph 		no flags Details
View All

Description Edwin Anthony Joseph 2019-02-18 06:02:41 UTC 
Created attachment 1535820 [details]
Docker Service restart hang

Description of problem:
Docker service restart hangs post Openshift install 


Version-Release number of selected component (if applicable):
[root@oshift-fc-m1 ansible_3par_docker_plugin]# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-91.git07f3374.el7.x86_64
 Go version:      go1.10.3
 Git commit:      07f3374/1.13.1
 Built:           Fri Feb  8 20:24:43 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-91.git07f3374.el7.x86_64
 Go version:      go1.10.3
 Git commit:      07f3374/1.13.1
 Built:           Fri Feb  8 20:24:43 2019
 OS/Arch:         linux/amd64
 Experimental:    false
[root@oshift-fc-m1 ansible_3par_docker_plugin]#

[root@oshift-fc-m1 ansible_3par_docker_plugin]# oc version
oc v3.10.101
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://oshift-fc-nlb1.tennet.local:8443
openshift v3.10.101
kubernetes v1.10.0+b81c8f8
[root@oshift-fc-m1 ansible_3par_docker_plugin]#

How reproducible:
Run Ansible play to install openshift prerequisites 

ansible-playbook -e /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml 
Run 
systemctl restart docker.service 
Works fine 
Then install openshift cluster 

ansible-playbook -e /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

systemctl restart docker.service 

docker service fails to respond

Steps to Reproduce:
1.ansible-playbook -e /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

2. systemctl restart docker.service 
3. 

Actual results:

docker service afails to respond 

Expected results:

Docker service restarts cleanly 
Additional info:
Comment 4 weiwei jiang 2019-02-25 08:10:35 UTC 
Checked but the docker in rhel-extra is still 1.13.1.91 version, so waiting the fixed version in this channel for another try.
Move back to modified.

# yum info docker 
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Installed Packages
Name        : docker
Arch        : x86_64
Epoch       : 2
Version     : 1.13.1
Release     : 91.git07f3374.el7
Size        : 64 M
Repo        : installed
From repo   : rhel7-extra
Summary     : Automates deployment of containerized applications
URL         : https://github.com/docker/docker
License     : ASL 2.0
Description : Docker is an open-source engine that automates the deployment of any
            : application as a lightweight, portable, self-sufficient container that will
            : run virtually anywhere.
            : 
            : Docker containers can encapsulate any payload, and will run consistently on
            : and between virtually any server. The same container that a developer builds
            : and tests on a laptop will run at scale, in production*, on VMs, bare-metal
            : servers, OpenStack clusters, public instances, or combinations of the above.
Comment 8 weiwei jiang 2019-02-27 04:57:07 UTC 
verified on docker-1.13.1-94.gitb2f74b2.el7.x86_64.rpm


# systemctl stop docker 
# yum install http://download.eng.bos.redhat.com/brewroot/work/tasks/490/20360490/docker-1.13.1-94.gitb2f74b2.el7.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/490/20360490/docker-client-1.13.1-94.gitb2f74b2.el7.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/490/20360490/docker-common-1.13.1-94.gitb2f74b2.el7.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/490/20360490/docker-rhel-push-plugin-1.13.1-94.gitb2f74b2.el7.x86_64.rpm

[root@qe-wjiang310-mrre-1 ~]# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-94.gitb2f74b2.el7.x86_64
 Go version:      go1.10.8
 Git commit:      b2f74b2/1.13.1
 Built:           Mon Feb 25 14:45:39 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-94.gitb2f74b2.el7.x86_64
 Go version:      go1.10.8
 Git commit:      b2f74b2/1.13.1
 Built:           Mon Feb 25 14:45:39 2019
 OS/Arch:         linux/amd64
 Experimental:    false
[root@qe-wjiang310-mrre-1 ~]# time systemctl restart docker

real	0m20.290s
user	0m0.007s
sys	0m0.018s
[root@qe-wjiang310-mrre-1 ~]# time systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─custom.conf
   Active: active (running) since Tue 2019-02-26 23:54:07 EST; 7s ago
     Docs: http://docs.docker.com
 Main PID: 4676 (dockerd-current)
   CGroup: /system.slice/docker.service
           ├─4676 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-pr...
           ├─4685 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtim...
           ├─5275 /usr/bin/docker-containerd-shim-current 4cba1382e2aa6d1043076406b318b52d149ba18a3de49047476edc7bc3fa8631 /var/run/docker/libcontainerd/4cba1382e2aa6d1043076406b318b52d149ba18a3de49047476edc7bc3fa8631 /usr/libexec/docker/docker-runc-current
           ├─5276 /usr/bin/docker-containerd-shim-current c35dca757d773338e48b3e75a38e6564b5b9cb656dea4b22d121693327385404 /var/run/docker/libcontainerd/c35dca757d773338e48b3e75a38e6564b5b9cb656dea4b22d121693327385404 /usr/libexec/docker/docker-runc-current
           ├─5298 /usr/bin/docker-containerd-shim-current f579de25735ab901c68ca063db43c53316d79b47d7d6ef4be1ad9da40b48726a /var/run/docker/libcontainerd/f579de25735ab901c68ca063db43c53316d79b47d7d6ef4be1ad9da40b48726a /usr/libexec/docker/docker-runc-current
           ├─5346 /usr/bin/docker-containerd-shim-current 15d7eeff3d30223edd58378bea11e05a35266ce284c48f05929b424b74781363 /var/run/docker/libcontainerd/15d7eeff3d30223edd58378bea11e05a35266ce284c48f05929b424b74781363 /usr/libexec/docker/docker-runc-current
           ├─5373 /usr/bin/docker-containerd-shim-current 37279ce5b4411a6a8c01314de3f92c39f7a2f77ad7a827056c3e47817558582e /var/run/docker/libcontainerd/37279ce5b4411a6a8c01314de3f92c39f7a2f77ad7a827056c3e47817558582e /usr/libexec/docker/docker-runc-current
           ├─5440 /usr/bin/docker-containerd-shim-current 165bdf1f870e521971618dce96616d1834d3df6b4c7d320167cf4d9f6e26dc43 /var/run/docker/libcontainerd/165bdf1f870e521971618dce96616d1834d3df6b4c7d320167cf4d9f6e26dc43 /usr/libexec/docker/docker-runc-current
           ├─5647 /usr/bin/docker-containerd-shim-current f55b61d83c0e856c69f4c3caaf91707a1b5e33306398715c6781fffde3df712c /var/run/docker/libcontainerd/f55b61d83c0e856c69f4c3caaf91707a1b5e33306398715c6781fffde3df712c /usr/libexec/docker/docker-runc-current
           └─5734 /usr/bin/docker-containerd-shim-current f66cc016c5f3e96cff171d5137768103e6d9b59e1f76825475e6fa36e61c8348 /var/run/docker/libcontainerd/f66cc016c5f3e96cff171d5137768103e6d9b59e1f76825475e6fa36e61c8348 /usr/libexec/docker/docker-runc-current

Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.122072191-05:00" level=error msg="Handler for POST /v1.26/containers/565bc4c5157d6817d0dd97717ca68a4aafb9f8dadf1f9d51d7d25a83e0f47169/stop returned error: Container ...is already stopped"
Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.316800255-05:00" level=error msg="Handler for POST /v1.26/containers/3c8368a61f7f685b1eed21d0a799741635fb3d8ad6429b560ace9306bb669641/stop?t=10 returned error: Conta...is already stopped"
Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.316958175-05:00" level=error msg="Handler for POST /v1.26/containers/3c8368a61f7f685b1eed21d0a799741635fb3d8ad6429b560ace9306bb669641/stop returned error: Container ...is already stopped"
Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.318791767-05:00" level=error msg="Handler for POST /v1.26/containers/4db0543b6ddd780fa199b2ef48a6287fc9df18d9a0f47208cab0aad50c7f0378/stop?t=10 returned error: Conta...is already stopped"
Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.318937004-05:00" level=error msg="Handler for POST /v1.26/containers/4db0543b6ddd780fa199b2ef48a6287fc9df18d9a0f47208cab0aad50c7f0378/stop returned error: Container ...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:13.143943759-05:00" level=error msg="Handler for POST /v1.26/containers/988b72b04e6f9d197e3d742fdc1d923624614ebaa51a1a73b4b47b30616bb063/stop?t=10 returned error: Conta...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:13.144166320-05:00" level=error msg="Handler for POST /v1.26/containers/988b72b04e6f9d197e3d742fdc1d923624614ebaa51a1a73b4b47b30616bb063/stop returned error: Container ...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:13.150641027-05:00" level=error msg="Handler for POST /v1.26/containers/23d0a4ba7632f672a15b022abf0761327a76b1833c372605d7fe22d367822d19/stop?t=10 returned error: Conta...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:13.150932678-05:00" level=error msg="Handler for POST /v1.26/containers/23d0a4ba7632f672a15b022abf0761327a76b1833c372605d7fe22d367822d19/stop returned error: Container ...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 oci-umount[5764]: umounthook <debug>: prestart container_id:f66cc016c5f3 rootfs:/var/lib/docker/overlay2/239d52e5183c1d58339958ffa90ef768058f1942a4ed09dc593a328bef663a65/merged
Hint: Some lines were ellipsized, use -l to show in full.

real	0m0.046s
user	0m0.003s
sys	0m0.011s

[root@qe-wjiang310-mrre-1 ~]# oc version 
oc v3.10.118
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-wjiang310-mrre-1:8443
openshift v3.10.118
kubernetes v1.10.0+b81c8f8

[root@qe-wjiang310-mrre-1 ~]# oc get pods --all-namespaces -o wide 
NAMESPACE                           NAME                                     READY     STATUS      RESTARTS   AGE       IP            NODE
default                             docker-registry-1-56r58                  1/1       Running     3          1h        10.128.0.43   qe-wjiang310-mrre-1
default                             registry-console-1-88z9b                 1/1       Running     3          1h        10.128.0.42   qe-wjiang310-mrre-1
default                             router-1-fjtrb                           1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
install-test                        mongodb-1-h552k                          1/1       Running     3          1h        10.128.0.40   qe-wjiang310-mrre-1
install-test                        nodejs-mongodb-example-1-build           0/1       Completed   0          1h        10.128.0.14   qe-wjiang310-mrre-1
install-test                        nodejs-mongodb-example-1-zbx6w           1/1       Running     3          1h        10.128.0.38   qe-wjiang310-mrre-1
kube-service-catalog                apiserver-wt72k                          1/1       Running     3          1h        10.128.0.44   qe-wjiang310-mrre-1
kube-service-catalog                controller-manager-l9sf2                 1/1       Running     3          1h        10.128.0.46   qe-wjiang310-mrre-1
kube-system                         master-api-qe-wjiang310-mrre-1           1/1       Running     5          1h        10.0.77.89    qe-wjiang310-mrre-1
kube-system                         master-controllers-qe-wjiang310-mrre-1   1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
kube-system                         master-etcd-qe-wjiang310-mrre-1          1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
openshift-ansible-service-broker    asb-1-x7ph4                              1/1       Running     4          1h        10.128.0.39   qe-wjiang310-mrre-1
openshift-node                      sync-kjlvz                               1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
openshift-sdn                       ovs-6bc2b                                1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
openshift-sdn                       sdn-n6sm2                                1/1       Running     8          1h        10.0.77.89    qe-wjiang310-mrre-1
openshift-template-service-broker   apiserver-85mf4                          1/1       Running     3          1h        10.128.0.45   qe-wjiang310-mrre-1
openshift-web-console               webconsole-785689b664-8bkhn              1/1       Running     3          1h        10.128.0.41   qe-wjiang310-mrre-1
Comment 9 Lars Kellogg-Stedman 2019-03-04 01:57:36 UTC 
*** Bug 1684646 has been marked as a duplicate of this bug. ***
Comment 10 Edwin Anthony Joseph 2019-03-06 05:21:23 UTC 
The under Operating System is 7.5
Comment 11 Edwin Anthony Joseph 2019-03-06 05:23:11 UTC 
[root@ansibleengine ~]# yum list docker
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Available Packages
docker.x86_64                                                                       2:1.13.1-90.git07f3374.el7                                                                        rhel-7-server-extras-rpms
[root@ansibleengine ~]#

so the docker version available through YUM is 1.13.1-90 
is there a way we can verify the finding by installing 1.13.1-94
Note You need to log in before you can comment on or make changes to this bug.