Bug 1678096

Summary:

Docker service hang

Product:

Red Hat Enterprise Linux 7

Reporter:

Edwin Anthony Joseph <eaj>

Component:

docker

Assignee:

Lokesh Mandvekar <lsm5>

Status:

CLOSED DUPLICATE

QA Contact:

weiwei jiang <wjiang>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

7.6

CC:

amurdaca, aos-bugs, dornelas, jokerman, lars, lsm5, mmccomas, mpatel, pasik, wmeng

Target Milestone:

Keywords:

Automation, Extras

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

docker-1.13.1-94.gitb2f74b2.el7_6

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

HPE Synergy Platform Openshift container platform 3.10 on Rhel 7.5 https://github.com/HewlettPackard/hpe-solutions-openshift/tree/master/synergy/entry-level-config/rhhi

Last Closed:

2019-03-06 20:24:43 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Docker Service restart hang	none

Description Edwin Anthony Joseph 2019-02-18 06:02:41 UTC

Created attachment 1535820 [details]
Docker Service restart hang

Description of problem:
Docker service restart hangs post Openshift install 


Version-Release number of selected component (if applicable):
[root@oshift-fc-m1 ansible_3par_docker_plugin]# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-91.git07f3374.el7.x86_64
 Go version:      go1.10.3
 Git commit:      07f3374/1.13.1
 Built:           Fri Feb  8 20:24:43 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-91.git07f3374.el7.x86_64
 Go version:      go1.10.3
 Git commit:      07f3374/1.13.1
 Built:           Fri Feb  8 20:24:43 2019
 OS/Arch:         linux/amd64
 Experimental:    false
[root@oshift-fc-m1 ansible_3par_docker_plugin]#

[root@oshift-fc-m1 ansible_3par_docker_plugin]# oc version
oc v3.10.101
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://oshift-fc-nlb1.tennet.local:8443
openshift v3.10.101
kubernetes v1.10.0+b81c8f8
[root@oshift-fc-m1 ansible_3par_docker_plugin]#

How reproducible:
Run Ansible play to install openshift prerequisites 

ansible-playbook -e /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml 
Run 
systemctl restart docker.service 
Works fine 
Then install openshift cluster 

ansible-playbook -e /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

systemctl restart docker.service 

docker service fails to respond

Steps to Reproduce:
1.ansible-playbook -e /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

2. systemctl restart docker.service 
3. 

Actual results:

docker service afails to respond 

Expected results:

Docker service restarts cleanly 
Additional info:

Comment 4 weiwei jiang 2019-02-25 08:10:35 UTC

Checked but the docker in rhel-extra is still 1.13.1.91 version, so waiting the fixed version in this channel for another try.
Move back to modified.

# yum info docker 
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Installed Packages
Name        : docker
Arch        : x86_64
Epoch       : 2
Version     : 1.13.1
Release     : 91.git07f3374.el7
Size        : 64 M
Repo        : installed
From repo   : rhel7-extra
Summary     : Automates deployment of containerized applications
URL         : https://github.com/docker/docker
License     : ASL 2.0
Description : Docker is an open-source engine that automates the deployment of any
            : application as a lightweight, portable, self-sufficient container that will
            : run virtually anywhere.
            : 
            : Docker containers can encapsulate any payload, and will run consistently on
            : and between virtually any server. The same container that a developer builds
            : and tests on a laptop will run at scale, in production*, on VMs, bare-metal
            : servers, OpenStack clusters, public instances, or combinations of the above.

Comment 8 weiwei jiang 2019-02-27 04:57:07 UTC

verified on docker-1.13.1-94.gitb2f74b2.el7.x86_64.rpm


# systemctl stop docker 
# yum install http://download.eng.bos.redhat.com/brewroot/work/tasks/490/20360490/docker-1.13.1-94.gitb2f74b2.el7.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/490/20360490/docker-client-1.13.1-94.gitb2f74b2.el7.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/490/20360490/docker-common-1.13.1-94.gitb2f74b2.el7.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/490/20360490/docker-rhel-push-plugin-1.13.1-94.gitb2f74b2.el7.x86_64.rpm

[root@qe-wjiang310-mrre-1 ~]# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-94.gitb2f74b2.el7.x86_64
 Go version:      go1.10.8
 Git commit:      b2f74b2/1.13.1
 Built:           Mon Feb 25 14:45:39 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-94.gitb2f74b2.el7.x86_64
 Go version:      go1.10.8
 Git commit:      b2f74b2/1.13.1
 Built:           Mon Feb 25 14:45:39 2019
 OS/Arch:         linux/amd64
 Experimental:    false
[root@qe-wjiang310-mrre-1 ~]# time systemctl restart docker

real	0m20.290s
user	0m0.007s
sys	0m0.018s
[root@qe-wjiang310-mrre-1 ~]# time systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─custom.conf
   Active: active (running) since Tue 2019-02-26 23:54:07 EST; 7s ago
     Docs: http://docs.docker.com
 Main PID: 4676 (dockerd-current)
   CGroup: /system.slice/docker.service
           ├─4676 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-pr...
           ├─4685 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtim...
           ├─5275 /usr/bin/docker-containerd-shim-current 4cba1382e2aa6d1043076406b318b52d149ba18a3de49047476edc7bc3fa8631 /var/run/docker/libcontainerd/4cba1382e2aa6d1043076406b318b52d149ba18a3de49047476edc7bc3fa8631 /usr/libexec/docker/docker-runc-current
           ├─5276 /usr/bin/docker-containerd-shim-current c35dca757d773338e48b3e75a38e6564b5b9cb656dea4b22d121693327385404 /var/run/docker/libcontainerd/c35dca757d773338e48b3e75a38e6564b5b9cb656dea4b22d121693327385404 /usr/libexec/docker/docker-runc-current
           ├─5298 /usr/bin/docker-containerd-shim-current f579de25735ab901c68ca063db43c53316d79b47d7d6ef4be1ad9da40b48726a /var/run/docker/libcontainerd/f579de25735ab901c68ca063db43c53316d79b47d7d6ef4be1ad9da40b48726a /usr/libexec/docker/docker-runc-current
           ├─5346 /usr/bin/docker-containerd-shim-current 15d7eeff3d30223edd58378bea11e05a35266ce284c48f05929b424b74781363 /var/run/docker/libcontainerd/15d7eeff3d30223edd58378bea11e05a35266ce284c48f05929b424b74781363 /usr/libexec/docker/docker-runc-current
           ├─5373 /usr/bin/docker-containerd-shim-current 37279ce5b4411a6a8c01314de3f92c39f7a2f77ad7a827056c3e47817558582e /var/run/docker/libcontainerd/37279ce5b4411a6a8c01314de3f92c39f7a2f77ad7a827056c3e47817558582e /usr/libexec/docker/docker-runc-current
           ├─5440 /usr/bin/docker-containerd-shim-current 165bdf1f870e521971618dce96616d1834d3df6b4c7d320167cf4d9f6e26dc43 /var/run/docker/libcontainerd/165bdf1f870e521971618dce96616d1834d3df6b4c7d320167cf4d9f6e26dc43 /usr/libexec/docker/docker-runc-current
           ├─5647 /usr/bin/docker-containerd-shim-current f55b61d83c0e856c69f4c3caaf91707a1b5e33306398715c6781fffde3df712c /var/run/docker/libcontainerd/f55b61d83c0e856c69f4c3caaf91707a1b5e33306398715c6781fffde3df712c /usr/libexec/docker/docker-runc-current
           └─5734 /usr/bin/docker-containerd-shim-current f66cc016c5f3e96cff171d5137768103e6d9b59e1f76825475e6fa36e61c8348 /var/run/docker/libcontainerd/f66cc016c5f3e96cff171d5137768103e6d9b59e1f76825475e6fa36e61c8348 /usr/libexec/docker/docker-runc-current

Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.122072191-05:00" level=error msg="Handler for POST /v1.26/containers/565bc4c5157d6817d0dd97717ca68a4aafb9f8dadf1f9d51d7d25a83e0f47169/stop returned error: Container ...is already stopped"
Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.316800255-05:00" level=error msg="Handler for POST /v1.26/containers/3c8368a61f7f685b1eed21d0a799741635fb3d8ad6429b560ace9306bb669641/stop?t=10 returned error: Conta...is already stopped"
Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.316958175-05:00" level=error msg="Handler for POST /v1.26/containers/3c8368a61f7f685b1eed21d0a799741635fb3d8ad6429b560ace9306bb669641/stop returned error: Container ...is already stopped"
Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.318791767-05:00" level=error msg="Handler for POST /v1.26/containers/4db0543b6ddd780fa199b2ef48a6287fc9df18d9a0f47208cab0aad50c7f0378/stop?t=10 returned error: Conta...is already stopped"
Feb 26 23:54:11 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:11.318937004-05:00" level=error msg="Handler for POST /v1.26/containers/4db0543b6ddd780fa199b2ef48a6287fc9df18d9a0f47208cab0aad50c7f0378/stop returned error: Container ...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:13.143943759-05:00" level=error msg="Handler for POST /v1.26/containers/988b72b04e6f9d197e3d742fdc1d923624614ebaa51a1a73b4b47b30616bb063/stop?t=10 returned error: Conta...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:13.144166320-05:00" level=error msg="Handler for POST /v1.26/containers/988b72b04e6f9d197e3d742fdc1d923624614ebaa51a1a73b4b47b30616bb063/stop returned error: Container ...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:13.150641027-05:00" level=error msg="Handler for POST /v1.26/containers/23d0a4ba7632f672a15b022abf0761327a76b1833c372605d7fe22d367822d19/stop?t=10 returned error: Conta...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 dockerd-current[4676]: time="2019-02-26T23:54:13.150932678-05:00" level=error msg="Handler for POST /v1.26/containers/23d0a4ba7632f672a15b022abf0761327a76b1833c372605d7fe22d367822d19/stop returned error: Container ...is already stopped"
Feb 26 23:54:13 qe-wjiang310-mrre-1 oci-umount[5764]: umounthook <debug>: prestart container_id:f66cc016c5f3 rootfs:/var/lib/docker/overlay2/239d52e5183c1d58339958ffa90ef768058f1942a4ed09dc593a328bef663a65/merged
Hint: Some lines were ellipsized, use -l to show in full.

real	0m0.046s
user	0m0.003s
sys	0m0.011s

[root@qe-wjiang310-mrre-1 ~]# oc version 
oc v3.10.118
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-wjiang310-mrre-1:8443
openshift v3.10.118
kubernetes v1.10.0+b81c8f8

[root@qe-wjiang310-mrre-1 ~]# oc get pods --all-namespaces -o wide 
NAMESPACE                           NAME                                     READY     STATUS      RESTARTS   AGE       IP            NODE
default                             docker-registry-1-56r58                  1/1       Running     3          1h        10.128.0.43   qe-wjiang310-mrre-1
default                             registry-console-1-88z9b                 1/1       Running     3          1h        10.128.0.42   qe-wjiang310-mrre-1
default                             router-1-fjtrb                           1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
install-test                        mongodb-1-h552k                          1/1       Running     3          1h        10.128.0.40   qe-wjiang310-mrre-1
install-test                        nodejs-mongodb-example-1-build           0/1       Completed   0          1h        10.128.0.14   qe-wjiang310-mrre-1
install-test                        nodejs-mongodb-example-1-zbx6w           1/1       Running     3          1h        10.128.0.38   qe-wjiang310-mrre-1
kube-service-catalog                apiserver-wt72k                          1/1       Running     3          1h        10.128.0.44   qe-wjiang310-mrre-1
kube-service-catalog                controller-manager-l9sf2                 1/1       Running     3          1h        10.128.0.46   qe-wjiang310-mrre-1
kube-system                         master-api-qe-wjiang310-mrre-1           1/1       Running     5          1h        10.0.77.89    qe-wjiang310-mrre-1
kube-system                         master-controllers-qe-wjiang310-mrre-1   1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
kube-system                         master-etcd-qe-wjiang310-mrre-1          1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
openshift-ansible-service-broker    asb-1-x7ph4                              1/1       Running     4          1h        10.128.0.39   qe-wjiang310-mrre-1
openshift-node                      sync-kjlvz                               1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
openshift-sdn                       ovs-6bc2b                                1/1       Running     6          1h        10.0.77.89    qe-wjiang310-mrre-1
openshift-sdn                       sdn-n6sm2                                1/1       Running     8          1h        10.0.77.89    qe-wjiang310-mrre-1
openshift-template-service-broker   apiserver-85mf4                          1/1       Running     3          1h        10.128.0.45   qe-wjiang310-mrre-1
openshift-web-console               webconsole-785689b664-8bkhn              1/1       Running     3          1h        10.128.0.41   qe-wjiang310-mrre-1

Comment 9 Lars Kellogg-Stedman 2019-03-04 01:57:36 UTC

*** Bug 1684646 has been marked as a duplicate of this bug. ***

Comment 10 Edwin Anthony Joseph 2019-03-06 05:21:23 UTC

The under Operating System is 7.5

Comment 11 Edwin Anthony Joseph 2019-03-06 05:23:11 UTC

[root@ansibleengine ~]# yum list docker
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Available Packages
docker.x86_64                                                                       2:1.13.1-90.git07f3374.el7                                                                        rhel-7-server-extras-rpms
[root@ansibleengine ~]#

so the docker version available through YUM is 1.13.1-90 
is there a way we can verify the finding by installing 1.13.1-94