Bug 1457978

Summary: enabling docker socket activation can cause docker-storage-setup to hang forever
Product: Red Hat Enterprise Linux 7 Reporter: Scott Lowe <scott.s.lowe>
Component: container-storage-setupAssignee: Vivek Goyal <vgoyal>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Micah Abbott <miabbott>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: dustymabe
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-14 12:14:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
user-data file for instance creation none

Description Scott Lowe 2017-06-01 16:28:59 UTC 
Description of problem:
When using cloud-init on AWS AMI to customize the Docker daemon (to make it socket-activated), the docker.service systemd unit hangs and won't start. Investigating the problem shows that the docker daemon becomes a child of docker-storage-setup and causes both docker and the docker-storage-setup script to hang indefinitely.

Version-Release number of selected component (if applicable):
1.12.6 (Docker)

How reproducible:
Use cloud-init to make the Docker daemon socket-activated

Steps to Reproduce:
1. Boot an AWS instance using a CentOS Atomic AMI and supply user-data that configures Docker to be socket-activated.
2. Log into the instance and note that docker.service is dead (inactive).

Actual results:
Docker systemd unit hangs and won't restart.

Expected results:
Docker systemd unit would restart and operate as a socket-activated unit.

Additional info:
Comment 1 Dusty Mabe 2017-06-01 17:30:02 UTC 
ok, some more relevant info:

docker-1.12.6-16.el7.centos.x86_64

Scott was setting up some other systemd units on his system to enable docker to be socket activated (will upload user-date file attachment shortly). He was then running a systemd daemon-reload and starting docker again. Since he now has docker socket activated the call to `docker version` in container-storage-setup [1] actually will try to activate docker. This is because `docker version` by default will try to report the version of both the client and the server. Since it tries to access the socket the systemd unit for the docker socket tries to start the docker-daemon, which will attempt to start but will wait forever because docker-storage-setup.service must complete first. 

This means we end up with an infinitely waiting docker because docker storage setup is waiting on it as well:

```
 5779     1 root       20   0 23604  2008  1476 S  0.0  0.0  0:00.01 ├─ /bin/bash /usr/bin/docker-storage-setup
 6088  5779 root       20   0 23604  1048   516 S  0.0  0.0  0:00.00 │  └─ /bin/bash /usr/bin/docker-storage-setup
 6110  6088 root       20   0 23604   920   388 S  0.0  0.0  0:00.00 │     └─ /bin/bash /usr/bin/docker-storage-setup
 6111  6110 root       20   0 23604   920   388 S  0.0  0.0  0:00.00 │        └─ /bin/bash /usr/bin/docker-storage-setup
 6112  6111 root       20   0 23604   944   412 S  0.0  0.0  0:00.00 │           └─ /bin/bash /usr/bin/docker-storage-setup
 6113  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.02 │              └─ /usr/bin/docker-current version --format={{.Client.Version}}
 6212  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6210  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6207  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6206  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6205  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6204  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 └+ /usr/bin/docker-current version --format={{.Client.Version}}
```

We can probably get away from this problem by using `docker -v` to find the version instead. 


[1] https://github.com/projectatomic/container-storage-setup/blob/8276a1e9abf9607e82018c96001858d04420208e/container-storage-setup.sh#L86
Comment 2 Dusty Mabe 2017-06-01 17:31:04 UTC 
Created attachment 1284222 [details]
user-data file for instance creation
Comment 4 Dusty Mabe 2017-06-01 18:23:13 UTC 
it is also worth nothing that i was recreating this using ami-4a70015c from us-east-1, but you should be able to recreate this with any of the images/AMIs from: 2017-05-11 on https://wiki.centos.org/SpecialInterestGroup/Atomic/Download
Comment 5 Vivek Goyal 2020-04-13 21:34:30 UTC 
Is this still an issue. In general we have now moved away from docker-storage-setup.
Comment 6 Scott Lowe 2020-04-13 22:34:12 UTC 
I don't know if this is still a problem or not. Given that this is a three year old issue, it's probably safe to close.