1457978 – enabling docker socket activation can cause docker-storage-setup to hang forever

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1457978 - enabling docker socket activation can cause docker-storage-setup to hang forever

Summary: enabling docker socket activation can cause docker-storage-setup to hang forever

Keywords:
Status:	CLOSED INSUFFICIENT_DATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	container-storage-setup
Sub Component:
Version:	7.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Vivek Goyal
QA Contact:	Micah Abbott
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-06-01 16:28 UTC by Scott Lowe
Modified:	2020-04-14 12:14 UTC (History)
CC List:	1 user (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-04-14 12:14:32 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
user-data file for instance creation (1.76 KB, text/plain) 2017-06-01 17:31 UTC, Dusty Mabe	no flags	Details
View All

Description Scott Lowe 2017-06-01 16:28:59 UTC

Description of problem:
When using cloud-init on AWS AMI to customize the Docker daemon (to make it socket-activated), the docker.service systemd unit hangs and won't start. Investigating the problem shows that the docker daemon becomes a child of docker-storage-setup and causes both docker and the docker-storage-setup script to hang indefinitely.

Version-Release number of selected component (if applicable):
1.12.6 (Docker)

How reproducible:
Use cloud-init to make the Docker daemon socket-activated

Steps to Reproduce:
1. Boot an AWS instance using a CentOS Atomic AMI and supply user-data that configures Docker to be socket-activated.
2. Log into the instance and note that docker.service is dead (inactive).

Actual results:
Docker systemd unit hangs and won't restart.

Expected results:
Docker systemd unit would restart and operate as a socket-activated unit.

Additional info:

Comment 1 Dusty Mabe 2017-06-01 17:30:02 UTC

ok, some more relevant info:

docker-1.12.6-16.el7.centos.x86_64

Scott was setting up some other systemd units on his system to enable docker to be socket activated (will upload user-date file attachment shortly). He was then running a systemd daemon-reload and starting docker again. Since he now has docker socket activated the call to `docker version` in container-storage-setup [1] actually will try to activate docker. This is because `docker version` by default will try to report the version of both the client and the server. Since it tries to access the socket the systemd unit for the docker socket tries to start the docker-daemon, which will attempt to start but will wait forever because docker-storage-setup.service must complete first. 

This means we end up with an infinitely waiting docker because docker storage setup is waiting on it as well:

```
 5779     1 root       20   0 23604  2008  1476 S  0.0  0.0  0:00.01 ├─ /bin/bash /usr/bin/docker-storage-setup
 6088  5779 root       20   0 23604  1048   516 S  0.0  0.0  0:00.00 │  └─ /bin/bash /usr/bin/docker-storage-setup
 6110  6088 root       20   0 23604   920   388 S  0.0  0.0  0:00.00 │     └─ /bin/bash /usr/bin/docker-storage-setup
 6111  6110 root       20   0 23604   920   388 S  0.0  0.0  0:00.00 │        └─ /bin/bash /usr/bin/docker-storage-setup
 6112  6111 root       20   0 23604   944   412 S  0.0  0.0  0:00.00 │           └─ /bin/bash /usr/bin/docker-storage-setup
 6113  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.02 │              └─ /usr/bin/docker-current version --format={{.Client.Version}}
 6212  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6210  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6207  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6206  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6205  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 ├─ /usr/bin/docker-current version --format={{.Client.Version}}
 6204  6112 root       20   0  270M 10956  5324 S  0.0  0.2  0:00.00 │                 └+ /usr/bin/docker-current version --format={{.Client.Version}}
```

We can probably get away from this problem by using `docker -v` to find the version instead. 


[1] https://github.com/projectatomic/container-storage-setup/blob/8276a1e9abf9607e82018c96001858d04420208e/container-storage-setup.sh#L86

Comment 2 Dusty Mabe 2017-06-01 17:31:04 UTC

Created attachment 1284222 [details]
user-data file for instance creation

Comment 4 Dusty Mabe 2017-06-01 18:23:13 UTC

it is also worth nothing that i was recreating this using ami-4a70015c from us-east-1, but you should be able to recreate this with any of the images/AMIs from: 2017-05-11 on https://wiki.centos.org/SpecialInterestGroup/Atomic/Download

Comment 5 Vivek Goyal 2020-04-13 21:34:30 UTC

Is this still an issue. In general we have now moved away from docker-storage-setup.

Comment 6 Scott Lowe 2020-04-13 22:34:12 UTC

I don't know if this is still a problem or not. Given that this is a three year old issue, it's probably safe to close.

Note You need to log in before you can comment on or make changes to this bug.