Bug 1323794

Summary:	Running "atomic install rhel7/etcd" fails to properly set up etcd service
Product:	Red Hat Enterprise Linux 7	Reporter:	Chris Negus <cnegus>
Component:	etcd-container	Assignee:	Avesh Agarwal <avagarwa>
Status:	CLOSED CURRENTRELEASE	QA Contact:	atomic-bugs <atomic-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.2	CC:	ajia
Target Milestone:	rc	Keywords:	Extras
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-09-19 17:08:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Chris Negus 2016-04-04 18:01:58 UTC

Description of problem:
The etcd container (registry.access.redhat.com/rhel7/etcd) fails to install properly as a systemd service from "atomic install rhel7/etcd". Although an etcd systemd service is created and enabled, it doesn't start.


Version-Release number of selected component (if applicable):

Inside the etcd container:
   "Created": "2016-03-01T16:49:38.99184Z"
   "DockerVersion": "1.8.2-el7"
   "Version": "2.2.2"
atomic host:
   2016-03-01 01:35:21     7.2.2-2     8b2cf24b42     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

How reproducible:
Always

Steps to Reproduce:
1. Run "atomic install rhel7/etcd" from an Atomic or RHEL 7.2 host. 
2. Run "systemctl -l etcd_container_etcd.service" to see if etcd is running. (It's enabled, but not running.)
3. Run "systemctl start etcd_container_etcd.service" (which was created from the "atomic install").
4. Run "systemctl -l etcd_container_etcd.service" to check again if etcd is running. It is not.

Actual results:
The "atomic install" creates an /etc/systemd/system/etcd_container_etcd.service file. However, that file fails to start the etcd service automatically. Trying to manually start the service with "systemctl start" also fails.

Expected results:

The etcd service should be running from the container and listening for services on multiple ports. The services fails to start with the following output:

systemctl status -l etcd_container_etcd.service 
● etcd_container_etcd.service - The Etcd Server container
   Loaded: loaded (/etc/systemd/system/etcd_container_etcd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2016-04-04 13:43:18 EDT; 8min ago
  Process: 19850 ExecStop=/usr/bin/docker stop etcd (code=exited, status=1/FAILURE)
  Process: 19846 ExecStart=/usr/bin/docker start etcd (code=exited, status=1/FAILURE)


Additional info:
The "atomic run rhel7/etcd" command seems to work fine. The systemd service set up by "atomic install", however, runs "/usr/bin/docker start etcd". Perhaps the atomic command should be used instead of docker inside the /etc/systemd/system/etcd_container_etcd.service file.

Comment 2 Avesh Agarwal 2016-04-06 17:55:37 UTC

I can reproduce it. Still trying to find what is a better way to fix it. As per https://github.com/projectatomic/atomic, the example there suggests using docker create in the install script, but in the past it had an issue that the container was created without ports mapped to host at the install time as there was no way to pass them, so later the container was run/started without any ports mapped to host. So I am still looking into all options (including using atomic commands inside systemd service template).

Comment 3 Avesh Agarwal 2016-04-07 13:33:19 UTC

Hi Chris,

I have built a new version here:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=489104

Could you test it and let me know your feedback? Thanks.

Comment 4 Alex Jia 2016-04-07 14:14:51 UTC

# docker load -i etcd-docker-2.2.5-3-x86_64.tar.gz 
# docker images|grep etcd
avagarwa/etcd-docker                                  extras-rhel-7.2-docker-candidate-20160407092034   4b4b75af2acd        33 minutes ago      241.7 MB

# atomic install avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034
/usr/bin/docker run --rm --privileged -v /:/host -e HOST=/host -e NAME=etcd-docker -e IMAGE=avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 /usr/bin/install.sh
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
83c2d544cdc1e9422f8abda19886465e3efde48bee1a0c46b3f55656f4ba646c
Full path required for exclude: net:[4026532728].
Full path required for exclude: net:[4026531968].
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd_container_etcd-docker.service to /etc/systemd/system/etcd_container_etcd-docker.service.

# docker ps
CONTAINER ID        IMAGE                                                 COMMAND             CREATED             STATUS              PORTS               NAMES

# atomic run avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034
etcd-docker

# docker ps
CONTAINER ID        IMAGE                                                                  COMMAND                  CREATED             STATUS              PORTS                                                                              NAMES
83c2d544cdc1        avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034   "/usr/bin/etcd-env.sh"   51 seconds ago      Up 3 seconds        0.0.0.0:2379-2380->2379-2380/tcp, 0.0.0.0:4001->4001/tcp, 0.0.0.0:7001->7001/tcp   etcd-docker

# docker exec 83c2d544cdc1 systemctl status -l etcd_container_etcd.service
Failed to get D-Bus connection: Operation not permitted

NOTE: it's not a systemd container, so it's an expected result, right?


# docker exec 83c2d544cdc1 ps auxf|grep etcd
root          1  0.2  0.0  20400  7120 ?        Ssl  10:02   0:00 /usr/bin/etcd

NOTE: the etcd is indeed running.

# rpm -q atomic docker
atomic-1.9-4.gitff44c6a.el7.x86_64
docker-1.9.1-25.el7.x86_64

Comment 5 Avesh Agarwal 2016-04-07 14:28:03 UTC

Alex,

Thanks, For controlling it with systemd, you have to start the service using sysmtemctl:

# systemctl start etcd_container_etcd-docker.service

Please note file is etcd_container_etcd-docker.service not etcd_container_etcd.service in your test.

Also I think you either use atomic or systemd (systemdctl).

Comment 6 Alex Jia 2016-04-07 14:37:35 UTC

(In reply to Avesh Agarwal from comment #5)

> # systemctl start etcd_container_etcd-docker.service
> 
> Please note file is etcd_container_etcd-docker.service not
> etcd_container_etcd.service in your test.
> 
> Also I think you either use atomic or systemd (systemdctl).

I should start etcd-docker service in the Atomic Host, right?

# docker exec 83c2d544cdc1 systemctl start etcd_container_etcd-docker.service
Failed to get D-Bus connection: Operation not permitted

Comment 7 Alex Jia 2016-04-07 14:46:37 UTC

(In reply to Alex Jia from comment #6)

> I should start etcd-docker service in the Atomic Host, right?

s/Atomic Host/etcd-docker container/.

Comment 8 Avesh Agarwal 2016-04-07 14:52:27 UTC

Alex,

No, just systemctl start/stop to start/stop the container.

Comment 9 Avesh Agarwal 2016-04-07 14:53:35 UTC

Alex, you manage the lifecylce of containers either using atomic or systemd.

Comment 10 Alex Jia 2016-04-07 15:21:32 UTC

Okay, I think your new version should work well for me.

# atomic run avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034
etcd-docker

# docker ps
CONTAINER ID        IMAGE                                                                  COMMAND                  CREATED             STATUS              PORTS                                                                              NAMES
83c2d544cdc1        avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034   "/usr/bin/etcd-env.sh"   About an hour ago   Up 10 seconds       0.0.0.0:2379-2380->2379-2380/tcp, 0.0.0.0:4001->4001/tcp, 0.0.0.0:7001->7001/tcp   etcd-docker
d85b56b8b350        docker-registry.usersys.redhat.com/micah/fedora-uat                    "/bin/bash"              9 days ago          Up 9 days                                                                                              mad_bhabha

# systemctl status etcd_container_etcd-docker.service
● etcd_container_etcd-docker.service - The Etcd Server container
   Loaded: loaded (/etc/systemd/system/etcd_container_etcd-docker.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2016-04-07 23:12:26 CST; 43s ago
  Process: 114484 ExecStop=/usr/bin/docker stop etcd-docker (code=exited, status=0/SUCCESS)
  Process: 112552 ExecStart=/usr/bin/docker start -a etcd-docker (code=exited, status=0/SUCCESS)
 Main PID: 112552 (code=exited, status=0/SUCCESS)

Apr 07 22:34:41 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started The Etcd Server container.
Apr 07 22:34:41 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Starting The Etcd Server container...
Apr 07 23:12:25 dell-per630-02.qe.lab.eng.nay.redhat.com docker[112552]: 2016-04-07 11:12:25.917608 N | osutil: received terminated signal, shutting down...
Apr 07 23:12:26 dell-per630-02.qe.lab.eng.nay.redhat.com docker[114484]: etcd-docker

# systemctl start etcd_container_etcd-docker.service
# systemctl status etcd_container_etcd-docker.service
● etcd_container_etcd-docker.service - The Etcd Server container
   Loaded: loaded (/etc/systemd/system/etcd_container_etcd-docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-04-07 23:15:44 CST; 1s ago
  Process: 114484 ExecStop=/usr/bin/docker stop etcd-docker (code=exited, status=0/SUCCESS)
 Main PID: 114614 (docker)
   Memory: 6.6M
   CGroup: /system.slice/etcd_container_etcd-docker.service
           └─114614 /usr/bin/docker start -a etcd-docker

Apr 07 23:15:44 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started The Etcd Server container.
Apr 07 23:15:44 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Starting The Etcd Server container...

NOTE: well, it's okay to try manually start etcd_container_etcd-docker.service on a RHEL7 system.


# atomic stop 83c2d544cdc1
# systemctl status etcd_container_etcd-docker.service
● etcd_container_etcd-docker.service - The Etcd Server container
   Loaded: loaded (/etc/systemd/system/etcd_container_etcd-docker.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2016-04-07 23:15:58 CST; 1s ago
  Process: 114648 ExecStop=/usr/bin/docker stop etcd-docker (code=exited, status=0/SUCCESS)
  Process: 114614 ExecStart=/usr/bin/docker start -a etcd-docker (code=exited, status=0/SUCCESS)
 Main PID: 114614 (code=exited, status=0/SUCCESS)

Apr 07 23:15:44 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started The Etcd Server container.
Apr 07 23:15:44 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Starting The Etcd Server container...
Apr 07 23:15:58 dell-per630-02.qe.lab.eng.nay.redhat.com docker[114614]: 2016-04-07 11:15:58.503717 N | osutil: received terminated signal, shutting down...
Apr 07 23:15:58 dell-per630-02.qe.lab.eng.nay.redhat.com docker[114648]: etcd-docker

NOTE: the etcd_container_etcd-docker.service is inactive after atomic stop the running container.

Comment 11 Chris Negus 2016-09-19 16:35:34 UTC

I just tried the etcd container (registry.access.redhat.com/rhel7/etcd) using the instructions described when I entered this bug and everything works now. Thanks for the fix! I think this bug can be closed now.

Comment 12 Avesh Agarwal 2016-09-19 17:08:31 UTC

Thanks Chris. I will close it now as its been there for quite some time now.