Hide Forgot
Description of problem: The etcd container (registry.access.redhat.com/rhel7/etcd) fails to install properly as a systemd service from "atomic install rhel7/etcd". Although an etcd systemd service is created and enabled, it doesn't start. Version-Release number of selected component (if applicable): Inside the etcd container: "Created": "2016-03-01T16:49:38.99184Z" "DockerVersion": "1.8.2-el7" "Version": "2.2.2" atomic host: 2016-03-01 01:35:21 7.2.2-2 8b2cf24b42 rhel-atomic-host rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard How reproducible: Always Steps to Reproduce: 1. Run "atomic install rhel7/etcd" from an Atomic or RHEL 7.2 host. 2. Run "systemctl -l etcd_container_etcd.service" to see if etcd is running. (It's enabled, but not running.) 3. Run "systemctl start etcd_container_etcd.service" (which was created from the "atomic install"). 4. Run "systemctl -l etcd_container_etcd.service" to check again if etcd is running. It is not. Actual results: The "atomic install" creates an /etc/systemd/system/etcd_container_etcd.service file. However, that file fails to start the etcd service automatically. Trying to manually start the service with "systemctl start" also fails. Expected results: The etcd service should be running from the container and listening for services on multiple ports. The services fails to start with the following output: systemctl status -l etcd_container_etcd.service ● etcd_container_etcd.service - The Etcd Server container Loaded: loaded (/etc/systemd/system/etcd_container_etcd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Mon 2016-04-04 13:43:18 EDT; 8min ago Process: 19850 ExecStop=/usr/bin/docker stop etcd (code=exited, status=1/FAILURE) Process: 19846 ExecStart=/usr/bin/docker start etcd (code=exited, status=1/FAILURE) Additional info: The "atomic run rhel7/etcd" command seems to work fine. The systemd service set up by "atomic install", however, runs "/usr/bin/docker start etcd". Perhaps the atomic command should be used instead of docker inside the /etc/systemd/system/etcd_container_etcd.service file.
I can reproduce it. Still trying to find what is a better way to fix it. As per https://github.com/projectatomic/atomic, the example there suggests using docker create in the install script, but in the past it had an issue that the container was created without ports mapped to host at the install time as there was no way to pass them, so later the container was run/started without any ports mapped to host. So I am still looking into all options (including using atomic commands inside systemd service template).
Hi Chris, I have built a new version here: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=489104 Could you test it and let me know your feedback? Thanks.
# docker load -i etcd-docker-2.2.5-3-x86_64.tar.gz # docker images|grep etcd avagarwa/etcd-docker extras-rhel-7.2-docker-candidate-20160407092034 4b4b75af2acd 33 minutes ago 241.7 MB # atomic install avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 /usr/bin/docker run --rm --privileged -v /:/host -e HOST=/host -e NAME=etcd-docker -e IMAGE=avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 /usr/bin/install.sh Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. 83c2d544cdc1e9422f8abda19886465e3efde48bee1a0c46b3f55656f4ba646c Full path required for exclude: net:[4026532728]. Full path required for exclude: net:[4026531968]. Created symlink from /etc/systemd/system/multi-user.target.wants/etcd_container_etcd-docker.service to /etc/systemd/system/etcd_container_etcd-docker.service. # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES # atomic run avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 etcd-docker # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 83c2d544cdc1 avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 "/usr/bin/etcd-env.sh" 51 seconds ago Up 3 seconds 0.0.0.0:2379-2380->2379-2380/tcp, 0.0.0.0:4001->4001/tcp, 0.0.0.0:7001->7001/tcp etcd-docker # docker exec 83c2d544cdc1 systemctl status -l etcd_container_etcd.service Failed to get D-Bus connection: Operation not permitted NOTE: it's not a systemd container, so it's an expected result, right? # docker exec 83c2d544cdc1 ps auxf|grep etcd root 1 0.2 0.0 20400 7120 ? Ssl 10:02 0:00 /usr/bin/etcd NOTE: the etcd is indeed running. # rpm -q atomic docker atomic-1.9-4.gitff44c6a.el7.x86_64 docker-1.9.1-25.el7.x86_64
Alex, Thanks, For controlling it with systemd, you have to start the service using sysmtemctl: # systemctl start etcd_container_etcd-docker.service Please note file is etcd_container_etcd-docker.service not etcd_container_etcd.service in your test. Also I think you either use atomic or systemd (systemdctl).
(In reply to Avesh Agarwal from comment #5) > # systemctl start etcd_container_etcd-docker.service > > Please note file is etcd_container_etcd-docker.service not > etcd_container_etcd.service in your test. > > Also I think you either use atomic or systemd (systemdctl). I should start etcd-docker service in the Atomic Host, right? # docker exec 83c2d544cdc1 systemctl start etcd_container_etcd-docker.service Failed to get D-Bus connection: Operation not permitted
(In reply to Alex Jia from comment #6) > I should start etcd-docker service in the Atomic Host, right? s/Atomic Host/etcd-docker container/.
Alex, No, just systemctl start/stop to start/stop the container.
Alex, you manage the lifecylce of containers either using atomic or systemd.
Okay, I think your new version should work well for me. # atomic run avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 etcd-docker # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 83c2d544cdc1 avagarwa/etcd-docker:extras-rhel-7.2-docker-candidate-20160407092034 "/usr/bin/etcd-env.sh" About an hour ago Up 10 seconds 0.0.0.0:2379-2380->2379-2380/tcp, 0.0.0.0:4001->4001/tcp, 0.0.0.0:7001->7001/tcp etcd-docker d85b56b8b350 docker-registry.usersys.redhat.com/micah/fedora-uat "/bin/bash" 9 days ago Up 9 days mad_bhabha # systemctl status etcd_container_etcd-docker.service ● etcd_container_etcd-docker.service - The Etcd Server container Loaded: loaded (/etc/systemd/system/etcd_container_etcd-docker.service; enabled; vendor preset: disabled) Active: inactive (dead) since Thu 2016-04-07 23:12:26 CST; 43s ago Process: 114484 ExecStop=/usr/bin/docker stop etcd-docker (code=exited, status=0/SUCCESS) Process: 112552 ExecStart=/usr/bin/docker start -a etcd-docker (code=exited, status=0/SUCCESS) Main PID: 112552 (code=exited, status=0/SUCCESS) Apr 07 22:34:41 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started The Etcd Server container. Apr 07 22:34:41 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Starting The Etcd Server container... Apr 07 23:12:25 dell-per630-02.qe.lab.eng.nay.redhat.com docker[112552]: 2016-04-07 11:12:25.917608 N | osutil: received terminated signal, shutting down... Apr 07 23:12:26 dell-per630-02.qe.lab.eng.nay.redhat.com docker[114484]: etcd-docker # systemctl start etcd_container_etcd-docker.service # systemctl status etcd_container_etcd-docker.service ● etcd_container_etcd-docker.service - The Etcd Server container Loaded: loaded (/etc/systemd/system/etcd_container_etcd-docker.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2016-04-07 23:15:44 CST; 1s ago Process: 114484 ExecStop=/usr/bin/docker stop etcd-docker (code=exited, status=0/SUCCESS) Main PID: 114614 (docker) Memory: 6.6M CGroup: /system.slice/etcd_container_etcd-docker.service └─114614 /usr/bin/docker start -a etcd-docker Apr 07 23:15:44 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started The Etcd Server container. Apr 07 23:15:44 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Starting The Etcd Server container... NOTE: well, it's okay to try manually start etcd_container_etcd-docker.service on a RHEL7 system. # atomic stop 83c2d544cdc1 # systemctl status etcd_container_etcd-docker.service ● etcd_container_etcd-docker.service - The Etcd Server container Loaded: loaded (/etc/systemd/system/etcd_container_etcd-docker.service; enabled; vendor preset: disabled) Active: inactive (dead) since Thu 2016-04-07 23:15:58 CST; 1s ago Process: 114648 ExecStop=/usr/bin/docker stop etcd-docker (code=exited, status=0/SUCCESS) Process: 114614 ExecStart=/usr/bin/docker start -a etcd-docker (code=exited, status=0/SUCCESS) Main PID: 114614 (code=exited, status=0/SUCCESS) Apr 07 23:15:44 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started The Etcd Server container. Apr 07 23:15:44 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Starting The Etcd Server container... Apr 07 23:15:58 dell-per630-02.qe.lab.eng.nay.redhat.com docker[114614]: 2016-04-07 11:15:58.503717 N | osutil: received terminated signal, shutting down... Apr 07 23:15:58 dell-per630-02.qe.lab.eng.nay.redhat.com docker[114648]: etcd-docker NOTE: the etcd_container_etcd-docker.service is inactive after atomic stop the running container.
I just tried the etcd container (registry.access.redhat.com/rhel7/etcd) using the instructions described when I entered this bug and everything works now. Thanks for the fix! I think this bug can be closed now.
Thanks Chris. I will close it now as its been there for quite some time now.