Bug 1331881

Summary: ceph-ansible does not use systemd on Xenial
Product: [Red Hat Storage] Red Hat Storage Console Reporter: Ken Dreyer (Red Hat) <kdreyer>
Component: ceph-installerAssignee: Andrew Schoen <aschoen>
Status: CLOSED ERRATA QA Contact: sds-qe-bugs
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2CC: mkudlej, nthomas, poelstra, racpatel, sankarshan, sds-qe-bugs, tchandra
Target Milestone: ---   
Target Release: 2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-ansible-1.0.5-8.el7scon Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:49:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ken Dreyer (Red Hat) 2016-04-29 22:20:43 UTC 
Description of problem:
When deploying Ceph on Xenial (which uses systemd), the /api/mon/configure endpoint hangs forever. The reason ceph-ansible hangs forever is that it is waiting on the "ceph-mon | collect admin and bootstrap keys" task.

The problem is that ceph-create-keys waits for keys to appear, and the keys do not appear because Ansible has not properly started the ceph-mon service with systemd.

Running `systemctl start ceph-mon@<hostname>` creates the keys and allows Ansible to proceed with the rest of the playbook to configure the monitor.

Version-Release number of selected component (if applicable):
ceph-ansible-1.0.5-5.el7scon.noarch
ceph-installer-1.0.6-1.el7scon.noarch

How reproducible:
always

Steps to Reproduce:
1. Prepare Xenial machines to host Ceph
2. POST to the installer /api/mon/install , /api/osd/install endpoints
3. Verify the package installations succeeded on the cluster nodes.
4. POST to the /api/mon/configure endpoint
5. Check /api/tasks

Actual results:
The /api/mon/configure task hangs. (Running `systemctl start ceph-mon@<hostname>` on the monitor node(s) allows it to continue.)

Expected results:
The /api/mon/configure task succeeds without intervention.

Additional info:
I found this problem in the monitor config step, and we need ceph-ansible to use systemd on Xenial for all services: ceph-mon and ceph-osd.
Comment 2 Ken Dreyer (Red Hat) 2016-05-04 20:57:00 UTC 
As a reminder, until this bug is fixed, if you see "/api/mon/configure" taking longer than 50 seconds, log into the monitor node and run

sudo systemctl start ceph-mon@$(hostname)
Comment 3 Andrew Schoen 2016-05-04 21:22:29 UTC 
I've made a PR that fixes this issue here: https://github.com/ceph/ceph-ansible/pull/758
Comment 4 Ken Dreyer (Red Hat) 2016-05-05 15:31:56 UTC 
*** Bug 1332920 has been marked as a duplicate of this bug. ***
Comment 5 Rachana Patel 2016-05-05 21:43:31 UTC 
Adding mon to existing ceph cluster also fails and looks like due to this bug.
Once this bug is fixed we might re run that to confirm.
Comment 6 Alfredo Deza 2016-05-09 14:11:44 UTC 
*** Bug 1332920 has been marked as a duplicate of this bug. ***
Comment 7 Andrew Schoen 2016-05-09 16:00:15 UTC 
The PR has been merged upstream: https://github.com/ceph/ceph-ansible/pull/758
Comment 9 Andrew Schoen 2016-05-09 19:57:25 UTC 
*** Bug 1333442 has been marked as a duplicate of this bug. ***
Comment 10 Andrew Schoen 2016-05-09 19:58:32 UTC 
*** Bug 1333339 has been marked as a duplicate of this bug. ***
Comment 16 errata-xmlrpc 2016-08-23 19:49:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2016:1754