Bug 1361228 - [ceph-ansible] : Installation fails and exist in Task 'activate OSD(s)' if OSDS are directory (and path is other than '/var/lib/ceph/osd/*')
Summary: [ceph-ansible] : Installation fails and exist in Task 'activate OSD(s)' if O...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Storage Console
Classification: Red Hat
Component: ceph-ansible
Version: 2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3
Assignee: Christina Meno
QA Contact: ceph-qe-bugs
Bara Ancincova
URL:
Whiteboard:
Depends On:
Blocks: 1322504 1383917 1412948
TreeView+ depends on / blocked
 
Reported: 2016-07-28 14:44 UTC by Rachana Patel
Modified: 2017-05-23 20:33 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
.Ansible fails to install OSDs if they point to directories Ansible does not support installation of OSDs that point to directories and not to partitions. As a consequence, an attempt to install such OSDs fails.
Clone Of:
Environment:
Last Closed: 2017-03-03 17:12:30 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 20056 None None None 2017-05-23 20:33:12 UTC
Red Hat Bugzilla 1360937 None CLOSED [ceph-ansible] : purge cluster fails in task 'check for a device list' when osd is Directory 2019-01-16 10:37:25 UTC

Internal Links: 1360937

Description Rachana Patel 2016-07-28 14:44:07 UTC
Description of problem:
======================
Installation using ceph-ansible fails in task  'activate OSD(s)' if OSDs are Directory

if OSDS are directories '/var/lib/ceph/osd/*' then it creates cluster


Version-Release number of selected component (if applicable):
==============================================================
10.2.2-29.el7cp.x86_64
ceph-ansible-1.0.5-31.el7scon.noarch

How reproducible:
==================
always


Steps to Reproduce:
===================
1. update host file and other config file as below

host:-
[mons]
magna008
[osds]
magna051
magna057
magna078

fetch_directory: rhel31/
ceph_stable: true # use ceph stable branch
ceph_stable_rh_storage: true
ceph_stable_rh_storage_iso_install: true # usually used when nodes don't have access to cdn.redhat.com
ceph_stable_rh_storage_iso_path: /root/new/in/Ceph-2.0-RHEL-7-20160725.t.0-x86_64-dvd.iso
ceph_stable_rh_storage_mount_path: /home/c1/iso/abc/rh-storage-mount
ceph_stable_rh_storage_repository_path: /home/c1/abc/iso/rh-storage-repo # where to copy iso's content
cephx: true
monitor_interface: eno1
journal_size: 100
public_network: XXXX/21


osds

crush_location: false
osd_crush_location: "'root={{ ceph_crush_root }} rack={{ ceph_crush_rack }} host={{ ansible_hostname }}'"
osd_directory: true
osd_directories:
  - /var/lib/ceph/new/od/mydir1
  - /home/c1/new/od/mydir2
  - /home/ubuntu/new/od/mydir3
  - /root/new/od/mydir4

2. [root@magna044 ceph-ansible]#   ansible-playbook site.yml -u c1 -i /etc/ansible/ubuntu31 --verbose


Actual results:
==============
TASK: [ceph-osd | activate OSD(s)] ******************************************** 
ok: [magna051] => (item=/var/lib/ceph/new/od/mydir1) => {"changed": false, "cmd": ["ceph-disk", "activate", "/var/lib/ceph/new/od/mydir1"], "delta": "0:00:04.861051", "end": "2016-07-27 21:41:36.129559", "item": "/var/lib/ceph/new/od/mydir1", "rc": 0, "start": "2016-07-27 21:41:31.268508", "stderr": "got monmap epoch 1\nadded key for osd.0\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd@0.service to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []}
ok: [magna057] => (item=/var/lib/ceph/new/od/mydir1) => {"changed": false, "cmd": ["ceph-disk", "activate", "/var/lib/ceph/new/od/mydir1"], "delta": "0:00:05.928685", "end": "2016-07-27 21:41:35.283594", "item": "/var/lib/ceph/new/od/mydir1", "rc": 0, "start": "2016-07-27 21:41:29.354909", "stderr": "got monmap epoch 1\nadded key for osd.1\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd@1.service to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []}
ok: [magna078] => (item=/var/lib/ceph/new/od/mydir1) => {"changed": false, "cmd": ["ceph-disk", "activate", "/var/lib/ceph/new/od/mydir1"], "delta": "0:00:05.941504", "end": "2016-07-27 21:41:39.133965", "item": "/var/lib/ceph/new/od/mydir1", "rc": 0, "start": "2016-07-27 21:41:33.192461", "stderr": "got monmap epoch 1\nadded key for osd.2\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd@2.service to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []}
failed: [magna051] => (item=/home/c1/new/od/mydir2) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/c1/new/od/mydir2"], "delta": "0:00:01.154836", "end": "2016-07-27 21:41:37.510033", "item": "/home/c1/new/od/mydir2", "rc": 1, "start": "2016-07-27 21:41:36.355197", "warnings": []}
stderr: got monmap epoch 1
ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '3', '--monmap', '/home/c1/new/od/mydir2/activate.monmap', '--osd-data', '/home/c1/new/od/mydir2', '--osd-journal', '/home/c1/new/od/mydir2/journal', '--osd-uuid', '99b3eb35-806b-4af6-88f7-a03221e8b14f', '--keyring', '/home/c1/new/od/mydir2/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/c1/new/od/mydir2/activate.monmap: (13) Permission denied
failed: [magna057] => (item=/home/c1/new/od/mydir2) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/c1/new/od/mydir2"], "delta": "0:00:01.342660", "end": "2016-07-27 21:41:36.802715", "item": "/home/c1/new/od/mydir2", "rc": 1, "start": "2016-07-27 21:41:35.460055", "warnings": []}
stderr: got monmap epoch 1
ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '4', '--monmap', '/home/c1/new/od/mydir2/activate.monmap', '--osd-data', '/home/c1/new/od/mydir2', '--osd-journal', '/home/c1/new/od/mydir2/journal', '--osd-uuid', 'e2f11952-da43-4e98-8fdb-09ce8fb1914e', '--keyring', '/home/c1/new/od/mydir2/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/c1/new/od/mydir2/activate.monmap: (13) Permission denied
failed: [magna051] => (item=/home/ubuntu/new/od/mydir3) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/ubuntu/new/od/mydir3"], "delta": "0:00:01.125425", "end": "2016-07-27 21:41:38.807406", "item": "/home/ubuntu/new/od/mydir3", "rc": 1, "start": "2016-07-27 21:41:37.681981", "warnings": []}
stderr: got monmap epoch 1
ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '6', '--monmap', '/home/ubuntu/new/od/mydir3/activate.monmap', '--osd-data', '/home/ubuntu/new/od/mydir3', '--osd-journal', '/home/ubuntu/new/od/mydir3/journal', '--osd-uuid', '973391ff-4ca6-4737-a014-73902bc487e8', '--keyring', '/home/ubuntu/new/od/mydir3/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/ubuntu/new/od/mydir3/activate.monmap: (13) Permission denied
failed: [magna078] => (item=/home/c1/new/od/mydir2) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/c1/new/od/mydir2"], "delta": "0:00:01.429668", "end": "2016-07-27 21:41:40.761860", "item": "/home/c1/new/od/mydir2", "rc": 1, "start": "2016-07-27 21:41:39.332192", "warnings": []}
stderr: got monmap epoch 1
ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '5', '--monmap', '/home/c1/new/od/mydir2/activate.monmap', '--osd-data', '/home/c1/new/od/mydir2', '--osd-journal', '/home/c1/new/od/mydir2/journal', '--osd-uuid', '7e098c94-06d5-4863-8e45-e1783b174a17', '--keyring', '/home/c1/new/od/mydir2/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/c1/new/od/mydir2/activate.monmap: (13) Permission denied
failed: [magna078] => (item=/home/ubuntu/new/od/mydir3) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/ubuntu/new/od/mydir3"], "delta": "0:00:00.810338", "end": "2016-07-27 21:41:41.744600", "item": "/home/ubuntu/new/od/mydir3", "rc": 1, "start": "2016-07-27 21:41:40.934262", "warnings": []}
stderr: got monmap epoch 1
ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '9', '--monmap', '/home/ubuntu/new/od/mydir3/activate.monmap', '--osd-data', '/home/ubuntu/new/od/mydir3', '--osd-journal', '/home/ubuntu/new/od/mydir3/journal', '--osd-uuid', 'bfaaac1e-49f9-4d90-b3da-d231787df8d0', '--keyring', '/home/ubuntu/new/od/mydir3/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/ubuntu/new/od/mydir3/activate.monmap: (13) Permission denied
failed: [magna057] => (item=/home/ubuntu/new/od/mydir3) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/ubuntu/new/od/mydir3"], "delta": "0:00:00.991470", "end": "2016-07-27 21:41:37.965106", "item": "/home/ubuntu/new/od/mydir3", "rc": 1, "start": "2016-07-27 21:41:36.973636", "warnings": []}
stderr: got monmap epoch 1
ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '7', '--monmap', '/home/ubuntu/new/od/mydir3/activate.monmap', '--osd-data', '/home/ubuntu/new/od/mydir3', '--osd-journal', '/home/ubuntu/new/od/mydir3/journal', '--osd-uuid', 'd14aa886-6efe-448e-b8b0-10c7ebcad70d', '--keyring', '/home/ubuntu/new/od/mydir3/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/ubuntu/new/od/mydir3/activate.monmap: (13) Permission denied
ok: [magna051] => (item=/root/new/od/mydir4) => {"changed": false, "cmd": ["ceph-disk", "activate", "/root/new/od/mydir4"], "delta": "0:00:02.826376", "end": "2016-07-27 21:41:41.805719", "item": "/root/new/od/mydir4", "rc": 0, "start": "2016-07-27 21:41:38.979343", "stderr": "got monmap epoch 1\nadded key for osd.8\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd@8.service to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []}
ok: [magna057] => (item=/root/new/od/mydir4) => {"changed": false, "cmd": ["ceph-disk", "activate", "/root/new/od/mydir4"], "delta": "0:00:02.870615", "end": "2016-07-27 21:41:41.005749", "item": "/root/new/od/mydir4", "rc": 0, "start": "2016-07-27 21:41:38.135134", "stderr": "got monmap epoch 1\nadded key for osd.10\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd@10.service to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []}
ok: [magna078] => (item=/root/new/od/mydir4) => {"changed": false, "cmd": ["ceph-disk", "activate", "/root/new/od/mydir4"], "delta": "0:00:02.894505", "end": "2016-07-27 21:41:44.883140", "item": "/root/new/od/mydir4", "rc": 0, "start": "2016-07-27 21:41:41.988635", "stderr": "got monmap epoch 1\nadded key for osd.11\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd@11.service to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []}

FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/root/site.retry

magna008                   : ok=93   changed=16   unreachable=0    failed=0   
magna051                   : ok=118  changed=12   unreachable=0    failed=1   
magna057                   : ok=118  changed=12   unreachable=0    failed=1   
magna078                   : ok=118  changed=12   unreachable=0    failed=1   


Expected results:


Additional info:

Comment 7 Christina Meno 2016-08-17 16:37:10 UTC
Looks good

Comment 8 Ken Dreyer (Red Hat) 2017-03-03 17:12:30 UTC
We do not currently support OSD directories in the downstream product, and jenkins.ceph.com does not test this scenario upstream.

If supporting OSD directories is a priority from product management, please
re-open.

Comment 12 Alfredo Deza 2017-05-23 19:49:23 UTC
An update on this: even though we aren't supporting directories, it is important to note that this happens because the location of the monmap is in a directory where the 'ceph' user cannot get to. That is the reason this works in /var/lib/ceph/osd/ because the permissions are good enough in that path.

Related upstream issue: http://tracker.ceph.com/issues/20056


Note You need to log in before you can comment on or make changes to this bug.