Description of problem: ====================== Installation using ceph-ansible fails in task 'activate OSD(s)' if OSDs are Directory if OSDS are directories '/var/lib/ceph/osd/*' then it creates cluster Version-Release number of selected component (if applicable): ============================================================== 10.2.2-29.el7cp.x86_64 ceph-ansible-1.0.5-31.el7scon.noarch How reproducible: ================== always Steps to Reproduce: =================== 1. update host file and other config file as below host:- [mons] magna008 [osds] magna051 magna057 magna078 fetch_directory: rhel31/ ceph_stable: true # use ceph stable branch ceph_stable_rh_storage: true ceph_stable_rh_storage_iso_install: true # usually used when nodes don't have access to cdn.redhat.com ceph_stable_rh_storage_iso_path: /root/new/in/Ceph-2.0-RHEL-7-20160725.t.0-x86_64-dvd.iso ceph_stable_rh_storage_mount_path: /home/c1/iso/abc/rh-storage-mount ceph_stable_rh_storage_repository_path: /home/c1/abc/iso/rh-storage-repo # where to copy iso's content cephx: true monitor_interface: eno1 journal_size: 100 public_network: XXXX/21 osds crush_location: false osd_crush_location: "'root={{ ceph_crush_root }} rack={{ ceph_crush_rack }} host={{ ansible_hostname }}'" osd_directory: true osd_directories: - /var/lib/ceph/new/od/mydir1 - /home/c1/new/od/mydir2 - /home/ubuntu/new/od/mydir3 - /root/new/od/mydir4 2. [root@magna044 ceph-ansible]# ansible-playbook site.yml -u c1 -i /etc/ansible/ubuntu31 --verbose Actual results: ============== TASK: [ceph-osd | activate OSD(s)] ******************************************** ok: [magna051] => (item=/var/lib/ceph/new/od/mydir1) => {"changed": false, "cmd": ["ceph-disk", "activate", "/var/lib/ceph/new/od/mydir1"], "delta": "0:00:04.861051", "end": "2016-07-27 21:41:36.129559", "item": "/var/lib/ceph/new/od/mydir1", "rc": 0, "start": "2016-07-27 21:41:31.268508", "stderr": "got monmap epoch 1\nadded key for osd.0\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []} ok: [magna057] => (item=/var/lib/ceph/new/od/mydir1) => {"changed": false, "cmd": ["ceph-disk", "activate", "/var/lib/ceph/new/od/mydir1"], "delta": "0:00:05.928685", "end": "2016-07-27 21:41:35.283594", "item": "/var/lib/ceph/new/od/mydir1", "rc": 0, "start": "2016-07-27 21:41:29.354909", "stderr": "got monmap epoch 1\nadded key for osd.1\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []} ok: [magna078] => (item=/var/lib/ceph/new/od/mydir1) => {"changed": false, "cmd": ["ceph-disk", "activate", "/var/lib/ceph/new/od/mydir1"], "delta": "0:00:05.941504", "end": "2016-07-27 21:41:39.133965", "item": "/var/lib/ceph/new/od/mydir1", "rc": 0, "start": "2016-07-27 21:41:33.192461", "stderr": "got monmap epoch 1\nadded key for osd.2\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []} failed: [magna051] => (item=/home/c1/new/od/mydir2) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/c1/new/od/mydir2"], "delta": "0:00:01.154836", "end": "2016-07-27 21:41:37.510033", "item": "/home/c1/new/od/mydir2", "rc": 1, "start": "2016-07-27 21:41:36.355197", "warnings": []} stderr: got monmap epoch 1 ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '3', '--monmap', '/home/c1/new/od/mydir2/activate.monmap', '--osd-data', '/home/c1/new/od/mydir2', '--osd-journal', '/home/c1/new/od/mydir2/journal', '--osd-uuid', '99b3eb35-806b-4af6-88f7-a03221e8b14f', '--keyring', '/home/c1/new/od/mydir2/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/c1/new/od/mydir2/activate.monmap: (13) Permission denied failed: [magna057] => (item=/home/c1/new/od/mydir2) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/c1/new/od/mydir2"], "delta": "0:00:01.342660", "end": "2016-07-27 21:41:36.802715", "item": "/home/c1/new/od/mydir2", "rc": 1, "start": "2016-07-27 21:41:35.460055", "warnings": []} stderr: got monmap epoch 1 ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '4', '--monmap', '/home/c1/new/od/mydir2/activate.monmap', '--osd-data', '/home/c1/new/od/mydir2', '--osd-journal', '/home/c1/new/od/mydir2/journal', '--osd-uuid', 'e2f11952-da43-4e98-8fdb-09ce8fb1914e', '--keyring', '/home/c1/new/od/mydir2/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/c1/new/od/mydir2/activate.monmap: (13) Permission denied failed: [magna051] => (item=/home/ubuntu/new/od/mydir3) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/ubuntu/new/od/mydir3"], "delta": "0:00:01.125425", "end": "2016-07-27 21:41:38.807406", "item": "/home/ubuntu/new/od/mydir3", "rc": 1, "start": "2016-07-27 21:41:37.681981", "warnings": []} stderr: got monmap epoch 1 ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '6', '--monmap', '/home/ubuntu/new/od/mydir3/activate.monmap', '--osd-data', '/home/ubuntu/new/od/mydir3', '--osd-journal', '/home/ubuntu/new/od/mydir3/journal', '--osd-uuid', '973391ff-4ca6-4737-a014-73902bc487e8', '--keyring', '/home/ubuntu/new/od/mydir3/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/ubuntu/new/od/mydir3/activate.monmap: (13) Permission denied failed: [magna078] => (item=/home/c1/new/od/mydir2) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/c1/new/od/mydir2"], "delta": "0:00:01.429668", "end": "2016-07-27 21:41:40.761860", "item": "/home/c1/new/od/mydir2", "rc": 1, "start": "2016-07-27 21:41:39.332192", "warnings": []} stderr: got monmap epoch 1 ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '5', '--monmap', '/home/c1/new/od/mydir2/activate.monmap', '--osd-data', '/home/c1/new/od/mydir2', '--osd-journal', '/home/c1/new/od/mydir2/journal', '--osd-uuid', '7e098c94-06d5-4863-8e45-e1783b174a17', '--keyring', '/home/c1/new/od/mydir2/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/c1/new/od/mydir2/activate.monmap: (13) Permission denied failed: [magna078] => (item=/home/ubuntu/new/od/mydir3) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/ubuntu/new/od/mydir3"], "delta": "0:00:00.810338", "end": "2016-07-27 21:41:41.744600", "item": "/home/ubuntu/new/od/mydir3", "rc": 1, "start": "2016-07-27 21:41:40.934262", "warnings": []} stderr: got monmap epoch 1 ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '9', '--monmap', '/home/ubuntu/new/od/mydir3/activate.monmap', '--osd-data', '/home/ubuntu/new/od/mydir3', '--osd-journal', '/home/ubuntu/new/od/mydir3/journal', '--osd-uuid', 'bfaaac1e-49f9-4d90-b3da-d231787df8d0', '--keyring', '/home/ubuntu/new/od/mydir3/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/ubuntu/new/od/mydir3/activate.monmap: (13) Permission denied failed: [magna057] => (item=/home/ubuntu/new/od/mydir3) => {"changed": false, "cmd": ["ceph-disk", "activate", "/home/ubuntu/new/od/mydir3"], "delta": "0:00:00.991470", "end": "2016-07-27 21:41:37.965106", "item": "/home/ubuntu/new/od/mydir3", "rc": 1, "start": "2016-07-27 21:41:36.973636", "warnings": []} stderr: got monmap epoch 1 ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '7', '--monmap', '/home/ubuntu/new/od/mydir3/activate.monmap', '--osd-data', '/home/ubuntu/new/od/mydir3', '--osd-journal', '/home/ubuntu/new/od/mydir3/journal', '--osd-uuid', 'd14aa886-6efe-448e-b8b0-10c7ebcad70d', '--keyring', '/home/ubuntu/new/od/mydir3/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to read/decode monmap from /home/ubuntu/new/od/mydir3/activate.monmap: (13) Permission denied ok: [magna051] => (item=/root/new/od/mydir4) => {"changed": false, "cmd": ["ceph-disk", "activate", "/root/new/od/mydir4"], "delta": "0:00:02.826376", "end": "2016-07-27 21:41:41.805719", "item": "/root/new/od/mydir4", "rc": 0, "start": "2016-07-27 21:41:38.979343", "stderr": "got monmap epoch 1\nadded key for osd.8\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []} ok: [magna057] => (item=/root/new/od/mydir4) => {"changed": false, "cmd": ["ceph-disk", "activate", "/root/new/od/mydir4"], "delta": "0:00:02.870615", "end": "2016-07-27 21:41:41.005749", "item": "/root/new/od/mydir4", "rc": 0, "start": "2016-07-27 21:41:38.135134", "stderr": "got monmap epoch 1\nadded key for osd.10\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []} ok: [magna078] => (item=/root/new/od/mydir4) => {"changed": false, "cmd": ["ceph-disk", "activate", "/root/new/od/mydir4"], "delta": "0:00:02.894505", "end": "2016-07-27 21:41:44.883140", "item": "/root/new/od/mydir4", "rc": 0, "start": "2016-07-27 21:41:41.988635", "stderr": "got monmap epoch 1\nadded key for osd.11\nCreated symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd to /usr/lib/systemd/system/ceph-osd@.service.", "stdout": "", "warnings": []} FATAL: all hosts have already failed -- aborting PLAY RECAP ******************************************************************** to retry, use: --limit @/root/site.retry magna008 : ok=93 changed=16 unreachable=0 failed=0 magna051 : ok=118 changed=12 unreachable=0 failed=1 magna057 : ok=118 changed=12 unreachable=0 failed=1 magna078 : ok=118 changed=12 unreachable=0 failed=1 Expected results: Additional info:
Looks good
We do not currently support OSD directories in the downstream product, and jenkins.ceph.com does not test this scenario upstream. If supporting OSD directories is a priority from product management, please re-open.
An update on this: even though we aren't supporting directories, it is important to note that this happens because the location of the monmap is in a directory where the 'ceph' user cannot get to. That is the reason this works in /var/lib/ceph/osd/ because the permissions are good enough in that path. Related upstream issue: http://tracker.ceph.com/issues/20056