Description of problem:

1) Cluster 2-Mon's and 3 OSDs'

TASK: [ceph-mon | enable systemd unit file for mon instance (for or after infernalis)] *** 
ok: [clara006.ceph.redhat.com] => {"changed": false, "dest": "/etc/systemd/system/multi-user.target.wants/ceph-mon", "failed": false, "failed_when_result": false, "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:systemd_unit_file_t:s0", "size": 41, "src": "/usr/lib/systemd/system/ceph-mon@.service", "state": "link", "uid": 0}
ok: [clara014.ceph.redhat.com] => {"changed": false, "dest": "/etc/systemd/system/multi-user.target.wants/ceph-mon", "failed": false, "failed_when_result": false, "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:systemd_unit_file_t:s0", "size": 41, "src": "/usr/lib/systemd/system/ceph-mon@.service", "state": "link", "uid": 0}

TASK: [ceph-mon | start and add that the monitor service to the init sequence (for or after infernalis)] *** 
failed: [clara006.ceph.redhat.com] => {"changed": false, "failed": true}
msg: Failed to start ceph-mon: Unit ceph-mon failed to load: No such file or directory.

failed: [clara014.ceph.redhat.com] => {"changed": false, "failed": true}
msg: Failed to start ceph-mon: Unit ceph-mon failed to load: No such file or directory.


FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/home/ubuntu/site.retry



Full Logs:
http://magna002.ceph.redhat.com/vasu-2016-05-07_22:49:51-smoke-jewel---basic-clara/230833/teuthology.log
clara006.ceph.redhat.com   : ok=61   changed=12   unreachable=0    failed=1   
clara014.ceph.redhat.com   : ok=61   changed=12   unreachable=0    failed=1   



Steps to recreate:
2016-05-08T03:21:24.861 INFO:teuthology.orchestra.run.clara014.stdout:[mons]
2016-05-08T03:21:24.862 INFO:teuthology.orchestra.run.clara014.stdout:clara006.ceph.redhat.com devices='["/dev/sdb", "/dev/sdc", "/dev/sdd"]' monitor_interface='eno1' public_network='10.8.128.0/21'
2016-05-08T03:21:24.863 INFO:teuthology.orchestra.run.clara014.stdout:clara014.ceph.redhat.com devices='["/dev/sdb", "/dev/sdc", "/dev/sdd"]' monitor_interface='eno1' public_network='10.8.128.0/21'
2016-05-08T03:21:24.864 INFO:teuthology.orchestra.run.clara014.stdout:
2016-05-08T03:21:24.865 INFO:teuthology.orchestra.run.clara014.stdout:[osds]
2016-05-08T03:21:24.866 INFO:teuthology.orchestra.run.clara014.stdout:clara007.ceph.redhat.com devices='["/dev/sdb", "/dev/sdc", "/dev/sdd"]' monitor_interface='eno1' public_network='10.8.128.0/21'
2016-05-08T03:21:24.867 INFO:teuthology.orchestra.run.clara014.stdout:clara008.ceph.redhat.com devices='["/dev/sdb", "/dev/sdc", "/dev/sdd"]' monitor_interface='eno1' public_network='10.8.128.0/21'
2016-05-08T03:21:24.868 INFO:teuthology.orchestra.run.clara014.stdout:clara010.ceph.redhat.com devices='["/dev/sdb", "/dev/sdc", "/dev/sdd"]' monitor_interface='eno1' public_network='10.8.128.0/21'
2016-05-08T03:21:24.870 INFO:teuthology.orchestra.run.clara014:Running: 'cat ceph-ansible/site.yml'
2016-05-08T03:21:24.927 INFO:teuthology.orchestra.run.clara014.stdout:---
2016-05-08T03:21:24.929 INFO:teuthology.orchestra.run.clara014.stdout:- become: true
2016-05-08T03:21:24.930 INFO:teuthology.orchestra.run.clara014.stdout:  hosts: mons
2016-05-08T03:21:24.931 INFO:teuthology.orchestra.run.clara014.stdout:  roles: [ceph-mon]
2016-05-08T03:21:24.932 INFO:teuthology.orchestra.run.clara014.stdout:- become: true
2016-05-08T03:21:24.933 INFO:teuthology.orchestra.run.clara014.stdout:  hosts: osds
2016-05-08T03:21:24.934 INFO:teuthology.orchestra.run.clara014.stdout:  roles: [ceph-osd]
2016-05-08T03:21:24.935 INFO:teuthology.orchestra.run.clara014.stdout:- become: true
2016-05-08T03:21:24.935 INFO:teuthology.orchestra.run.clara014.stdout:  hosts: mdss
2016-05-08T03:21:24.936 INFO:teuthology.orchestra.run.clara014.stdout:  roles: [ceph-mds]
2016-05-08T03:21:24.937 INFO:teuthology.orchestra.run.clara014.stdout:- become: true
2016-05-08T03:21:24.938 INFO:teuthology.orchestra.run.clara014.stdout:  hosts: rgws
2016-05-08T03:21:24.939 INFO:teuthology.orchestra.run.clara014.stdout:  roles: [ceph-rgw]
2016-05-08T03:21:24.940 INFO:teuthology.orchestra.run.clara014.stdout:- become: true
2016-05-08T03:21:24.941 INFO:teuthology.orchestra.run.clara014.stdout:  hosts: client
2016-05-08T03:21:24.942 INFO:teuthology.orchestra.run.clara014.stdout:  roles: [ceph-common]
2016-05-08T03:21:24.943 INFO:teuthology.orchestra.run:Running command with timeout 4200
2016-05-08T03:21:24.943 INFO:teuthology.orchestra.run.clara014:Running: 'cd ceph-ansible ; ansible-playbook -v --extra-vars \'{"ceph_origin": "distro", "ansible_ssh_user": "ubuntu", "journal_collocation": true, "osd_auto_discovery": false, "ceph_stable_rh_storage": true, "ceph_stable": true, "journal_size": 1024}\' -i /tmp/teuth_ansible_hosts_NLcRIB --limit