Bug 1628386
| Summary: | Overcloud deployment succeeds even if ceph-ansible failed to set up the OSDs | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> |
| Component: | rhosp-director | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | Gurenko Alex <agurenko> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 14.0 (Rocky) | CC: | dbecker, gfidente, mburns, morazi |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-09-19 12:20:01 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 1541152 *** |
Description of problem: Overcloud deployment succeeds even if ceph-ansible failed to set up the OSDs. The deployment process should fail fast(at the ceph-ansible step) pointing to what went wrong. Currently the deployment ends up successfully but no OSDs are set up. This can be easily reproduced by doing a deployment with OSD disks configured on a previous deployment. Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-9.0.0-0.20180831204457.17bb71e.0rc1.el7ost.noarch ceph-ansible-3.1.0-0.1.rc21.el7cp.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with ceph osd nodes containing osd disks from a previous deployment Actual results: Overcloud deployment succeeds but no OSDs are set up. Expected results: Overcloud deployment should fail and stop after failing to set up the OSD disks and not continue further. Additional info: On the OSD nodes the ceph-osd-overcloud-cephstorage-0-sdb container keeps getting restarted failing to come up because it was part of a different deployment: Sep 12 21:11:43 overcloud-cephstorage-0 docker[47810]: Error response from daemon: No such container: ceph-osd-overcloud-cephstorage-0-sdb Sep 12 21:11:43 overcloud-cephstorage-0 systemd[1]: Started Ceph OSD. Sep 12 21:11:45 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: 2018-09-12 21:11:45 /entrypoint.sh: static: does not generate config Sep 12 21:11:45 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: /dev/sdb2 Sep 12 21:11:46 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: main_activate: path = /dev/sdb1 Sep 12 21:11:46 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid Sep 12 21:11:46 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: command: Running command: /usr/sbin/blkid -o udev -p /dev/sdb1 Sep 12 21:11:46 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sdb1 Sep 12 21:11:46 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs Sep 12 21:11:47 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.OQSmAi with options noatime,largeio,inode64,swalloc Sep 12 21:11:47 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,largeio,inode64,swalloc -- /dev/sdb1 /var/lib/ceph/tmp/mnt.OQSmAi Sep 12 21:11:47 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: command: Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.OQSmAi Sep 12 21:11:47 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: activate: Cluster uuid is a6a29320-aac8-11e8-b0b2-525400d05954 Sep 12 21:11:47 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: mount_activate: Failed to activate Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: unmount: Unmounting /var/lib/ceph/tmp/mnt.OQSmAi Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.OQSmAi Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: Traceback (most recent call last): Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: File "/usr/sbin/ceph-disk", line 9, in <module> Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')() Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5735, in run Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: main(sys.argv[1:]) Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5686, in main Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: args.func(args) Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3776, in main_activate Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: reactivate=args.reactivate, Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3539, in mount_activate Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: (osd_id, cluster) = activate(path, activate_key_template, init) Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3686, in activate Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: ' with fsid %s' % ceph_fsid) Sep 12 21:11:48 overcloud-cephstorage-0 ceph-osd-run.sh[47823]: ceph_disk.main.Error: Error: No cluster conf found in /etc/ceph with fsid a6a29320-aac8-11e8-b0b2-525400d05954 Sep 12 21:11:48 overcloud-cephstorage-0 systemd[1]: ceph-osd: main process exited, code=exited, status=1/FAILURE Sep 12 21:11:48 overcloud-cephstorage-0 docker[48343]: Error response from daemon: No such container: ceph-osd-overcloud-cephstorage-0-sdb Sep 12 21:11:48 overcloud-cephstorage-0 systemd[1]: Unit ceph-osd entered failed state. Sep 12 21:11:48 overcloud-cephstorage-0 systemd[1]: ceph-osd failed.