Description of problem:
Customer is preparing for upgrade from RHOSP12 to RHOSP13, your Ceph node has failed. That ceph node was removed (openstack overcloud node delete..)
Next he wanted to add new but already used baremetal node, so you have configured autocleaning=true in undercloud.conf and re-run the 'openstack undercloud install'
This has updated the packages on their undercloud and it failed to scale-out because of https://bugzilla.redhat.com/show_bug.cgi?id=1603182 due to undercloud install has a mismatch between the heat templates and nova rpm version in the containers.
In the meantime, you have reported that one of your controller nodes have failed.
So the situation is that:
1. stack deploy command fails with package mismatch
2. controller is gone
3. ceph node was never scaled-out (now running on 4/5 nodes )