rhel-osp-director: 7.2 -> 8.0 after yum update on the undercloud nodes ceilometer services are down. Environment: instack-undercloud-2.2.0-1.el7ost.noarch Steps to reproduce: 1. Successfully deploy overcloud7.2 2. Attempt to update the undercloud machine - run "yum update" after updating the yum repos. 3. Run "openstack-service status" Result: MainPID=0 Id=openstack-ceilometer-alarm-evaluator.service ActiveState=failed MainPID=0 Id=openstack-ceilometer-alarm-notifier.service ActiveState=failed MainPID=0 Id=openstack-ceilometer-api.service ActiveState=failed MainPID=0 Id=openstack-ceilometer-central.service ActiveState=failed Expected result: No failed services. After rebooting the node, these services are still down: MainPID=0 Id=openstack-ceilometer-alarm-evaluator.service ActiveState=failed MainPID=0 Id=openstack-ceilometer-alarm-notifier.service ActiveState=failed MainPID=0 Id=openstack-ceilometer-api.service ActiveState=failed MainPID=0 Id=openstack-ceilometer-central.service ActiveState=failed MainPID=2683 Id=openstack-nova-compute.service ActiveState=activating MainPID=0 Id=openstack-swift-proxy.service ActiveState=failed
Created attachment 1109018 [details] /var/log/messages from the undercloud
Looks like this is similar to a problem we encountered in the overcloud DriverLoadFailure: Failed to load transport driver "ceilometer.openstack.common.rpc.impl_kombu": No 'oslo.messaging.drivers' driver found, looking for 'ceilometer.openstack.common.rpc.impl_kombu' openstack-ceilometer-central.service: main process exited, code=exited, status=1/FAILURE Unit openstack-ceilometer-central.service entered failed state. openstack-ceilometer-central.service failed. openstack-ceilometer-api.service: main process exited, code=exited, status=1/FAILURE Unit openstack-ceilometer-api.service entered failed state. openstack-ceilometer-api.service failed. See: https://bugzilla.redhat.com/show_bug.cgi?id=1284047 Also to upgrade the undercloud, you will need to rerun $ openstack undercloud install this should upgrade the packages for you (assuming new repositories have been added), it wont rerun the database syncs, we need a patch to make this happen one has been submitted upstream but may need some work first. https://review.openstack.org/#/c/244136/
Brad, if this is fixed, could you move to MODIFIED please? I thought we had undercloud upgrade finished at this point.
I am unable to reproduce this.
So right after yum update on the undercloud node these services are down: ● openstack-ceilometer-alarm-evaluator.service loaded failed failed OpenStack ceilometer alarm evaluation service ● openstack-ceilometer-alarm-notifier.service loaded failed failed OpenStack ceilometer alarm notification service ● openstack-ceilometer-api.service loaded failed failed OpenStack ceilometer API service ● openstack-ceilometer-central.service loaded failed failed OpenStack ceilometer central agent Re-running "openstack undercloud install" brings them up.
Verified: So until the openstack undercloud install is ran, the system is in an undefined state. Re-running "openstack undercloud install" brings the ceilometer services up. Need to run "openstack undercloud install" after running yum update on the undercloud node.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0604.html