Description of problem: When we attempt to live migrate instances, we get this error: ERROR oslo_messaging.notify.messaging NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout Version-Release number of selected component (if applicable): openstack-nova-common-16.0.2-9.el7ost.noarch How reproducible: All the time Steps to Reproduce: 1. deploy OSP12 2. live migrate instances between 2 hosts Actual results: Migration is not working with nova complaining that it's unable to find the Q. I've found this BZ [1] that lead me to [2]. When we change the notification_format from "both" (default) to "unversioned" we don't have the error anymore and migration works. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1455793 [2] https://access.redhat.com/solutions/3139721
This seems to be a known issue: 1478274
Fixed upstream in https://review.openstack.org/522644
This bugzilla has been removed from the release since it has not been triaged, and needs to be reviewed for targeting another release.
Marking the bug as VERIFIED. Verification steps: 1. create instances. Start/stop them. Live migrate them across a few compute nodes. 2. List the notification queues inside the rabbitmq container and validate that versioned_notifications are gone: [heat-admin@controller-0 ~]$ sudo docker exec rabbitmq-bundle-docker-0 rabbitmqctl list_queues | grep notifications notifications.info 0 notifications.warn 0 notifications.audit 0 notifications.sample 0 notifications.critical 0 notifications.error 0 notifications.debug 0 3. Verify `notification_format` in the nova.conf: [heat-admin@compute-0 ~]$ sudo docker exec -ti -u root nova_compute bash ()[root@compute-0 /]# grep notification_format /etc/nova/nova.conf #notification_format=both notification_format=unversioned [heat-admin@compute-0 ~]$ sudo grep notification_format /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf #notification_format=both notification_format=unversioned [heat-admin@controller-0 ~]$ sudo docker exec -ti nova_api bash ()[root@controller-0 /]# grep notification_format /etc/nova/nova.conf #notification_format=both notification_format=unversioned [heat-admin@controller-0 ~]$ sudo grep notification_format /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf #notification_format=both notification_format=unversioned Note that the issue still persists on the undercloud where I can still see versioned notifications: (overcloud) [stack@undercloud-0 ~]$ sudo rabbitmqctl list_queues name consumers messages | grep notifications versioned_notifications.info 0 51 notifications.info 0 104
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0602