Bug 1551137 - Queue versioned_notifications.info not found
Summary: Queue versioned_notifications.info not found
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: z2
: 12.0 (Pike)
Assignee: nova-maint
QA Contact: nova-maint
URL:
Whiteboard:
Depends On:
Blocks: 1478274 1579316
TreeView+ depends on / blocked
 
Reported: 2018-03-02 21:06 UTC by David Vallee Delisle
Modified: 2019-09-09 16:53 UTC (History)
16 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.9-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-28 17:17:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:0602 None None None 2018-03-28 17:18:18 UTC
OpenStack gerrit 522644 None None None 2018-03-15 23:29:29 UTC

Description David Vallee Delisle 2018-03-02 21:06:44 UTC
Description of problem:
When we attempt to live migrate instances, we get this error:
ERROR oslo_messaging.notify.messaging NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout

Version-Release number of selected component (if applicable):
openstack-nova-common-16.0.2-9.el7ost.noarch

How reproducible:
All the time

Steps to Reproduce:
1. deploy OSP12
2. live migrate instances between 2 hosts

Actual results:
Migration is not working with nova complaining that it's unable to find the Q.

I've found this BZ [1] that lead me to [2].

When we change the notification_format from "both" (default) to "unversioned" we don't have the error anymore and migration works.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1455793
[2] https://access.redhat.com/solutions/3139721

Comment 1 Dan Smith 2018-03-09 15:16:57 UTC
This seems to be a known issue: 1478274

Comment 2 Ollie Walsh 2018-03-15 23:46:07 UTC
Fixed upstream in https://review.openstack.org/522644

Comment 3 Red Hat Bugzilla Rules Engine 2018-03-15 23:46:16 UTC
This bugzilla has been removed from the release since it has not been triaged, and needs to be reviewed for targeting another release.

Comment 8 Joe H. Rahme 2018-03-21 09:26:02 UTC
Marking the bug as VERIFIED. Verification steps:

1. create instances. Start/stop them. Live migrate them across a few compute nodes.
2. List the notification queues inside the rabbitmq container and validate that versioned_notifications are gone:

	[heat-admin@controller-0 ~]$ sudo docker exec rabbitmq-bundle-docker-0 rabbitmqctl list_queues | grep notifications
	notifications.info	0
	notifications.warn	0
	notifications.audit	0
	notifications.sample	0
	notifications.critical	0
	notifications.error	0
	notifications.debug	0
	
3. Verify `notification_format` in the nova.conf:

	[heat-admin@compute-0 ~]$ sudo docker exec -ti -u root nova_compute bash
	()[root@compute-0 /]# grep notification_format /etc/nova/nova.conf
	#notification_format=both
	notification_format=unversioned

	[heat-admin@compute-0 ~]$ sudo grep notification_format /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf
	#notification_format=both
	notification_format=unversioned


	[heat-admin@controller-0 ~]$ sudo docker exec -ti nova_api bash
	()[root@controller-0 /]# grep notification_format /etc/nova/nova.conf
	#notification_format=both
	notification_format=unversioned

	[heat-admin@controller-0 ~]$ sudo grep notification_format  /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf
	#notification_format=both
	notification_format=unversioned



Note that the issue still persists on the undercloud where I can still see versioned notifications:

	(overcloud) [stack@undercloud-0 ~]$ sudo rabbitmqctl list_queues name consumers messages | grep notifications
	versioned_notifications.info	0	51
	notifications.info	0	104

Comment 11 errata-xmlrpc 2018-03-28 17:17:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0602


Note You need to log in before you can comment on or make changes to this bug.