Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1551137

Summary:	Queue versioned_notifications.info not found
Product:	Red Hat OpenStack	Reporter:	David Vallee Delisle <dvd>
Component:	openstack-tripleo-heat-templates	Assignee:	OSP DFG:Compute <osp-dfg-compute>
Status:	CLOSED ERRATA	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	12.0 (Pike)	CC:	berrange, dasmith, eglynn, jhakimra, kchamart, lyarwood, marjones, mburns, owalsh, rhel-osp-director-maint, sbauza, sclewis, sferdjao, sgordon, srevivo, vromanso
Target Milestone:	z2	Keywords:	Triaged, ZStream
Target Release:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-7.0.9-1.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-28 17:17:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1478274, 1579316

Description David Vallee Delisle 2018-03-02 21:06:44 UTC

Description of problem:
When we attempt to live migrate instances, we get this error:
ERROR oslo_messaging.notify.messaging NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout

Version-Release number of selected component (if applicable):
openstack-nova-common-16.0.2-9.el7ost.noarch

How reproducible:
All the time

Steps to Reproduce:
1. deploy OSP12
2. live migrate instances between 2 hosts

Actual results:
Migration is not working with nova complaining that it's unable to find the Q.

I've found this BZ [1] that lead me to [2].

When we change the notification_format from "both" (default) to "unversioned" we don't have the error anymore and migration works.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1455793
[2] https://access.redhat.com/solutions/3139721

Comment 1 Dan Smith 2018-03-09 15:16:57 UTC

This seems to be a known issue: 1478274

Comment 2 Ollie Walsh 2018-03-15 23:46:07 UTC

Fixed upstream in https://review.openstack.org/522644

Comment 3 Red Hat Bugzilla Rules Engine 2018-03-15 23:46:16 UTC

This bugzilla has been removed from the release since it has not been triaged, and needs to be reviewed for targeting another release.

Comment 8 Joe H. Rahme 2018-03-21 09:26:02 UTC

Marking the bug as VERIFIED. Verification steps:

1. create instances. Start/stop them. Live migrate them across a few compute nodes.
2. List the notification queues inside the rabbitmq container and validate that versioned_notifications are gone:

	[heat-admin@controller-0 ~]$ sudo docker exec rabbitmq-bundle-docker-0 rabbitmqctl list_queues | grep notifications
	notifications.info	0
	notifications.warn	0
	notifications.audit	0
	notifications.sample	0
	notifications.critical	0
	notifications.error	0
	notifications.debug	0
	
3. Verify `notification_format` in the nova.conf:

	[heat-admin@compute-0 ~]$ sudo docker exec -ti -u root nova_compute bash
	()[root@compute-0 /]# grep notification_format /etc/nova/nova.conf
	#notification_format=both
	notification_format=unversioned

	[heat-admin@compute-0 ~]$ sudo grep notification_format /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf
	#notification_format=both
	notification_format=unversioned


	[heat-admin@controller-0 ~]$ sudo docker exec -ti nova_api bash
	()[root@controller-0 /]# grep notification_format /etc/nova/nova.conf
	#notification_format=both
	notification_format=unversioned

	[heat-admin@controller-0 ~]$ sudo grep notification_format  /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf
	#notification_format=both
	notification_format=unversioned



Note that the issue still persists on the undercloud where I can still see versioned notifications:

	(overcloud) [stack@undercloud-0 ~]$ sudo rabbitmqctl list_queues name consumers messages | grep notifications
	versioned_notifications.info	0	51
	notifications.info	0	104

Comment 11 errata-xmlrpc 2018-03-28 17:17:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0602