Red Hat Bugzilla – Bug 1522872
[UPDATES] Failed to spawn vm after OC upate Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info'
Last modified: 2018-11-02 18:04:01 EDT
Description of problem: ----------------------- After RHOS-12 minor update failed to spawn new vm. In conductor.log got next traceback: 2017-12-06 13:22:49.881 24 ERROR oslo.messaging._drivers.impl_rabbit [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] Failed to publi sh message to topic 'nova': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] Could not send noti fication to versioned_notifications. Payload={'_context_domain': None, '_context_roles': [u'admin'], '_context_global_request_id': None, '_context_quota_class': None, 'event_type': u'instance.update', '_context_ request_id': u'req-a8ce222c-010a-4a27-80e9-8da0972d6278', '_context_service_catalog': [{u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v2/358cf3d490424818a6365fc4379637a0', u'region': u'reg ionOne', u'internalURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v2/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v2/358cf3d490424818a6365fc4379637a0'}], u'type ': u'volumev2', u'name': u'cinderv2'}, {u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v1/358cf3d490424818a6365fc4379637a0', u'region': u'regionOne', u'internalURL': u'http://[fd00:fd00:fd0 0:2000::16]:8776/v1/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v1/358cf3d490424818a6365fc4379637a0'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints' : [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8778/placement', u'region': u'regionOne', u'internalURL': u'http://[fd00:fd00:fd00:2000::16]:8778/placement', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:f e3e:1]:13778/placement'}], u'type': u'placement', u'name': u'placement'}, {u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v3/358cf3d490424818a6365fc4379637a0', u'region': u'regionOne', u'in ternalURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v3/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v3/358cf3d490424818a6365fc4379637a0'}], u'type': u'volumev3 ', u'name': u'cinderv3'}], 'timestamp': u'2017-12-06 13:21:46.885360', '_context_user': u'3f51df1556e748ce82d6e12188baefee', '_unique_id': '91f739fd62a94d65a95e488cb8676299', '_context_resource_uuid': None, '_co ntext_instance_lock_checked': False, '_context_is_admin_project': True, '_context_user_id': u'3f51df1556e748ce82d6e12188baefee', 'payload': {'nova_object.version': '1.6', 'nova_object.name': 'InstanceUpdatePaylo ad', 'nova_object.namespace': 'nova', 'nova_object.data': {'availability_zone': u'nova', 'terminated_at': None, 'ip_addresses': [], 'ramdisk_id': u'', 'updated_at': None, 'state_update': {'nova_object.version': '1.0', 'nova_object.name': 'InstanceStateUpdatePayload', 'nova_object.namespace': 'nova', 'nova_object.data': {'old_state': u'building', 'state': u'building', 'old_task_state': u'scheduling', 'new_task_state': u 'scheduling'}}, 'bandwidth': [], 'old_display_name': None, 'image_uuid': u'51672902-0005-42b8-922e-9649506799f4', 'flavor': {'nova_object.version': '1.3', 'nova_object.name': 'FlavorPayload', 'nova_object.namesp ace': 'nova', 'nova_object.data': {'memory_mb': 128, 'root_gb': 0, 'name': u'm1.micro', 'ephemeral_gb': 0, 'disabled': False, 'vcpus': 1, 'extra_specs': {}, 'swap': 0, 'rxtx_factor': 1.0, 'is_public': True, 'fla vorid': u'09363262-fd58-472d-bebb-0e4dc0405b77', 'vcpu_weight': 0, 'projects': None}}, 'deleted_at': None, 'reservation_id': u'r-6037twbk', 'user_id': u'3f51df1556e748ce82d6e12188baefee', 'uuid': '29eac6da-bcd1- 4b55-87a2-17f5868d6072', 'display_description': u'post-update-b4-reboot', 'state': u'building', 'power_state': u'pending', 'host_name': u'post-update-b4-reboot', 'progress': 0, 'launched_at': None, 'metadata': { }, 'node': None, 'tags': [], 'kernel_id': u'', 'key_name': u'userkey', 'host': None, 'display_name': u'post-update-b4-reboot', 'task_state': u'scheduling', 'locked': False, 'tenant_id': u'358cf3d490424818a6365fc 4379637a0', 'created_at': '2017-12-06T13:21:44Z', 'block_devices': None, 'architecture': None, 'audit_period': {'nova_object.version': '1.0', 'nova_object.name': 'AuditPeriodPayload', 'nova_object.namespace': 'n ova', 'nova_object.data': {'audit_period_ending': '2017-12-06T13:21:46Z', 'audit_period_beginning': '2017-12-01T00:00:00Z'}}, 'auto_disk_config': u'MANUAL', 'os_type': None}}, '_context_project_name': u'admin', '_context_read_deleted': u'no', '_context_user_identity': u'3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default', '_context_auth_token': u'gAAAAABaJ-7iwlHT0qzrIatuzRvxK9n2j6cRdoSR Da-8sBaICio8giGn3UHsJZqN--ke_eGbCrKPajy3EO8fRjyMUea8XxjsbtsRZoptOZzTUAu0gLUH5JdMDqP_R8kGDDy7oh39_qWV5OM67oqfa_CJ1ngk1_G5UqhSc9Lf4ju-5gQR_8vGxtU', '_context_show_deleted': False, '_context_tenant': u'358cf3d49042 4818a6365fc4379637a0', 'priority': 'INFO', '_context_read_only': False, '_context_is_admin': True, '_context_project_id': u'358cf3d490424818a6365fc4379637a0', '_context_project_domain': u'default', '_context_timestamp': '2017-12-06T13:21:42.084054', '_context_user_domain': u'default', '_context_user_name': u'admin', 'publisher_id': u'nova-conductor:controller-0.localdomain', 'message_id': u'591d3ba8-82bd-4d33-b0a9-1c821974be71', '_context_remote_address': u'fd00:fd00:fd00:2000::15'}: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging Traceback (most recent call last): 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/notify/messaging.py", line 70, in notify 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 130, in _send_notification 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 582, in send_notification 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging envelope=(version == 2.0), notify=True, retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 547, in _send 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging conn.notify_send(exchange, target.topic, msg, retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1297, in notify_send 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging exchange, msg, routing_key=topic, retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging self.ensure(method, retry=retry, error_callback=_error_callback) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging ret, channel = autoretry_method() 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging return fun(*args, **kwargs) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__ 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging return fun(*args, channel=channels[0], **kwargs), channels[0] 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging method() 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1225, in _publish_and_creates_default_queue 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging queue.declare() 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 605, in declare 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging self._create_queue(nowait=nowait, channel=channel) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 614, in _create_queue 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging self.queue_declare(nowait=nowait, passive=False, channel=channel) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 649, in queue_declare 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging nowait=nowait, 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1165, in queue_declare 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging spec.Queue.DeclareOk, returns_tuple=True, 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 93, in wait 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging self.connection.drain_events(timeout=timeout) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 465, in drain_events 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging return self.blocking_read(timeout) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 470, in blocking_read 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging return self.on_inbound_frame(frame) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 68, in on_frame 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging callback(channel, method_sig, buf, None) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 474, in on_inbound_method 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging method_sig, payload, content, 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 142, in dispatch_method 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging listener(*args) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 293, in _on_close 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging reply_code, reply_text, (class_id, method_id), ChannelError, 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging 2017-12-06 13:22:49.891 24 DEBUG nova.conductor.manager [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] [instance: 29eac6da-bcd1-4b55-87a2-17f5868d6072] block_device_mapping [BlockDeviceMapping(attachment_id=<?>,boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=True,deleted=<?>,deleted_at=<?>,destination_type='local',device_name=None,device_type='disk',disk_bus=None,guest_format=None,id=<?>,image_id='51672902-0005-42b8-922e-9649506799f4',instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=<?>,volume_id=None,volume_size=None)] _create_block_device_mapping /usr/lib/python2.7/site-packages/nova/conductor/manager.py:936 Version-Release number of selected component (if applicable): ------------------------------------------------------------- Images from 12.0-20171201.1 Steps to Reproduce: ------------------- 1. Install UC/OC with 2017-11-29.2 2. Install repos from 2017-12-01.4 3. Update UC 4. Reboot UC 5. Run init-minor-update 6. Perform oc update starting with Controller nodes 7. Try to spawn VM after update is finished Actual results: --------------- Fail to launch vm. Expected results: ----------------- VM is launched Additional info: ---------------- Virtual setup: 3controller + 2computes + 3ceph; uc/oc ssl VM spawned before update with associated floating ip is reachable after update.
Forgot to mention that overcloud deployed with IPv6
This is a dupe of: https://bugzilla.redhat.com/show_bug.cgi?id=1455793 which is closed NOTABUG because the issue is actually in rabbit: https://bugzilla.redhat.com/show_bug.cgi?id=1488499 There is a workaround, which is to disable the versioned notifications in config so that this won't happen, as described here: https://access.redhat.com/solutions/3139721 So I'm closing this as NOTABUG as well.
*** Bug 1550210 has been marked as a duplicate of this bug. ***
Reproduced. Re-opening and switching to documentation. We should include the w/a in the kb[1] with the minor update documentation[2]. [1] https://access.redhat.com/solutions/3139721 [2] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html-single/upgrading_red_hat_openstack_platform/#keeping_openstack_platform_updated
Scoping old BZs. The issue appears to be resolved for OSP12. Sasha, can you verify if this is so? Or do we still need a note?
Yurii, Is the note still needed?