Bug 1511041 - Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
Summary: Failed to consume message from queue: 'NoneType' object has no attribute 'ite...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-amqp
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 12.0 (Pike)
Assignee: John Eckersberg
QA Contact: Udi Shkalim
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-08 15:23 UTC by Sai Sindhur Malleni
Modified: 2018-02-05 19:15 UTC (History)
10 users (show)

Fixed In Version: python-amqp-2.1.4-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-13 22:20:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Copy of https://gist.github.com/anonymous/eda3c3589806eddcbff108e76feea29a (3.36 KB, text/plain)
2017-11-08 15:45 UTC, Victor Stinner
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1504725 0 None None None 2017-11-08 18:47:31 UTC
Red Hat Product Errata RHEA-2017:3462 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Internal Links: 1747226

Description Sai Sindhur Malleni 2017-11-08 15:23:43 UTC
Description of problem:
Seeing 100s of errors like this in the neutron logs
https://gist.github.com/anonymous/eda3c3589806eddcbff108e76feea29a
Version-Release number of selected component (if applicable):
OSP 12
Puddle: 2017-10-31.2

How reproducible:
100%

Steps to Reproduce:
1. Create 100s of neutron resources
2.
3.

Actual results:
Seeing this error in the neutron-server logs

Expected results:
No Errors

Additional info:

Comment 2 John Eckersberg 2017-11-08 18:47:32 UTC
This looks similar/same as https://bugs.launchpad.net/oslo.messaging/+bug/1504725, adding tracker.

Comment 3 John Eckersberg 2017-11-08 22:12:54 UTC
Possibly this issue?

https://github.com/celery/py-amqp/pull/121

Comment 4 Victor Stinner 2017-11-13 10:47:29 UTC
It looks like this bug which was fixed in python-amqp 2.2.0: https://github.com/celery/py-amqp/pull/128

Comment 5 Marian Krcmarik 2017-11-13 11:05:39 UTC
I could reproduce the bug several times on OSP12 containerized deployment, specifically I see the traceback on nova compute log and then I am not able to run an instance on such compute node unless I restart nova compute service/container so in my case it has fatal consequences.
I am triggering the problem by running recovery tests which ungracefully reset controller nodes, especially rabbitmq servers which is relevant to the bug I assume.

Comment 6 Eran Kuris 2017-11-13 12:39:19 UTC
I could reproduce the same issue on OSP-12 with OVN.
During HA recovery test, after the system was fully recovered I failed to boot new VM.
I left the system for few minutes and after that, I could launch new VM.

Comment 7 Eran Kuris 2017-11-13 12:40:32 UTC
(In reply to Eran Kuris from comment #6)
> I could reproduce the same issue on OSP-12 with OVN.
> During HA recovery test, after the system was fully recovered I failed to
> boot new VM.
> I left the system for few minutes and after that, I could launch new VM.

PS 
I saw this error also on nova-scheduler.log & Neutron server-log.

Comment 8 John Eckersberg 2017-11-13 14:59:46 UTC
(In reply to Victor Stinner from comment #4)
> It looks like this bug which was fixed in python-amqp 2.2.0:
> https://github.com/celery/py-amqp/pull/128

+1, this is exactly the problem.

Comment 11 Udi Shkalim 2017-11-20 15:26:08 UTC
Verified on: python-amqp-2.1.4-2.el7ost.noarch

Ran HA recovery tests and was able to boot an instance with no reoccurring messages in the logs mentioned

Comment 15 errata-xmlrpc 2017-12-13 22:20:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.