Red Hat Bugzilla – Bug 1511041
Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
Last modified: 2018-02-05 14:15:23 EST
Description of problem: Seeing 100s of errors like this in the neutron logs https://gist.github.com/anonymous/eda3c3589806eddcbff108e76feea29a Version-Release number of selected component (if applicable): OSP 12 Puddle: 2017-10-31.2 How reproducible: 100% Steps to Reproduce: 1. Create 100s of neutron resources 2. 3. Actual results: Seeing this error in the neutron-server logs Expected results: No Errors Additional info:
Created attachment 1349496 [details] Copy of https://gist.github.com/anonymous/eda3c3589806eddcbff108e76feea29a
This looks similar/same as https://bugs.launchpad.net/oslo.messaging/+bug/1504725, adding tracker.
Possibly this issue? https://github.com/celery/py-amqp/pull/121
It looks like this bug which was fixed in python-amqp 2.2.0: https://github.com/celery/py-amqp/pull/128
I could reproduce the bug several times on OSP12 containerized deployment, specifically I see the traceback on nova compute log and then I am not able to run an instance on such compute node unless I restart nova compute service/container so in my case it has fatal consequences. I am triggering the problem by running recovery tests which ungracefully reset controller nodes, especially rabbitmq servers which is relevant to the bug I assume.
I could reproduce the same issue on OSP-12 with OVN. During HA recovery test, after the system was fully recovered I failed to boot new VM. I left the system for few minutes and after that, I could launch new VM.
(In reply to Eran Kuris from comment #6) > I could reproduce the same issue on OSP-12 with OVN. > During HA recovery test, after the system was fully recovered I failed to > boot new VM. > I left the system for few minutes and after that, I could launch new VM. PS I saw this error also on nova-scheduler.log & Neutron server-log.
(In reply to Victor Stinner from comment #4) > It looks like this bug which was fixed in python-amqp 2.2.0: > https://github.com/celery/py-amqp/pull/128 +1, this is exactly the problem.
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=624602
Verified on: python-amqp-2.1.4-2.el7ost.noarch Ran HA recovery tests and was able to boot an instance with no reoccurring messages in the logs mentioned
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462