Bug 1747226
| Summary: | Unable to connect to AMQP server on HOSTNAME:5672 after None tries: 'NoneType' object has no attribute '__getitem__' | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Brendan Shephard <bshephar> | |
| Component: | python-amqp | Assignee: | RHOS Maint <rhos-maint> | |
| Status: | CLOSED ERRATA | QA Contact: | pkomarov | |
| Severity: | high | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 13.0 (Queens) | CC: | apevec, athomas, chjones, coldford, dasmith, hberaud, jeckersb, lhh, lmiccini, pkomarov, pmannidi, rhos-maint, rosingh, supadhya | |
| Target Milestone: | async | Keywords: | Triaged, ZStream | |
| Target Release: | 13.0 (Queens) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | python-amqp-2.3.2-5.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: PyAMQP doesn't inform client that the connexion have been reset.
Consequence: Client are not informed that the connection have been reset and continue to try to publish message and then they fails to publish the messages.
Fix: Raise a connection exception to inform client to handle retry, the connection will be properly reinitialized.
Result: Client properly reconnect to the rabbitmq server and can now publish messages. Also clients stop to put related errors in logs.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1751595 (view as bug list) | Environment: | ||
| Last Closed: | 2019-11-07 14:04:36 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1751595 | |||
The big question is why are these running on the controllers instead of in containers? Verified HOTFIX for regression testing : [stack@undercloud-0 ~]$ ansible overcloud_nodes -b -mshell -a 'for i in `docker ps -q`;do docker exec $i sh -c "rpm -qa|grep amqp";done' compute-0 | SUCCESS | rc=0 >> python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch compute-1 | SUCCESS | rc=0 >> python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch controller-2 | FAILED | rc=1 >> python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarchnon-zero return code controller-1 | FAILED | rc=1 >> python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarchnon-zero return code controller-0 | FAILED | rc=1 >> python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarch python2-amqp-2.3.2-5.el7ost.noarchnon-zero return code [stack@undercloud-0 ~]$ grep -r RECAP -A20 ansible_sts_results/ ansible_sts_results/01_POST_DEPLOY_CHECK.log:PLAY RECAP **************************************************************************************************************************************************** ansible_sts_results/01_POST_DEPLOY_CHECK.log-controller-0 : ok=3 changed=3 unreachable=0 failed=0 ansible_sts_results/01_POST_DEPLOY_CHECK.log-controller-1 : ok=1 changed=1 unreachable=0 failed=0 ansible_sts_results/01_POST_DEPLOY_CHECK.log-controller-2 : ok=1 changed=1 unreachable=0 failed=0 ansible_sts_results/01_POST_DEPLOY_CHECK.log-undercloud : ok=5 changed=4 unreachable=0 failed=0 ansible_sts_results/01_POST_DEPLOY_CHECK.log- -- ansible_sts_results/02_HARD_RESET_ALL_CONTROLLERS.log:PLAY RECAP **************************************************************************************************************************************************** ansible_sts_results/02_HARD_RESET_ALL_CONTROLLERS.log-controller-0 : ok=5 changed=5 unreachable=0 failed=0 ansible_sts_results/02_HARD_RESET_ALL_CONTROLLERS.log-controller-1 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/02_HARD_RESET_ALL_CONTROLLERS.log-controller-2 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/02_HARD_RESET_ALL_CONTROLLERS.log-undercloud : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/02_HARD_RESET_ALL_CONTROLLERS.log- -- ansible_sts_results/03_HARD_RESET_CONTROLLER_MAIN_VIP.log:PLAY RECAP ************************************************************************************************************ ansible_sts_results/03_HARD_RESET_CONTROLLER_MAIN_VIP.log-controller-0 : ok=5 changed=5 unreachable=0 failed=0 ansible_sts_results/03_HARD_RESET_CONTROLLER_MAIN_VIP.log-controller-1 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/03_HARD_RESET_CONTROLLER_MAIN_VIP.log-controller-2 : ok=0 changed=0 unreachable=1 failed=0 ansible_sts_results/03_HARD_RESET_CONTROLLER_MAIN_VIP.log-undercloud : ok=8 changed=7 unreachable=0 failed=0 ansible_sts_results/03_HARD_RESET_CONTROLLER_MAIN_VIP.log- -- ansible_sts_results/04_HARD_RESET_CONTROLLER_NON_VIP.log:PLAY RECAP ************************************************************************************ ansible_sts_results/04_HARD_RESET_CONTROLLER_NON_VIP.log-controller-0 : ok=5 changed=5 unreachable=0 failed=0 ansible_sts_results/04_HARD_RESET_CONTROLLER_NON_VIP.log-controller-1 : ok=0 changed=0 unreachable=1 failed=0 ansible_sts_results/04_HARD_RESET_CONTROLLER_NON_VIP.log-controller-2 : ok=0 changed=0 unreachable=1 failed=0 ansible_sts_results/04_HARD_RESET_CONTROLLER_NON_VIP.log-undercloud : ok=8 changed=7 unreachable=0 failed=0 ansible_sts_results/04_HARD_RESET_CONTROLLER_NON_VIP.log- -- ansible_sts_results/08_HARD_RESET_CONTROLLER_INTERNAL_VIP.log:PLAY RECAP ********************************************************************* ansible_sts_results/08_HARD_RESET_CONTROLLER_INTERNAL_VIP.log-controller-0 : ok=6 changed=6 unreachable=0 failed=0 ansible_sts_results/08_HARD_RESET_CONTROLLER_INTERNAL_VIP.log-controller-1 : ok=3 changed=3 unreachable=0 failed=0 ansible_sts_results/08_HARD_RESET_CONTROLLER_INTERNAL_VIP.log-controller-2 : ok=3 changed=3 unreachable=0 failed=0 ansible_sts_results/08_HARD_RESET_CONTROLLER_INTERNAL_VIP.log-undercloud : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/08_HARD_RESET_CONTROLLER_INTERNAL_VIP.log- -- ansible_sts_results/09_HARD_RESET_CONTROLLER_REDIS_VIP.log:PLAY RECAP ************************************************************************************ ansible_sts_results/09_HARD_RESET_CONTROLLER_REDIS_VIP.log-controller-0 : ok=6 changed=6 unreachable=0 failed=0 ansible_sts_results/09_HARD_RESET_CONTROLLER_REDIS_VIP.log-controller-1 : ok=3 changed=3 unreachable=0 failed=0 ansible_sts_results/09_HARD_RESET_CONTROLLER_REDIS_VIP.log-controller-2 : ok=3 changed=3 unreachable=0 failed=0 ansible_sts_results/09_HARD_RESET_CONTROLLER_REDIS_VIP.log-undercloud : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/09_HARD_RESET_CONTROLLER_REDIS_VIP.log- -- ansible_sts_results/10_NETWORK_DISRUPTOR_CONTROLLER_MAIN_VIP.log:PLAY RECAP ************************************************************************************ ansible_sts_results/10_NETWORK_DISRUPTOR_CONTROLLER_MAIN_VIP.log-controller-0 : ok=5 changed=5 unreachable=0 failed=0 ansible_sts_results/10_NETWORK_DISRUPTOR_CONTROLLER_MAIN_VIP.log-controller-1 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/10_NETWORK_DISRUPTOR_CONTROLLER_MAIN_VIP.log-controller-2 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/10_NETWORK_DISRUPTOR_CONTROLLER_MAIN_VIP.log-undercloud : ok=8 changed=7 unreachable=0 failed=0 ansible_sts_results/10_NETWORK_DISRUPTOR_CONTROLLER_MAIN_VIP.log- -- ansible_sts_results/12_HARD_RESET_ONE_BY_ONE_CONTROLLER.log:PLAY RECAP ************************************************************************************ ansible_sts_results/12_HARD_RESET_ONE_BY_ONE_CONTROLLER.log-controller-0 : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/12_HARD_RESET_ONE_BY_ONE_CONTROLLER.log-controller-1 : ok=4 changed=3 unreachable=0 failed=0 ansible_sts_results/12_HARD_RESET_ONE_BY_ONE_CONTROLLER.log-controller-2 : ok=4 changed=3 unreachable=0 failed=0 ansible_sts_results/12_HARD_RESET_ONE_BY_ONE_CONTROLLER.log-undercloud : ok=5 changed=5 unreachable=0 failed=0 ansible_sts_results/12_HARD_RESET_ONE_BY_ONE_CONTROLLER.log- -- ansible_sts_results/14_KILL_RABBIT_RANDOM_CONTROLLER.log:PLAY RECAP ************************************************************************************ ansible_sts_results/14_KILL_RABBIT_RANDOM_CONTROLLER.log-controller-0 : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/14_KILL_RABBIT_RANDOM_CONTROLLER.log-controller-1 : ok=1 changed=1 unreachable=0 failed=0 ansible_sts_results/14_KILL_RABBIT_RANDOM_CONTROLLER.log-controller-2 : ok=1 changed=1 unreachable=0 failed=0 ansible_sts_results/14_KILL_RABBIT_RANDOM_CONTROLLER.log-undercloud : ok=5 changed=5 unreachable=0 failed=0 ansible_sts_results/14_KILL_RABBIT_RANDOM_CONTROLLER.log- -- ansible_sts_results/16_KILL_ALL_GALERA.log:PLAY RECAP ************************************************************************************ ansible_sts_results/16_KILL_ALL_GALERA.log-controller-0 : ok=5 changed=5 unreachable=0 failed=0 ansible_sts_results/16_KILL_ALL_GALERA.log-controller-1 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/16_KILL_ALL_GALERA.log-controller-2 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/16_KILL_ALL_GALERA.log-undercloud : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/16_KILL_ALL_GALERA.log- -- ansible_sts_results/17_REMOVE_ALL_GRASTATE_GALERA.log:PLAY RECAP ************************************************************************************ ansible_sts_results/17_REMOVE_ALL_GRASTATE_GALERA.log-controller-0 : ok=6 changed=6 unreachable=0 failed=0 ansible_sts_results/17_REMOVE_ALL_GRASTATE_GALERA.log-controller-1 : ok=3 changed=3 unreachable=0 failed=0 ansible_sts_results/17_REMOVE_ALL_GRASTATE_GALERA.log-controller-2 : ok=3 changed=3 unreachable=0 failed=0 ansible_sts_results/17_REMOVE_ALL_GRASTATE_GALERA.log-undercloud : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/17_REMOVE_ALL_GRASTATE_GALERA.log- -- ansible_sts_results/18_REMOVE_ONE_GRASTATE_GALERA.log:PLAY RECAP ************************************************************************************ ansible_sts_results/18_REMOVE_ONE_GRASTATE_GALERA.log-controller-0 : ok=7 changed=7 unreachable=0 failed=0 ansible_sts_results/18_REMOVE_ONE_GRASTATE_GALERA.log-controller-1 : ok=1 changed=1 unreachable=0 failed=0 ansible_sts_results/18_REMOVE_ONE_GRASTATE_GALERA.log-controller-2 : ok=1 changed=1 unreachable=0 failed=0 ansible_sts_results/18_REMOVE_ONE_GRASTATE_GALERA.log-undercloud : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/18_REMOVE_ONE_GRASTATE_GALERA.log- -- ansible_sts_results/19_GENERATE_BIG_DATA_GALERA.log:PLAY RECAP ************************************************************************************ ansible_sts_results/19_GENERATE_BIG_DATA_GALERA.log-controller-0 : ok=7 changed=7 unreachable=0 failed=0 ansible_sts_results/19_GENERATE_BIG_DATA_GALERA.log-controller-1 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/19_GENERATE_BIG_DATA_GALERA.log-controller-2 : ok=2 changed=2 unreachable=0 failed=0 ansible_sts_results/19_GENERATE_BIG_DATA_GALERA.log-undercloud : ok=6 changed=5 unreachable=0 failed=0 ansible_sts_results/19_GENERATE_BIG_DATA_GALERA.log- If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to -. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3791 *** Bug 1776083 has been marked as a duplicate of this bug. *** The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |
Description of problem: While trying to create a large number of VM's, some of them are getting stuck in BUILD state. The only relevant error we can see is coming from nova-conductor and is: Unable to connect to AMQP server on HOSTNAME:5672 after None tries: 'NoneType' object has no attribute '__getitem__' A restart of all Nova services seems to have resolved this issue. However, we had a repeat over night and saw the same errors in the heat-engine.log. Version-Release number of selected component (if applicable): Red Hat OpenStack Platform release 13.0.7 (Queens) python2-oslo-messaging-5.35.4-1.el7ost.noarch How reproducible: Tough to reproduce. Steps to Reproduce: Initially, before restarting Nova, we could reproduce it by running: for i in {10..60}; do openstack server create --image a3806e23-1df5-40b0-baf3-f78ef7b1bf4c --nic net-id=54d8b69b-e8a0-4ce7-87ca-37b34fc368a2 --flavor VM1-flavor --availability-zone nova:compute-$i; done If we added a --wait in there, it worked a lot better. But we still observed 1 or two failures when doing it that way. Actual results: Some Instances would get stuck in BUILD state and not move. Expected results: Instances would complete or ERROR Additional info: After restarting all nova_* containers, we were able to execute: for i in {10..100}; do openstack server create --image a3806e23-1df5-40b0-baf3-f78ef7b1bf4c --nic net-id=54d8b69b-e8a0-4ce7-87ca-37b34fc368a2 --flavor VM1-flavor; done And have all instances build successfully.