Description of problem: The CPU load of the health manager skyrocketed (CPU load average: 794.80, 467.48, 491.34). Logs show connections to rabbitmq being refused. Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-8.0.4-16.el7ost.noarch openstack-octavia-health-manager-2.0.1-6.d137eaagit.el7ost.noarch openstack-octavia-common-2.0.1-6.d137eaagit.el7ost.noarch openstack-octavia-api-2.0.1-6.d137eaagit.el7ost.noarch openstack-octavia-housekeeping-2.0.1-6.d137eaagit.el7ost.noarch openstack-octavia-worker-2.0.1-6.d137eaagit.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Create load balancer 2. Stop rabbitmq in the controller node Octavia services are connecting to. 3. Observe Octavia logs, processes in controller nodes and their CPU load. Actual results: Very high CPU load. Amphorae would probably start to failover as reported in rhbz#1607276. 2018-08-17 09:31:49.651 1810 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused 2018-08-17 09:31:49.613 1810 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused 2018-08-17 09:31:49.652 1810 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused 2018-08-17 09:31:49.653 1810 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused ==> /var/log/containers/octavia/api.log <== 2018-08-17 09:31:51.643 1 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused ==> /var/log/containers/octavia/health-manager.log <== 2018-08-17 09:31:49.568 1810 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused 2018-08-17 09:31:49.631 1810 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused 2018-08-17 09:31:49.658 1810 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused 2018-08-17 09:31:49.656 1810 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused [root@controller-0 heat-admin]# ps aux | grep octavia 42437 129463 0.0 0.2 367452 84596 ? Ss Aug14 0:01 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 129530 0.0 0.2 588828 81076 ? Sl Aug14 2:54 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 129531 0.4 0.2 371644 81528 ? S Aug14 18:47 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 130349 2.2 0.3 671604 123768 ? Ssl Aug14 90:21 /usr/bin/python2 /usr/bin/octavia-api --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/api.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-api 42437 131439 1.4 0.3 595464 106908 ? Ssl Aug14 60:46 /usr/bin/python2 /usr/bin/octavia-housekeeping --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/housekeeping.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-housekeeping 42437 131605 1.4 0.1 177180 38448 ? Ss Aug14 60:07 octavia-worker: master process [/usr/bin/octavia-worker --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/worker.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-worker] 42437 131698 1.3 0.3 1290132 115248 ? Sl Aug14 54:32 octavia-worker: ConsumerService worker(0) 42437 631264 38.0 3.7 16265304 1218968 ? Sl Aug14 1524:16 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 631265 20.1 2.2 9169856 732844 ? Sl Aug14 807:07 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 631266 31.9 3.1 14336816 1037200 ? Sl Aug14 1280:07 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 631267 26.8 2.7 11699104 903332 ? Sl Aug14 1074:25 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 631268 46.4 4.2 18236484 1402736 ? Sl Aug14 1862:21 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 631269 11.8 2.0 8880656 668704 ? Sl Aug14 473:48 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 631270 11.7 1.9 8197496 637928 ? Sl Aug14 469:27 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager 42437 631271 46.7 4.0 17608884 1311676 ? Sl Aug14 1874:30 /usr/bin/python2 /usr/bin/octavia-health-manager --config-file /usr/share/octavia/octavia-dist.conf --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-health-manager root 679181 0.0 0.0 112708 984 pts/1 S+ 09:33 0:00 grep --color=auto octavia Expected results: [health_manager]/event_streamer_driver option should be set by default to queue_event_streamer. Driver queue_event_streamer is deprecated and will be removed around by the time neutron-lbaas reaches EOL. THT is the component setting octavia::health_manager::event_streamer_driver. Additional info: Restarting affected Octavia services seem to temporarily resolve the issue by reconnecting to a different rabbitmq endpoint running in another controller node.
*** Bug 1631234 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3587
Thanks for article I found it on [url=https://google.com]google.com[/url] [url]https://google.com[/url] (https://google.com)[google.com]