Bug 1267736
| Summary: | queues are not distributed in an osp-d / osp7 distribution | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Mark Wagner <mwagner> | |
| Component: | python-oslo-messaging | Assignee: | hguemar | |
| Status: | CLOSED ERRATA | QA Contact: | Udi Shkalim <ushkalim> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | high | |||
| Version: | 7.0 (Kilo) | CC: | apevec, astellwa, clincoln, dmaley, fpercoco, hguemar, jeckersb, jmelvin, johfulto, jschluet, jtaleric, lhh, michele, pneedle, sclewis, tdunnon, yeylon | |
| Target Milestone: | async | Keywords: | Triaged, ZStream | |
| Target Release: | 7.0 (Kilo) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | python-oslo-messaging-1.8.3-2.el7ost | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1310799 (view as bug list) | Environment: | ||
| Last Closed: | 2016-01-20 13:12:11 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1310799 | |||
I've just checked, this worked correctly for Juno/OSP6. During the Kilo cycle, this commit broke the behavior: https://github.com/openstack/oslo.messaging/commit/973301aa70527171749fa34897276c43898aeeb2#diff-09f54761ad97db5984a04016ceafa15cL484 Before, it would generate the list of broker_params, which contained an entry for each server configured in rabbit_hosts. Then it would shuffle the list and hand it off to kombu. The important thing to note is that kombu will always connect to the first node in the list. It does support a failover_strategy, which is being set to "shuffle", but that strategy *only matters on failover*. It does not have an effect when deciding which node to connect to under normal circumstances. From Kilo forward, the kombu URL is generated from rabbit_hosts, but the ordering is *not* randomized, so whichever host is listed first in the configuration file will end up receiving the majority if not all of the connections. So, the poor man's load balancing behavior we were relying on is broken in OSP7. I see three options to correct this, in order of most to least preferred: 1. Patch oslo.messaging to shuffle rabbit_hosts before generating the URL to hand off to kombu. 2. Change the architecture as to route AMQP traffic through HAProxy. 3. Modify director to shuffle the addresses before it generates the value of rabbit_hosts in the configuration files. Perry, thoughts on the above options? (In reply to John Eckersberg from comment #3) > Perry, thoughts on the above options? Option 1 is what I would do. Least invasive, works outside of Director, doesn't require architecture changes. Upstream review - https://review.openstack.org/#/c/230592/ I'm following-up on this upstream. There was an issue with the gate that is now, hopefully, fixed. *** Bug 1290538 has been marked as a duplicate of this bug. *** @Dave: go ahead with this build as hotfix, the newer build has multiple patches that needs to be tested by QE. You may keep it around if you find other issues. Deployed ospd-7.2 latest puddle - 2015-12-16.1 The package of oslo.messaging : [root@overcloud-controller-2 ~]# rpm -qa | grep python-oslo-messaging python-oslo-messaging-1.8.3-1.el7ost.noarch which does not contain the fix. Please advise. Udi Update to the latest puddle which is 2016-01-06.1 python-oslo-messaging-1.8.3-3.el7ost.noarch.rpm Needed to build the images for the overcloud in order to get latest core packages.
followed the steps to reproduce in the description.
Verified on python-oslo-messaging-1.8.3-3.el7ost.noarch OSPD 7.2
Connection are spread across the nodes
[root@overcloud-controller-2 ~]# rabbitmqctl status
Status of node 'rabbit@overcloud-controller-2' ...
{file_descriptors,[{total_limit,3996},
{total_used,78},
{sockets_limit,3594},
{sockets_used,76}]},
{processes,[{limit,1048576},{used,1422}]},
{run_queue,0},
{uptime,323}]
...done.
[root@overcloud-controller-1 ~]# rabbitmqctl status
Status of node 'rabbit@overcloud-controller-1' ...
{file_descriptors,[{total_limit,3996},
{total_used,43},
{sockets_limit,3594},
{sockets_used,41}]},
{processes,[{limit,1048576},{used,983}]},
{run_queue,0},
{uptime,299}]
...done.
[root@overcloud-controller-0 ~]# rabbitmqctl status
Status of node 'rabbit@overcloud-controller-0' ...
{file_descriptors,[{total_limit,3996},
{total_used,62},
{sockets_limit,3594},
{sockets_used,60}]},
{processes,[{limit,1048576},{used,1199}]},
{run_queue,0},
{uptime,305}]
...done.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:0047 |
Description of problem: In the scale lab we are seeing that the overwhelming majority of the rabbit connections happen on one of the three controllers. Version-Release number of selected component (if applicable): How reproducible: every time Steps to Reproduce: 1.pcs cluster standby --all (wait a bit) 2.pcs cluster unstandby --all 3. monitor sockets used on all three controllers with `rabbitmqctl status` Actual results: root@overcloud-controller-0 {file_descriptors,[{total_limit,3996}, {total_used,1542}, {sockets_limit,3594}, {sockets_used,1540}]}, {processes,[{limit,1048576},{used,18726}]}, {run_queue,0}, {uptime,639}] root@overcloud-controller-1 {file_descriptors,[{total_limit,3996}, {total_used,6}, {sockets_limit,3594}, {sockets_used,4}]}, {processes,[{limit,1048576},{used,3331}]}, {run_queue,0}, {uptime,573}] root@overcloud-controller-2 {file_descriptors,[{total_limit,3996}, {total_used,5}, {sockets_limit,3594}, {sockets_used,3}]}, {processes,[{limit,1048576},{used,3326}]}, {run_queue,0}, {uptime,565}] Expected results: A more even distribution Additional info: