Bug 1267736 - queues are not distributed in an osp-d / osp7 distribution
queues are not distributed in an osp-d / osp7 distribution
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-oslo-messaging (Show other bugs)
7.0 (Kilo)
x86_64 Linux
high Severity urgent
: async
: 7.0 (Kilo)
Assigned To: hguemar
Udi Shkalim
: Triaged, ZStream
: 1290538 (view as bug list)
Depends On:
Blocks: 1310799
  Show dependency treegraph
 
Reported: 2015-09-30 16:01 EDT by Mark Wagner
Modified: 2016-04-26 15:30 EDT (History)
17 users (show)

See Also:
Fixed In Version: python-oslo-messaging-1.8.3-2.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1310799 (view as bug list)
Environment:
Last Closed: 2016-01-20 08:12:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 230592 None None None Never
OpenStack gerrit 231039 None None None Never

  None (edit)
Description Mark Wagner 2015-09-30 16:01:15 EDT
Description of problem:

In the scale lab we are seeing that the overwhelming majority of the rabbit connections happen on one of the three controllers. 

Version-Release number of selected component (if applicable):


How reproducible:
every time

Steps to Reproduce:
1.pcs cluster standby --all (wait a bit)
2.pcs cluster unstandby --all
3. monitor sockets used on all three controllers with `rabbitmqctl status`

Actual results:

root@overcloud-controller-0
 {file_descriptors,[{total_limit,3996},
                    {total_used,1542},
                    {sockets_limit,3594},
                    {sockets_used,1540}]},
 {processes,[{limit,1048576},{used,18726}]},
 {run_queue,0},
 {uptime,639}]

root@overcloud-controller-1
 {file_descriptors,[{total_limit,3996},
                    {total_used,6},
                    {sockets_limit,3594},
                    {sockets_used,4}]},
 {processes,[{limit,1048576},{used,3331}]},
 {run_queue,0},
 {uptime,573}]


root@overcloud-controller-2
 {file_descriptors,[{total_limit,3996},
                    {total_used,5},
                    {sockets_limit,3594},
                    {sockets_used,3}]},
 {processes,[{limit,1048576},{used,3326}]},
 {run_queue,0},
 {uptime,565}]

Expected results:

A more even distribution

Additional info:
Comment 2 John Eckersberg 2015-10-01 12:28:40 EDT
I've just checked, this worked correctly for Juno/OSP6.  During the Kilo cycle, this commit broke the behavior:

https://github.com/openstack/oslo.messaging/commit/973301aa70527171749fa34897276c43898aeeb2#diff-09f54761ad97db5984a04016ceafa15cL484

Before, it would generate the list of broker_params, which contained an entry for each server configured in rabbit_hosts.  Then it would shuffle the list and hand it off to kombu.

The important thing to note is that kombu will always connect to the first node in the list.  It does support a failover_strategy, which is being set to "shuffle", but that strategy *only matters on failover*.  It does not have an effect when deciding which node to connect to under normal circumstances.

From Kilo forward, the kombu URL is generated from rabbit_hosts, but the ordering is *not* randomized, so whichever host is listed first in the configuration file will end up receiving the majority if not all of the connections.

So, the poor man's load balancing behavior we were relying on is broken in OSP7.

I see three options to correct this, in order of most to least preferred:

1.  Patch oslo.messaging to shuffle rabbit_hosts before generating the URL to hand off to kombu.

2.  Change the architecture as to route AMQP traffic through HAProxy.

3.  Modify director to shuffle the addresses before it generates the value of rabbit_hosts in the configuration files.
Comment 3 John Eckersberg 2015-10-01 12:36:17 EDT
Perry, thoughts on the above options?
Comment 4 Perry Myers 2015-10-01 16:23:05 EDT
(In reply to John Eckersberg from comment #3)
> Perry, thoughts on the above options?

Option 1 is what I would do. Least invasive, works outside of Director, doesn't require architecture changes.
Comment 6 John Eckersberg 2015-10-02 13:51:04 EDT
Upstream review - https://review.openstack.org/#/c/230592/
Comment 7 John Eckersberg 2015-10-05 10:01:26 EDT
https://code.engineering.redhat.com/gerrit/#/c/58841/
Comment 10 Flavio Percoco 2015-10-19 07:01:02 EDT
I'm following-up on this upstream. There was an issue with the gate that is now, hopefully, fixed.
Comment 15 John Eckersberg 2015-12-10 13:30:17 EST
*** Bug 1290538 has been marked as a duplicate of this bug. ***
Comment 22 hguemar 2015-12-17 17:27:53 EST
@Dave: go ahead with this build as hotfix, the newer build has multiple patches that needs to be tested by QE. You may keep it around if you find other issues.
Comment 25 Udi Shkalim 2016-01-11 14:30:46 EST
Deployed ospd-7.2 latest puddle -  2015-12-16.1

The package of oslo.messaging :
[root@overcloud-controller-2 ~]# rpm -qa | grep python-oslo-messaging
python-oslo-messaging-1.8.3-1.el7ost.noarch

which does not contain the fix. Please advise.

Udi
Comment 26 hguemar 2016-01-11 16:23:08 EST
Update to the latest puddle which is 2016-01-06.1
python-oslo-messaging-1.8.3-3.el7ost.noarch.rpm
Comment 27 Udi Shkalim 2016-01-14 09:02:46 EST
Needed to build the images for the overcloud in order to get latest core packages.
followed the steps to reproduce in the description.
Verified on python-oslo-messaging-1.8.3-3.el7ost.noarch OSPD 7.2
Connection are spread across the nodes
[root@overcloud-controller-2 ~]# rabbitmqctl status
Status of node 'rabbit@overcloud-controller-2' ...
 {file_descriptors,[{total_limit,3996},
                    {total_used,78},
                    {sockets_limit,3594},
                    {sockets_used,76}]},
 {processes,[{limit,1048576},{used,1422}]},
 {run_queue,0},
 {uptime,323}]
...done.



[root@overcloud-controller-1 ~]# rabbitmqctl status
Status of node 'rabbit@overcloud-controller-1' ...
 {file_descriptors,[{total_limit,3996},
                    {total_used,43},
                    {sockets_limit,3594},
                    {sockets_used,41}]},
 {processes,[{limit,1048576},{used,983}]},
 {run_queue,0},
 {uptime,299}]
...done.

[root@overcloud-controller-0 ~]#  rabbitmqctl status
Status of node 'rabbit@overcloud-controller-0' ...

 {file_descriptors,[{total_limit,3996},
                    {total_used,62},
                    {sockets_limit,3594},
                    {sockets_used,60}]},
 {processes,[{limit,1048576},{used,1199}]},
 {run_queue,0},
 {uptime,305}]
...done.
Comment 29 errata-xmlrpc 2016-01-20 08:12:11 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:0047

Note You need to log in before you can comment on or make changes to this bug.