Bug 1267736 - queues are not distributed in an osp-d / osp7 distribution
Summary: queues are not distributed in an osp-d / osp7 distribution
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-oslo-messaging
Version: 7.0 (Kilo)
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: async
: 7.0 (Kilo)
Assignee: hguemar
QA Contact: Udi Shkalim
URL:
Whiteboard:
: 1290538 (view as bug list)
Depends On:
Blocks: 1310799
TreeView+ depends on / blocked
 
Reported: 2015-09-30 20:01 UTC by Mark Wagner
Modified: 2019-10-10 10:16 UTC (History)
17 users (show)

Fixed In Version: python-oslo-messaging-1.8.3-2.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1310799 (view as bug list)
Environment:
Last Closed: 2016-01-20 13:12:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 230592 0 None None None Never
OpenStack gerrit 231039 0 None None None Never
Red Hat Product Errata RHBA-2016:0047 0 normal SHIPPED_LIVE python-oslo-messaging bug fix advisory 2016-01-20 18:10:29 UTC

Description Mark Wagner 2015-09-30 20:01:15 UTC
Description of problem:

In the scale lab we are seeing that the overwhelming majority of the rabbit connections happen on one of the three controllers. 

Version-Release number of selected component (if applicable):


How reproducible:
every time

Steps to Reproduce:
1.pcs cluster standby --all (wait a bit)
2.pcs cluster unstandby --all
3. monitor sockets used on all three controllers with `rabbitmqctl status`

Actual results:

root@overcloud-controller-0
 {file_descriptors,[{total_limit,3996},
                    {total_used,1542},
                    {sockets_limit,3594},
                    {sockets_used,1540}]},
 {processes,[{limit,1048576},{used,18726}]},
 {run_queue,0},
 {uptime,639}]

root@overcloud-controller-1
 {file_descriptors,[{total_limit,3996},
                    {total_used,6},
                    {sockets_limit,3594},
                    {sockets_used,4}]},
 {processes,[{limit,1048576},{used,3331}]},
 {run_queue,0},
 {uptime,573}]


root@overcloud-controller-2
 {file_descriptors,[{total_limit,3996},
                    {total_used,5},
                    {sockets_limit,3594},
                    {sockets_used,3}]},
 {processes,[{limit,1048576},{used,3326}]},
 {run_queue,0},
 {uptime,565}]

Expected results:

A more even distribution

Additional info:

Comment 2 John Eckersberg 2015-10-01 16:28:40 UTC
I've just checked, this worked correctly for Juno/OSP6.  During the Kilo cycle, this commit broke the behavior:

https://github.com/openstack/oslo.messaging/commit/973301aa70527171749fa34897276c43898aeeb2#diff-09f54761ad97db5984a04016ceafa15cL484

Before, it would generate the list of broker_params, which contained an entry for each server configured in rabbit_hosts.  Then it would shuffle the list and hand it off to kombu.

The important thing to note is that kombu will always connect to the first node in the list.  It does support a failover_strategy, which is being set to "shuffle", but that strategy *only matters on failover*.  It does not have an effect when deciding which node to connect to under normal circumstances.

From Kilo forward, the kombu URL is generated from rabbit_hosts, but the ordering is *not* randomized, so whichever host is listed first in the configuration file will end up receiving the majority if not all of the connections.

So, the poor man's load balancing behavior we were relying on is broken in OSP7.

I see three options to correct this, in order of most to least preferred:

1.  Patch oslo.messaging to shuffle rabbit_hosts before generating the URL to hand off to kombu.

2.  Change the architecture as to route AMQP traffic through HAProxy.

3.  Modify director to shuffle the addresses before it generates the value of rabbit_hosts in the configuration files.

Comment 3 John Eckersberg 2015-10-01 16:36:17 UTC
Perry, thoughts on the above options?

Comment 4 Perry Myers 2015-10-01 20:23:05 UTC
(In reply to John Eckersberg from comment #3)
> Perry, thoughts on the above options?

Option 1 is what I would do. Least invasive, works outside of Director, doesn't require architecture changes.

Comment 6 John Eckersberg 2015-10-02 17:51:04 UTC
Upstream review - https://review.openstack.org/#/c/230592/

Comment 7 John Eckersberg 2015-10-05 14:01:26 UTC
https://code.engineering.redhat.com/gerrit/#/c/58841/

Comment 10 Flavio Percoco 2015-10-19 11:01:02 UTC
I'm following-up on this upstream. There was an issue with the gate that is now, hopefully, fixed.

Comment 15 John Eckersberg 2015-12-10 18:30:17 UTC
*** Bug 1290538 has been marked as a duplicate of this bug. ***

Comment 22 hguemar 2015-12-17 22:27:53 UTC
@Dave: go ahead with this build as hotfix, the newer build has multiple patches that needs to be tested by QE. You may keep it around if you find other issues.

Comment 25 Udi Shkalim 2016-01-11 19:30:46 UTC
Deployed ospd-7.2 latest puddle -  2015-12-16.1

The package of oslo.messaging :
[root@overcloud-controller-2 ~]# rpm -qa | grep python-oslo-messaging
python-oslo-messaging-1.8.3-1.el7ost.noarch

which does not contain the fix. Please advise.

Udi

Comment 26 hguemar 2016-01-11 21:23:08 UTC
Update to the latest puddle which is 2016-01-06.1
python-oslo-messaging-1.8.3-3.el7ost.noarch.rpm

Comment 27 Udi Shkalim 2016-01-14 14:02:46 UTC
Needed to build the images for the overcloud in order to get latest core packages.
followed the steps to reproduce in the description.
Verified on python-oslo-messaging-1.8.3-3.el7ost.noarch OSPD 7.2
Connection are spread across the nodes
[root@overcloud-controller-2 ~]# rabbitmqctl status
Status of node 'rabbit@overcloud-controller-2' ...
 {file_descriptors,[{total_limit,3996},
                    {total_used,78},
                    {sockets_limit,3594},
                    {sockets_used,76}]},
 {processes,[{limit,1048576},{used,1422}]},
 {run_queue,0},
 {uptime,323}]
...done.



[root@overcloud-controller-1 ~]# rabbitmqctl status
Status of node 'rabbit@overcloud-controller-1' ...
 {file_descriptors,[{total_limit,3996},
                    {total_used,43},
                    {sockets_limit,3594},
                    {sockets_used,41}]},
 {processes,[{limit,1048576},{used,983}]},
 {run_queue,0},
 {uptime,299}]
...done.

[root@overcloud-controller-0 ~]#  rabbitmqctl status
Status of node 'rabbit@overcloud-controller-0' ...

 {file_descriptors,[{total_limit,3996},
                    {total_used,62},
                    {sockets_limit,3594},
                    {sockets_used,60}]},
 {processes,[{limit,1048576},{used,1199}]},
 {run_queue,0},
 {uptime,305}]
...done.

Comment 29 errata-xmlrpc 2016-01-20 13:12:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:0047


Note You need to log in before you can comment on or make changes to this bug.