Bug 1357522 - RabbitMQ should use predefined ports below ephemeral ports range
Summary: RabbitMQ should use predefined ports below ephemeral ports range
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rabbitmq-server
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 10.0 (Newton)
Assignee: Michele Baldessari
QA Contact: Udi Shkalim
URL:
Whiteboard:
Depends On: 1360165
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-18 12:35 UTC by Peter Lemenkov
Modified: 2016-12-14 15:46 UTC (History)
11 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.20160929150845.4cdc4fc.el7ost
Doc Type: Bug Fix
Doc Text:
RabbitMQ would bind to port 35672. However, port 35672 is in the ephemeral range, which leaves the possibility of other services opening up the same port. This could cause RabbitMQ to fail to start. This fix changes the RabbitMQ port to 25672, which is outside of the ephemeral port range. No other service listens on the same port and RabbitMQ starts successfully.
Clone Of:
Environment:
Last Closed: 2016-12-14 15:46:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1623818 0 None None None 2016-09-15 08:05:09 UTC
OpenStack gerrit 345851 0 None MERGED Move rabbit's clustering port away from the ephemeral port range 2020-05-28 03:30:14 UTC
Red Hat Product Errata RHEA-2016:2948 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC

Description Peter Lemenkov 2016-07-18 12:35:52 UTC
Currently RabbitMQ cluster uses a predefined port 35672 for clustering. This port belongs to so-called ephemeral ports range.

Ephemeral ports are the ports kernel assings to application if it doesn't specify which port to open. So there is a small chance that this application being started before RabbitMQ itself could grab this port. Unfortunately we've just saw this in the wild.

Sidenote - if you see "Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",eaddrinuse]} in /var/log/rabbitmq/startup_err then check if some application opened port 35672. Just stop this app, start RabbitMQ, and start this application again.

If we need static predefined port, then we'd better use port 25672. It doesn't belong to ephemeral ports range, so chances are low that anyone opens this port by mistake.

This change will likely require a change in our firewall rules, might require a change in SELinux rules, and certainly requires a change in Director.

Comment 2 Michele Baldessari 2016-07-18 12:56:16 UTC
So in tripleo we set this stuff in puppet/hieradata/controller.yaml

rabbitmq_kernel_variables:
  inet_dist_listen_min: '35672'
  inet_dist_listen_max: '35672'


tripleo::firewall::firewall_rules:
...
  '109 rabbitmq':
    dport:       
      - 4369     
      - 5672     
      - 35672    
...

Peter, are there other settings somewhere (rpm, default config, etc.) that
might affect this port or is the above all there is?

Comment 3 Peter Lemenkov 2016-07-18 13:07:23 UTC
(In reply to Michele Baldessari from comment #2)
> So in tripleo we set this stuff in puppet/hieradata/controller.yaml
> 
> rabbitmq_kernel_variables:
>   inet_dist_listen_min: '35672'
>   inet_dist_listen_max: '35672'
> 
> 
> tripleo::firewall::firewall_rules:
> ...
>   '109 rabbitmq':
>     dport:       
>       - 4369     
>       - 5672     
>       - 35672    
> ...
> 
> Peter, are there other settings somewhere (rpm, default config, etc.) that
> might affect this port or is the above all there is?

Nope. I'm not aware of any other places containing this.

Comment 4 Peter Lemenkov 2016-07-18 14:12:40 UTC
Reducing priority/severity down since it's not very likely issue.

Comment 5 Fabio Massimo Di Nitto 2016-07-28 03:21:22 UTC
should we bump this to osp10?

Comment 6 Michele Baldessari 2016-07-28 06:29:18 UTC
If we can get traction for the selinux part of this bug then osp10 is doable,
otherwise it will osp11 I am afraid

Comment 7 Michele Baldessari 2016-09-15 08:11:40 UTC
As Marian mentioned per mail, this is hitting us harder than before due to the HA-NG architecture: services will start before rabbit and so the chances of the ephemeral port being taken are higher

Comment 12 errata-xmlrpc 2016-12-14 15:46:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.