Bug 1357522

Summary:	RabbitMQ should use predefined ports below ephemeral ports range
Product:	Red Hat OpenStack	Reporter:	Peter Lemenkov <plemenko>
Component:	rabbitmq-server	Assignee:	Michele Baldessari <michele>
Status:	CLOSED ERRATA	QA Contact:	Udi Shkalim <ushkalim>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	9.0 (Mitaka)	CC:	apevec, dmacpher, fdinitto, jeckersb, jjoyce, lhh, michele, mkrcmari, oblaut, royoung, srevivo
Target Milestone:	rc	Keywords:	FutureFeature, Triaged
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-5.0.0-0.20160929150845.4cdc4fc.el7ost	Doc Type:	Bug Fix
Doc Text:	RabbitMQ would bind to port 35672. However, port 35672 is in the ephemeral range, which leaves the possibility of other services opening up the same port. This could cause RabbitMQ to fail to start. This fix changes the RabbitMQ port to 25672, which is outside of the ephemeral port range. No other service listens on the same port and RabbitMQ starts successfully.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-14 15:46:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1360165
Bug Blocks:

Description Peter Lemenkov 2016-07-18 12:35:52 UTC

Currently RabbitMQ cluster uses a predefined port 35672 for clustering. This port belongs to so-called ephemeral ports range.

Ephemeral ports are the ports kernel assings to application if it doesn't specify which port to open. So there is a small chance that this application being started before RabbitMQ itself could grab this port. Unfortunately we've just saw this in the wild.

Sidenote - if you see "Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",eaddrinuse]} in /var/log/rabbitmq/startup_err then check if some application opened port 35672. Just stop this app, start RabbitMQ, and start this application again.

If we need static predefined port, then we'd better use port 25672. It doesn't belong to ephemeral ports range, so chances are low that anyone opens this port by mistake.

This change will likely require a change in our firewall rules, might require a change in SELinux rules, and certainly requires a change in Director.

Comment 2 Michele Baldessari 2016-07-18 12:56:16 UTC

So in tripleo we set this stuff in puppet/hieradata/controller.yaml

rabbitmq_kernel_variables:
  inet_dist_listen_min: '35672'
  inet_dist_listen_max: '35672'


tripleo::firewall::firewall_rules:
...
  '109 rabbitmq':
    dport:       
      - 4369     
      - 5672     
      - 35672    
...

Peter, are there other settings somewhere (rpm, default config, etc.) that
might affect this port or is the above all there is?

Comment 3 Peter Lemenkov 2016-07-18 13:07:23 UTC

(In reply to Michele Baldessari from comment #2)
> So in tripleo we set this stuff in puppet/hieradata/controller.yaml
> 
> rabbitmq_kernel_variables:
>   inet_dist_listen_min: '35672'
>   inet_dist_listen_max: '35672'
> 
> 
> tripleo::firewall::firewall_rules:
> ...
>   '109 rabbitmq':
>     dport:       
>       - 4369     
>       - 5672     
>       - 35672    
> ...
> 
> Peter, are there other settings somewhere (rpm, default config, etc.) that
> might affect this port or is the above all there is?

Nope. I'm not aware of any other places containing this.

Comment 4 Peter Lemenkov 2016-07-18 14:12:40 UTC

Reducing priority/severity down since it's not very likely issue.

Comment 5 Fabio Massimo Di Nitto 2016-07-28 03:21:22 UTC

should we bump this to osp10?

Comment 6 Michele Baldessari 2016-07-28 06:29:18 UTC

If we can get traction for the selinux part of this bug then osp10 is doable,
otherwise it will osp11 I am afraid

Comment 7 Michele Baldessari 2016-09-15 08:11:40 UTC

As Marian mentioned per mail, this is hitting us harder than before due to the HA-NG architecture: services will start before rabbit and so the chances of the ephemeral port being taken are higher

Comment 12 errata-xmlrpc 2016-12-14 15:46:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html