This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1032595 - SELinux policy prevents rabbitmq-server from starting up
SELinux policy prevents rabbitmq-server from starting up
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: selinux-policy (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Miroslav Grepl
Fedora Extras Quality Assurance
:
Depends On: 998682
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-20 07:52 EST by Richard W.M. Jones
Modified: 2015-02-18 06:12 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-02-18 06:00:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
recursive strace from starting up rabbit-mq server with SELinux Enforcing (1.59 MB, text/plain)
2013-11-25 10:15 EST, Richard W.M. Jones
no flags Details
recursive strace from starting up rabbit-mq server with SELinux Enforcing (updated, xz-compressed) (100.23 KB, application/x-xz)
2013-11-25 10:27 EST, Richard W.M. Jones
no flags Details

  None (edit)
Description Richard W.M. Jones 2013-11-20 07:52:48 EST
Description of problem:

When starting RabbitMQ (an AMQP broker) you have to set SELinux
to Permissive, otherwise it cannot bind to a port.

The AVC is:

time->Wed Nov 20 12:48:53 2013
type=SYSCALL msg=audit(1384951733.587:103679): arch=c000003e syscall=49 success=yes exit=0 a0=10 a1=7f26c833cab0 a2=1c a3=a items=0 ppid=1 pid=28654 auid=4294967295 uid=991 gid=989 euid=991 suid=991 fsuid=991 egid=989 sgid=989 fsgid=989 ses=4294967295 tty=(none) comm="beam.smp" exe="/usr/lib64/erlang/erts-5.10.3/bin/beam.smp" subj=system_u:system_r:rabbitmq_beam_t:s0 key=(null)
type=AVC msg=audit(1384951733.587:103679): avc:  denied  { name_bind } for  pid=28654 comm="beam.smp" src=10000 scontext=system_u:system_r:rabbitmq_beam_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket

audit2allow says:

#============= rabbitmq_beam_t ==============

#!!!! This avc can be allowed using the boolean 'nis_enabled'
allow rabbitmq_beam_t unreserved_port_t:tcp_socket name_bind;

I don't believe the comment about the boolean is relevant
here, since this machine does not use/need NIS.

Version-Release number of selected component (if applicable):

selinux-policy-3.12.1-74.11.fc19.noarch
rabbitmq-server-3.1.5-1.fc19.noarch
erlang-erts-R16B-02.3.fc19.x86_64

How reproducible:

100%

Steps to Reproduce:
1. Enable SELinux.
2. systemctl start rabbitmq-server
3.

Actual results:

Fails to start.
Comment 1 Richard W.M. Jones 2013-11-20 07:59:05 EST
RabbitMQ listens on port 5672.  This port is standardized for AMQP
(unlike AMQP itself which is a total non-standard).

It also uses a frankly confusing set of other ports for Erlang,
apparently randomly assigned using some sort of Erlang portmapper.
NFSv3 called, wants its stupid design back.

No idea what NIS has to do with any of this.
Comment 2 Richard W.M. Jones 2013-11-20 08:16:54 EST
OK so what happens is this:

Erlang's "kernel" has some port settings:

inet_dist_listen_min
inet_dist_listen_max

These can be set in the RabbitMQ configuration file.  As far as
I can tell there is no published default for these settings.

When an Erlang process starts up it chooses a port from this
range (I think starting with the lowest).  It also registers its
use of this port with a process called epmd -- the Erlang Port Mapper
Daemon.  You can also query epmd's current port mapping:

# epmd -names
epmd: up and running on port 4369 with data:
name rabbit at port 56590

As noted there, epmd uses its own well-known port (4369).
Comment 3 Miroslav Grepl 2013-11-25 09:19:05 EST
I see from AVC msgs, it wants to use tcp/10000. So it can use random ports from the defined range, right?
Comment 4 Richard W.M. Jones 2013-11-25 09:37:20 EST
It's very unclear to me what's happening, and not documented
*at all*.

However I spent some time last week looking at the source
of the Erlang kernel:

https://github.com/erlang/otp/blob/maint/lib/kernel/src/inet_tcp_dist.erl

As you can see from the do_listen/1 function, if inet_dist_list_{min,max}
are not set and there are no options that override them, then
it chooses {0, 0}.

That then causes the following loop to count up from 0 looking for
a port that is not in use:

do_listen(First,Last,Options) ->
    case inet_tcp:listen(First, Options) of
        {error, eaddrinuse} ->
         do_listen(First+1,Last,Options);
        Other ->
         Other
    end.

The code gets very complex from then on, but if it's really binding
with "port 0" then obviously the Linux kernel is going to choose a
random port from the range in /proc/sys/net/ipv4/ip_local_port_range.
(Port 32768, not port 10000 - no idea where that's coming from).

TBH I really need to strace RabbitMQ to find out what's really
happening.  Unfortunately I just nuked that machine, but I'm going
to reinstall it shortly.
Comment 5 Richard W.M. Jones 2013-11-25 10:15:12 EST
Created attachment 828729 [details]
recursive strace from starting up rabbit-mq server with SELinux Enforcing
Comment 6 Richard W.M. Jones 2013-11-25 10:27:55 EST
Created attachment 828734 [details]
recursive strace from starting up rabbit-mq server with SELinux Enforcing (updated, xz-compressed)

The previous log was truncated in some way.  This is a new
log hopefully showing the complete start up sequence (also with
SELinux Enforcing, so it is not successful).  I have xz-compressed
it because the original size is 27 MB.
Comment 7 Richard W.M. Jones 2013-11-25 10:31:50 EST
Suspecting this is really RabbitMQ being broken, or at least
"under-specified", I'm making this dependent on bug 998682.
Comment 8 John Eckersberg 2014-01-20 15:38:18 EST
Here's where the port 10000 thing comes in:

https://github.com/rabbitmq/rabbitmq-server/blob/b47325e38b578b8e629baaeb2ab04040b273634f/src/rabbit_networking.erl#L42

Which is used later on while trying to probe for whether ipv6 is available and in single or dual stack:

https://github.com/rabbitmq/rabbitmq-server/blob/b47325e38b578b8e629baaeb2ab04040b273634f/src/rabbit_networking.erl#L399

Ultimately ipv6_status hits the case of "port in use" (which is wrong -- instead it gets EACCESS due to selinux denial) here :

https://github.com/rabbitmq/rabbitmq-server/blob/b47325e38b578b8e629baaeb2ab04040b273634f/src/rabbit_networking.erl#L477

and recursively calls itself with the next port.

In my case on Fedora 20, it tries every port in succession until it gets to 32768 where it finally succeeds.  This is where ephemeral_port_t begins and rabbitmq is allowed to name_bind there:

$ sesearch -A -s rabbitmq_beam_t -p name_bind | grep rabbitmq.*ephemeral.*tcp
   allow rabbitmq_beam_t ephemeral_port_t : tcp_socket { name_bind name_connect } ;

$ sepolicy network -t ephemeral_port_t | grep tcp
ephemeral_port_t: tcp: 32768-61000


The end result is a ton of AVC denials and an extremely slow boot as it recursivlely calls ipv6_status to a depth of 22768 frames.  But in the end, at least on my system, it does eventually start:

$ telnet localhost amqp
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
testing
AMQP    Connection closed by foreign host.
Comment 9 Lukas Vrabec 2014-09-11 11:27:43 EDT
Hi, 
I fixed this in selinux-policy-3.12.1-185.fc20

https://github.com/selinux-policy/selinux-policy/commit/0901e1c899feb997a224d87190209fc50b66e75a
Comment 10 Lukas Vrabec 2014-09-11 11:28:27 EDT
Sorry this is F19 bug, this will be also backported.
Comment 11 Fedora End Of Life 2015-01-09 17:22:15 EST
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 12 Fedora End Of Life 2015-02-18 06:00:16 EST
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 13 Roman C 2015-02-18 06:12:06 EST
I did experience this on F20.

Note You need to log in before you can comment on or make changes to this bug.