Bug 1424823

Summary: ejabberd won't start with SELinux enforcing on F26+
Product: [Fedora] Fedora Reporter: Randy Barlow <randy>
Component: ejabberdAssignee: Randy Barlow <randy>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 26CC: erlang, jasons, jeremy, lemenkov, martin, randy, redhat
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: ejabberd-17.01-5.fc28 ejabberd-17.01-5.fc27 ejabberd-17.01-3.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-01 18:18:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1429126    
Bug Blocks:    
Attachments:
Description Flags
selinux.patch none

Description Randy Barlow 2017-02-19 16:57:43 UTC
Description of problem:
ejabberd won't start with SELinux enforcing on Rawhide.


Version-Release number of selected component (if applicable):
ejabberd-17.01-1.fc26.x86_64


How reproducible:
Every time.


Steps to Reproduce:
1. sudo systemctl start ejabberd
2. ps aux | grep ejabberd


Actual results:
$ ps aux | grep ejab
ejabberd 10600  0.0  0.0  37184   260 ?        S    11:52   0:00 /usr/lib64/erlang/erts-8.2.2/bin/epmd -daemon
rbarlow  10790  0.0  0.0 119424   968 pts/3    S+   11:52   0:00 grep --color=auto ejab


Expected results:
$ ps aux | grep ejab
ejabberd 11110  0.0  0.0  37184   264 ?        S    11:53   0:00 /usr/lib64/erlang/erts-8.2.2/bin/epmd -daemon
ejabberd 11112  156  1.1 3189700 56260 ?       Sl   11:53   0:03 /usr/lib64/erlang/erts-8.2.2/bin/beam.smp -K true -P250000 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/ejabberd -- -sname ejabberd@localhost -noshell -noinput -noshell -noinput -mnesia dir "/var/lib/ejabberd" -ejabberd log_rate_limit 100 log_rotate_size 10485760 log_rotate_count 1 log_rotate_date "" -s ejabberd -smp auto start
ejabberd 11130  0.0  0.0   4392   716 ?        Ss   11:53   0:00 erl_child_setup 16000
ejabberd 11176  0.0  0.0  11700  1060 ?        Ss   11:53   0:00 inet_gethost 4
ejabberd 11177  0.0  0.0  18040  1764 ?        S    11:53   0:00 inet_gethost 4
rbarlow  11179  0.0  0.0 119424   980 pts/3    S+   11:53   0:00 grep --color=auto ejab


Additional info:
This is a good opportunity to write an SELinux policy for ejabberd. If we do this, we can stop running it with /usr/bin/bash in the systemd unit.

Comment 1 Fedora End Of Life 2017-02-28 12:22:57 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 2 Randy Barlow 2017-03-04 19:14:59 UTC
# audit2allow -b


#============= init_t ==============
allow init_t epmd_port_t:tcp_socket name_connect;

#!!!! WARNING: 'etc_t' is a base type.
allow init_t etc_t:file write;
allow init_t jabber_interserver_port_t:tcp_socket name_connect;
allow init_t mtrr_device_t:file mounton;
allow init_t rabbitmq_exec_t:file ioctl;
allow init_t rabbitmq_var_lib_t:dir { add_name read remove_name write };
allow init_t rabbitmq_var_lib_t:file { create getattr open read rename unlink write };
allow init_t rabbitmq_var_log_t:dir { add_name read write };
allow init_t rabbitmq_var_log_t:file { append create getattr open read write };

Comment 3 Randy Barlow 2017-04-05 14:09:11 UTC
As noted in https://bugzilla.redhat.com/show_bug.cgi?id=1429126, I have written a new SELinux policy and submitted it to the fedora selinux-policy-contrib module:

https://github.com/fedora-selinux/selinux-policy-contrib/pull/8
https://github.com/fedora-selinux/selinux-policy-contrib/pull/7

Once that is accepted, merged, and released into Fedora 26+, we will also need to adjust a few things on the ejabberd side to be compliant.

For one, I wasn't able to get ejabberd working with policykit and SELinux enforcing, so I may drop the policy kit patch. It would fail with this error message:

ejabberdctl[22397]: Refusing to render service to dead parents.

Secondly, we no longer need to use /bin/bash to launch ejabberdctl in the unit file, and we also cannot use PrivateDevices=true because that will prevent the domain transition from being allowed.

Because we have to wait on the pull requests, I'm going to attach a git diff of what I have in my checkout right now here. This git diff isn't quite what we'll want, because it makes an ejabberd-selinux subpackage (which I used for testing purposes while developing the policy), but it has some of the changes we'll need.

Comment 4 Randy Barlow 2017-04-05 14:11:04 UTC
Created attachment 1268991 [details]
selinux.patch

Comment 5 Jonas Sell 2017-09-20 14:25:09 UTC
Is this still a thing or is this fixed? 
I'm still getting this in my audit.log when trying to start ejabberd:
type=AVC msg=audit(1505917132.313:368): avc:  denied  { read } for  pid=2567 comm="async_2" name="ejabberd" dev="sda3" ino=3147866 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:ejabberd_var_lib_t:s0 tclass=dir permissive=0                                                                                                                                                                                                                                          
type=AVC msg=audit(1505917132.315:369): avc:  denied  { write } for  pid=2567 comm="async_2" name="ejabberd" dev="sda3" ino=3147866 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:ejabberd_var_lib_t:s0 tclass=dir permissive=0                                                                                                                                                                                                                                         
type=AVC msg=audit(1505917132.417:370): avc:  denied  { read } for  pid=2567 comm="async_7" name=".erlang.cookie" dev="sda3" ino=3146320 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:admin_home_t:s0 tclass=file permissive=0                                                                                                                                                                                                                                         
type=AVC msg=audit(1505917132.419:371): avc:  denied  { read } for  pid=2567 comm="async_8" name=".erlang.cookie" dev="sda3" ino=3146320 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:admin_home_t:s0 tclass=file permissive=0                                                                                                                                                                                                                                         
type=AVC msg=audit(1505917132.935:372): avc:  denied  { write } for  pid=2567 comm="1_scheduler" name="ejabberd" dev="sda3" ino=4982432 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:ejabberd_var_log_t:s0 tclass=dir permissive=0 
Is this a problem of this bug or something else? Then I'll open another bug.

Comment 6 Randy Barlow 2017-09-20 15:16:19 UTC
Hi Jonas!

I don't know a way to know if my pull requests have made it into Fedora 26 yet, but I suppose we can try to do the remaining work that is needed now. A patch for the ejabberd package is needed so that it runs as ejabberd_t instead of as init_t. I assume you are on F26, F27, or F28? F25 didn't have any SELinux issues I am aware of, since init_t was more permissive in that release (unless that has changed - I run my ejabberd server on F26 so that is the one I'm most familiar with).

Comment 7 Jonas Sell 2017-09-20 15:19:40 UTC
Yes, it is F26. On F25 it worked here, too. How did you get it working on F26 without disabling SELinux?

Comment 8 Randy Barlow 2017-09-20 15:33:15 UTC
I've got my server manually patched with parts of the patch I attached[0] on this ticket, as well as a manually installed version of the selinux policies.

Assuming my selinux patches have been released to F26 (which again, I don't know how to verify other than just trying it?), there are two things you can do manually that I think will get you running:

0) Edit the service file (/usr/lib/systemd/system/ejabberd.service) to do what I did in the patch[0]. This will get ejabberd to run as ejabberd_t instead of init_t, since it won't launch as bash anymore.
1) Unapply the policykit patch to /usr/bin/ejabberdctl, which basically means changing the shebang line back to /usr/bin/bash. I wasn't able to get ejabberd working with polkit *and* SELinux. I am on the fence about the value of policykit for ejabberd anyway, when you can use sudo to grant access to users.

Hopefully I will find some non-dayjob time soon to do the above to the real package so we can be back in business. I also need to finish updating ejabberd to the newer release in rawhide...


[0] https://bugzilla.redhat.com/attachment.cgi?id=1268991&action=diff
[1] https://src.fedoraproject.org/rpms/ejabberd/blob/master/f/ejabberd-0002-Enable-polkit-support.patch

Comment 9 Jonas Sell 2017-09-20 15:44:56 UTC
It doesn't seem as if the patches are included - I tried the two steps you described and ejabberd still doesn't start here, messages in audit.log are the same. When I've time I might try to manually install SELinux policies.

Comment 10 Randy Barlow 2017-09-20 17:38:48 UTC
That's disappointing news. I actually did think of a good test you can try to be sure. If you have the policy I wrote, you should see something like this:

$ sudo semodule -l | grep ejabberd
ejabberd

If you don't see any output, we need to see if we can get the selinux-policy maintainers to release my patches for F26. I've also considered pulling those policies out of the selinux package and providing my own package, but the problem is that ejabberd gets matched by the rabbitmq policy unless my patch is in there so I can't do that on my own either.

Comment 11 Jonas Sell 2017-09-21 05:01:37 UTC
Strange. When issuing the command I get the correct output. Is it possible to see which rules are applied in the ejabbered policy? Or maybe I'm facing an other problem eg. wrong labeled filesystem. It was an upgrade from F25 where everything went fine.

Comment 12 Randy Barlow 2017-09-21 17:25:23 UTC
Jonas,

I forgot to tell you a step that would matter - after you edit the systemd unit file, you need to tell systemd to reload the daemon:

$ sudo systemctl daemon-reload

Alternatively, you can reboot.

If you do that, does it help?

Comment 13 Jonas Sell 2017-09-21 19:59:18 UTC
Yes, I did that. That was the first thing systemd suggested after editing the files. Unfortunately it didn't help.

Comment 14 Jonas Sell 2017-09-21 20:32:52 UTC
Now I got it running! The problem was the ejabberd_http module which is still blocked by SELinux (by preventing opening port 5281). This showed up in ejabberd's crash. log:
@ejabberd_listener:socket_error:580 Failed to open socket:
  {5281,ejabberd_http,[inet,{ip,{0,0,0,0}}]}
Reason: permission denied
After disabling ejabberd_http in ejabberd.yml ejabberd started again. Ok, maybe I can get a workaround for this tomorrow.

Comment 15 Randy Barlow 2017-09-21 21:07:28 UTC
Oh fantastic! Perhaps we just need the policy to allow port 5281.

Comment 16 Jonas Sell 2017-09-22 05:24:25 UTC
Yes, we need that. I added it manually with
semanage port -a -t jabber_client_port_t -p tcp 5281
After that ejabberd works with ejabberd_http! Not sure how to create a rule for selinux-policy though.

Comment 17 Randy Barlow 2017-09-23 17:13:49 UTC
Hi Jonas!

I'm going to use this ticket to track the work I need to do to get ejabberd running under the correct SELinux context. I've filed a new issue for you about port 5281, against the selinux-policy-targeted package:

https://bugzilla.redhat.com/show_bug.cgi?id=1494854

Comment 18 Fedora Update System 2017-09-23 17:19:39 UTC
ejabberd-17.01-3.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-55b5527f23

Comment 19 Fedora Update System 2017-09-23 17:19:45 UTC
ejabberd-17.01-5.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-213b09d56d

Comment 20 Jason Spangler 2017-09-23 23:50:52 UTC
Possibly related to this issue, even though I have SELinux disabled: ejabberd can't do PAM auth when PrivateDevices=true is present in ejabberd.service.   ejabberd runs epam as ejabberd user instead of root user when PrivateDevices=true is present, so epam can't access /etc/shadow, etc files to do auth.  Disabling PrivateDevices=true and making epam setuid root was the only workaround I found that worked.

Comment 21 Jason Spangler 2017-09-23 23:53:30 UTC
I should've included my versions in comment 20 :

[jasons@linus ~]$ cat /etc/fedora-release
Fedora release 26 (Twenty Six)
[jasons@linus ~]$ rpm -q ejabberd erlang-p1_pam
ejabberd-17.01-2.fc26.x86_64
erlang-p1_pam-1.0.0-3.fc26.x86_64

Comment 22 Jason Spangler 2017-09-23 23:55:37 UTC
Related to comment 20 , I tried the Rawhide fc27 packages and had the same result:
ejabberd-17.01-4.fc27.x86_64.rpm
erlang-p1_pam-1.0.3-3.fc27.x86_64.rpm

Comment 23 Fedora Update System 2017-09-24 05:52:19 UTC
ejabberd-17.01-3.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-55b5527f23

Comment 24 Randy Barlow 2017-09-24 16:45:18 UTC
Hi Jason,

As part of the change I made to get ejabberd to run confined, I also found it necessary to remove PrivateDevices=true. Please try out the new versions linked by the Fedora Update System here and report back karma!

Comment 25 Fedora Update System 2017-09-24 20:53:23 UTC
ejabberd-17.01-5.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-213b09d56d

Comment 26 Fedora Update System 2017-10-01 18:18:06 UTC
ejabberd-17.01-5.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 27 Fedora Update System 2017-10-02 16:21:29 UTC
ejabberd-17.01-3.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.