Bug 1650046

The issue is already reported as https://bugzilla.redhat.com/show_bug.cgi?id=1450789.
But the issue still occurs.

Interesting is that the required allow rule is already present in policy:

# rpm -qa selinux\*
selinux-policy-minimum-3.13.1-229.el7_6.5.noarch
selinux-policy-3.13.1-229.el7_6.5.noarch
selinux-policy-doc-3.13.1-229.el7_6.5.noarch
selinux-policy-targeted-3.13.1-229.el7_6.5.noarch
selinux-policy-devel-3.13.1-229.el7_6.5.noarch
selinux-policy-mls-3.13.1-229.el7_6.5.noarch
selinux-policy-sandbox-3.13.1-229.el7_6.5.noarch
# sesearch -s logrotate_t -t logrotate_t -c passwd -A --dontaudit -C
Found 1 semantic av rules:
   allow logrotate_t logrotate_t : passwd passwd ; 

# 

Unfortunately, I don't know why the USER_AVC appears.

Hello Lukas,

Do you have any updates on this bugzilla?
This issue affects to all users who use rabbitmq and it causes the lost of rabbitmq's log. If they hit some issues and support side can't see the logs by this issue.

Best Regards,
Keigo Noha

This issue was not selected to be included in Red Hat Enterprise Linux 7 because it is seen either as low or moderate impact to a small number of use-cases. The next minor release will be in Maintenance Support 1 Phase, which means that qualified Critical and Important Security errata advisories (RHSAs) and Urgent Priority Bug Fix errata advisories (RHBAs) may be released as they become available.

We will now close this issue, but if you believe that it qualifies for the Maintenance Support 1 Phase, please re-open; otherwise, we recommend moving the request to Red Hat Enterprise Linux 8 if applicable.


For a workaround, please refer to previous comments or follow these steps to create a custom policy using the CIL language:

# cat > local_rabbitmq.cil << EOF
(allow logrotate_t logrotate_t (capability (net_admin)))
(allow logrotate_t logrotate_t (passwd (passwd)))
(allow logrotate_t logrotate_t (tcp_socket (accept listen )))
(allow logrotate_t epmd_port_t (tcp_socket (name_bind name_connect)))
(allow logrotate_t rabbitmq_port_t (tcp_socket (name_connect)))
(allow logrotate_t rabbitmq_var_lib_t (dir (read write)))
(allow logrotate_t rabbitmq_var_lib_t (file (getattr open read)))
(allow logrotate_t security_t (file (write)))
(allow logrotate_t security_t (security (compute_av)))
(allow logrotate_t systemd_logind_sessions_t (fifo_file (write)))
EOF
  # semodule -i local_rabbitmq.cil

Hi Zdenek,

I don't think this is a small number of use-cases. This issue has an impact to all RHOSP customers on RHEL7.
And if we cannot fix it in RHEL side, we need to consider to add the rules in RHOSP side while undercloud installation.
And this issue also has an impact to all rabbitmq installed systems too.
log rotation of a component is a basic feature.   

Could you please re-consider the priority and severity of this issue and fix it in the next RHEL minor release?

Kind Regards,
Keigo Noha

Hi Keigo,

The SELinux team decided not to include the additional permissions into the selinux-policy package shipped with RHEL 7 as the permissions have not been test upstream, therefore any patch like this would pose a risk. RHEL 7 is in transition from Full Support Phase to Maintenance one so there are different criteria for inclusion; in general, stability is the key.

Nevertheless, there is a workaround available, both in te and cil formats. If you need additional support for OSP, the permissions can be included in this product. Our team can help you with connecting.

Hi Zdenek,

Thank you for your update. I understand SELinux team's decision and the background. Would it be possible to include the workaround into openstack-selinux package?
Would it be possible to ask openstack-selinux maintainer to include this into the package?

Kind Regards,
Keigo Noha

Switching product and component.

openstack-selinux maintainers, could you consider adding the permissions listed in #10 or #25 to the package?

Hi. Could you confirm the openstack-selinux version on the system? And also attach a permissive audit.log file that shows all the denials? Thank you.

It would seem the first 4 AVCs and the self:tcp_socket one already are allowed after installing openstack-selinux, while the other 5 appear to be handled in some way in the regular SELinux policy already... This is strange.

#============= logrotate_t ==============

#!!!! This avc is allowed in the current policy
allow logrotate_t epmd_port_t:tcp_socket { name_bind name_connect };

#!!!! This avc is allowed in the current policy
allow logrotate_t rabbitmq_port_t:tcp_socket name_connect;

#!!!! This avc is allowed in the current policy
allow logrotate_t rabbitmq_var_lib_t:dir { read write };

#!!!! This avc is allowed in the current policy
allow logrotate_t rabbitmq_var_lib_t:file { getattr open read };

#!!!! This avc has a dontaudit rule in the current policy
allow logrotate_t security_t:file write;

#!!!! This avc is allowed in the current policy
allow logrotate_t security_t:security compute_av;

#!!!! This avc has a dontaudit rule in the current policy
allow logrotate_t self:capability net_admin;

#!!!! This avc is allowed in the current policy
allow logrotate_t self:passwd passwd;

#!!!! This avc is allowed in the current policy
allow logrotate_t self:tcp_socket { accept listen };

#!!!! This avc has a dontaudit rule in the current policy
allow logrotate_t systemd_logind_sessions_t:fifo_file write;


On RHEL 7.7:

$ rpm -qa | grep selinux | sort
container-selinux-2.107-3.el7.noarch
libselinux-2.5-14.1.el7.x86_64
libselinux-python-2.5-14.1.el7.x86_64
libselinux-utils-2.5-14.1.el7.x86_64
openstack-selinux-0.8.18-1.el7ost.noarch
selinux-policy-3.13.1-251.el7.noarch
selinux-policy-targeted-3.13.1-251.el7.noarch

Also, what is the result when running $ getsebool nis_enabled ? Thank you.

Hi Julie,

I've confirmed the behaviour with RHOSP13z7.

~~~
# less /var/log/messages |grep logrotate
Sep  1 03:10:07 undercloud-0 logrotate: ALERT exited abnormally with [1]
~~~

~~~
# aureport -a -if /var/log/audit/audit.log 

AVC Report
===============================================================
# date time comm subj syscall class permission obj result event
===============================================================
1. 08/29/2019 05:02:51 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 364
2. 08/29/2019 05:03:01 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 365
3. 08/29/2019 05:03:16 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 376
4. 08/29/2019 05:03:18 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 377
5. 08/29/2019 05:03:22 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 412
6. 08/29/2019 05:03:22 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 413
7. 08/29/2019 05:03:22 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 414
8. 08/29/2019 05:03:22 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 415
9. 08/29/2019 05:06:08 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 891
10. 08/29/2019 05:06:09 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 901
11. 08/29/2019 05:06:37 inet_gethost system_u:system_r:rabbitmq_t:s0 21 file read system_u:object_r:proc_net_t:s0 denied 960
12. 08/29/2019 05:06:40 inet_gethost system_u:system_r:rabbitmq_t:s0 21 file read system_u:object_r:proc_net_t:s0 denied 961
13. 09/01/2019 03:10:04 ? system_u:system_r:logrotate_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 10866 <-- HERE
~~~

The package list is below.
~~~
# rpm -qa |grep selinux | sort
container-selinux-2.107-1.el7_6.noarch
libselinux-2.5-14.1.el7.x86_64
libselinux-python-2.5-14.1.el7.x86_64
libselinux-ruby-2.5-14.1.el7.x86_64
libselinux-utils-2.5-14.1.el7.x86_64
openstack-selinux-0.8.18-1.el7ost.noarch
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
selinux-policy-3.13.1-229.el7.noarch
selinux-policy-targeted-3.13.1-229.el7.noarch
~~~

I'll turn SELinux mode to permissive and get the whole error.

Kind Regards,
Keigo Noha

Hi Keigo,

Any luck with retrieving the permissive audit logs? I looked at the suggested rules again, but many already seem included in openstack-selinux (cf. https://github.com/redhat-openstack/openstack-selinux/pull/17/files for bug 1413775) and should be in the version installed.

Created attachment 1614018 [details]
audit.log under permissive mode

Hi Julie,

~~~
# aureport -a -if /var/log/audit/audit.log 

AVC Report
===============================================================
# date time comm subj syscall class permission obj result event
===============================================================
1. 09/04/2019 22:30:00 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 359
2. 09/04/2019 22:30:10 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 360
3. 09/04/2019 22:30:25 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 371
4. 09/04/2019 22:30:28 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 372
5. 09/04/2019 22:30:30 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 395
6. 09/04/2019 22:30:30 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 396
7. 09/04/2019 22:30:30 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 397
8. 09/04/2019 22:30:30 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 398
9. 09/04/2019 22:33:22 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 883
10. 09/04/2019 22:33:23 ? system_u:system_r:init_t:s0 0 (null) (null) (null) unset 893
11. 09/04/2019 22:33:51 inet_gethost system_u:system_r:rabbitmq_t:s0 21 file read system_u:object_r:proc_net_t:s0 denied 952
12. 09/04/2019 22:33:53 inet_gethost system_u:system_r:rabbitmq_t:s0 21 file read system_u:object_r:proc_net_t:s0 denied 953
13. 09/08/2019 03:09:08 ? system_u:system_r:logrotate_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 11723
~~~

The log is taken under permissive mode. As far as I saw, the issue always happen on undercloud.

Many thanks for attaching the logs. There is only one logrotate_t denial remaining in the logs attached and it should already be covered by openstack-selinux. The logs also indicate that the system was still enforcing when the denial showed up, so perhaps it's an old one.

Would it be possible to:

1. Run $ rpm -qa |grep selinux | sort
2. Set the system to permissive (that can be confirmed with running $ getenforce)
3. Set up the cron to reproduce the problem
4. Once the cron has run, grab the audit.log and attach it to this bug,

Thank you.

Hi Julie,

I sent an e-mail to you about the reproducing environment. The environment is now back and you can use it. Would you check the behavior on the environment?

Kind Regards,
Keigo Noha

Thank you for providing an environment. Unfortunately I did not manage to reproduce the error. Could you check if my steps are correct?

(In reply to Julie Pichon from comment #37)
> 1. Run $ rpm -qa |grep selinux | sort

libselinux-2.5-14.1.el7.x86_64
libselinux-python-2.5-14.1.el7.x86_64
libselinux-ruby-2.5-14.1.el7.x86_64
libselinux-utils-2.5-14.1.el7.x86_64
openstack-selinux-0.8.18-1.el7ost.noarch
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
selinux-policy-3.13.1-229.el7.noarch
selinux-policy-targeted-3.13.1-229.el7.noarch

> 2. Set the system to permissive (that can be confirmed with running $
> getenforce)

# getenforce
Enforcing
# setenforce 0
# getenforce
Permissive

> 3. Set up the cron to reproduce the problem

As suggested in another comment, I moved the logrotate daily cron to be hourly so that it runs more often.

> 4. Once the cron has run, grab the audit.log and attach it to this bug,

I tailed the audit.log while the hourly cron was running, but no AVC denials showed up.

Being unable to reproduce as described in the previous comment, and noting that most of the rules requested in comment 15 are already present in openstack-selinux ( https://github.com/redhat-openstack/openstack-selinux/blob/master/os-rabbitmq.te ) I am closing this bug. If this is incorrect, please feel free to reopen with the information requested in comment 37. Thank you!

Hi Julie,

Please don't close this bugzilla.
The issue is easily reproduced.

Let me write the reproducer again.
1. Create Undercloud.
2. Wait for 1 day.
3. After logrotate works, check the logrotate file and opened file by rabbitmq and related programs.

~~~
beam.smp   9631               rabbitmq   47w      REG              253,1    118597  155195899 /var/log/rabbitmq/rabbit
sys_sig_d  9631  9770         rabbitmq   47w      REG              253,1    118597  155195899 /var/log/rabbitmq/rabbit
sys_msg_d  9631  9771         rabbitmq   47w      REG              253,1    118597  155195899 /var/log/rabbitmq/rabbit
async_1    9631  9772         rabbitmq   47w      REG              253,1    118597  155195899 /var/log/rabbitmq/rabbit
~~~

By the lsof command result, rabbitmq related programs refer to the rotated file.
The expected file after the logroate is /var/log/rabbitmq/rabbit.

The issue can be found RHOSP13z8.

Apologies for the delay, I didn't get an email notification and missed the update on the 21st. Thank you for explaining again. I looked at the system provided.

There are 3 denials in the logs.

type=AVC msg=audit(1571388842.268:1080): avc:  denied  { read } for  pid=9736 comm="inet_gethost" name="unix" dev="proc" ino=4026532003 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=AVC msg=audit(1571388845.317:1081): avc:  denied  { read } for  pid=9917 comm="inet_gethost" name="unix" dev="proc" ino=4026532003 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=USER_AVC msg=audit(1571557386.273:12713): pid=17720 uid=0 auid=0 ses=337 subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 msg='avc:  denied  { passwd } for  scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tclass=passwd  exe="/usr/bin/su" sauid=0 hostname=? addr=? terminal=?'

The first two are related to running the command "inet_gethost 4" and appear unrelated to the current issue so let's ignore them for now.

The logrotate issue *should* already be resolved with the current set of rules:

# grep denied /var/log/audit/audit.log | grep logrotate | audit2allow


#============= logrotate_t ==============

#!!!! This avc is allowed in the current policy
allow logrotate_t self:passwd passwd;


# rpm -qa | grep selinux | sort
container-selinux-2.107-1.el7_6.noarch
libselinux-2.5-14.1.el7.x86_64
libselinux-python-2.5-14.1.el7.x86_64
libselinux-ruby-2.5-14.1.el7.x86_64
libselinux-utils-2.5-14.1.el7.x86_64
openstack-selinux-0.8.18-1.el7ost.noarch
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
pcp-selinux-4.1.0-4.el7.x86_64
selinux-policy-3.13.1-229.el7.noarch
selinux-policy-targeted-3.13.1-229.el7.noarch

Parsing again the audit log, the logrotate issue seems related to some PAM error:

type=USER_AVC msg=audit(10/20/2019 03:43:06.273:12713) : pid=17720 uid=root auid=root ses=337 subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 msg='avc:  denied  { passwd } for  scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tclass=passwd  exe=/usr/bin/su sauid=root hostname=? addr=? terminal=?' 
----
type=USER_AUTH msg=audit(10/20/2019 03:43:06.279:12714) : pid=17720 uid=root auid=root ses=337 subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct=rabbitmq exe=/usr/bin/su hostname=? addr=? terminal=? res=failed' 

Note the 'res=failed' at the end here. I also found other reports of rabbitmq log rotation failing *despite* a rule explicitly allowing the AVC being already present, e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1426600#c1 . We see the same error mentioned in the mail for root:

-----
Subject: Anacron job 'cron.daily' on undercloud-0.redhat.local
Date: Sun, 20 Oct 2019 03:43:09 -0400 (EDT)
Status: R

/etc/cron.daily/logrotate:

Password: su: Authentication failure
error: error running shared postrotate script for '/var/log/rabbitmq/*.log '
-----

Also in /var/log/secure-20191020:

Oct 20 03:43:06 undercloud-0 su: pam_unix(su:auth): auth could not identify password for [rabbitmq]
Oct 20 03:43:06 undercloud-0 su: pam_succeed_if(su:auth): requirement "uid >= 1000" not met by user "rabbitmq"

And /var/log/messages-20191020:

Oct 20 03:43:07 undercloud-0 su: FAILED SU (to rabbitmq) root on none

It seems related to this issue: https://github.com/rabbitmq/rabbitmq-server-release/issues/39 . Apparently some commands work correctly when called with sudo but not with su. There are fixes available in rabbitmq-server for this (see https://github.com/rabbitmq/rabbitmq-server-release/pull/40 ), however looking at the tags on the commit they are only present from rabbitmq 3.7 onwards and in the OSP13 system provided we use an earlier version:

# rpm -q rabbitmq-server
rabbitmq-server-3.6.15-4.el7ost.noarch

The patch involves calling commands with runuser instead of su, which seems to play better with SELinux. I am not sure how easy the patches would be to backport, especially since it seems follow-ups were necessary (e.g. https://github.com/rabbitmq/rabbitmq-server-release/pull/45 ).

Hi Petr,

I am wondering if you would have thoughts on comment 44 above (there is also a test system available, see comment 42). This seems to be a follow-up to the issue you helped with resolving in bug 1413775. The rule added to allow this AVC is already present in openstack-selinux, and we can see in the tests for your patch (L10 - https://github.com/redhat-openstack/openstack-selinux/pull/17/files#diff-5103669ece6ce89d45c68eb4a411fee4R10 ) that the very same denial should be resolved, however that doesn't seem to be sufficient due to hitting https://github.com/rabbitmq/rabbitmq-server-release/issues/39 which is only resolved with RabbitMQ 3.7, not available to OSP13 as far as I understand. As you helped with the previous SELinux bug, and I see you have patches on the downstream RabbitMQ package as well I am wondering if you would have any input on this. Thank you!

Hello,

Please try rabbitmq-server-3.6.15-5.el7ost version. It should fix this issue (considering that openstack-selinux-0.8.18 is installed).

Please try rabbitmq-server-3.6.15-6.el7ost

Hi Peter,

Thank you for your update!
I installed rabbitmq-server-3.6.15-6.el7ost on my lab environment.
I leave the undercloud for several days and confirm the fix.

If I get a result with the package, I'll update this bugzilla.

Thank you,
Keigo Noha

Would you be able to test the openstack-selinux-0.8.18-2.el7ost scratch build together with the rabbitmq-server-3.6.15-6.el7ost fix? Thank you.

Sanity checked:

$ rpm -q openstack-selinux
openstack-selinux-0.8.18-3.el7ost.noarch

$ cat /usr/share/openstack-selinux/0.8.18/tests/bz1650046 
type=AVC msg=audit(1583112189.890:19129): avc:  denied  { read } for  pid=13620 comm="inet_gethost" name="unix" dev="proc" ino=4026532003 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=1
type=AVC msg=audit(1583136605.180:19713): avc:  denied  { setrlimit } for  pid=20488 comm="runuser" scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tclass=process permissive=1
type=AVC msg=audit(1583136606.388:19715): avc:  denied  { read } for  pid=20640 comm="inet_gethost" name="unix" dev="proc" ino=4026532003 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=1

$ /usr/share/openstack-selinux/0.8.18/tests/check_all 
Results: 801 total, 0 failed
Overall result: PASS

Also confirmed that the following version of RabbitMQ Server is included in the puddle: rabbitmq-server-3.6.15-6.el7ost.noarch which the minimum version required for the dependent bug (bug 1811607).

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2719