+++ This bug was initially created as a clone of Bug #1650046 +++

Description of problem:
logrotate failed to do su for rabbitmqctl command at log rotation.

The system has following mail.
~~~
/etc/cron.hourly/logrotate:

Password: su: Authentication failure
error: error running shared postrotate script for '/var/log/rabbitmq/*.log '

You have mail in /var/spool/mail/root
~~~

Version-Release number of selected component (if applicable):
selinux-policy-3.13.1-229.el7_6.5.noarch

How reproducible:
Every time at logrotation

Steps to Reproduce:
1. Install Undercloud.
2. Run logrotate via cron for rabbitmq's logs.
3.

Actual results:
logrotate failed to execute rabbitmqctl command by failure of su.

Expected results:
logrotate successfully rotate rabbitmq's logs.

Additional info:
Related audit logs are
~~~
type=USER_AVC msg=audit(1542272461.277:11408): pid=22074 uid=0 auid=0 ses=75 subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 msg='avc:  denied  { passwd } for  scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tclass=passwd  exe="/usr/bin/su" sauid=0 hostname=? addr=? terminal=?'
~~~

--- Additional comment from Julie Pichon on 2019-10-29 17:35:35 UTC ---

Apologies for the delay, I didn't get an email notification and missed the update on the 21st. Thank you for explaining again. I looked at the system provided.

There are 3 denials in the logs.

type=AVC msg=audit(1571388842.268:1080): avc:  denied  { read } for  pid=9736 comm="inet_gethost" name="unix" dev="proc" ino=4026532003 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=AVC msg=audit(1571388845.317:1081): avc:  denied  { read } for  pid=9917 comm="inet_gethost" name="unix" dev="proc" ino=4026532003 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=USER_AVC msg=audit(1571557386.273:12713): pid=17720 uid=0 auid=0 ses=337 subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 msg='avc:  denied  { passwd } for  scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tclass=passwd  exe="/usr/bin/su" sauid=0 hostname=? addr=? terminal=?'

The first two are related to running the command "inet_gethost 4" and appear unrelated to the current issue so let's ignore them for now.

The logrotate issue *should* already be resolved with the current set of rules:

# grep denied /var/log/audit/audit.log | grep logrotate | audit2allow


#============= logrotate_t ==============

#!!!! This avc is allowed in the current policy
allow logrotate_t self:passwd passwd;


# rpm -qa | grep selinux | sort
container-selinux-2.107-1.el7_6.noarch
libselinux-2.5-14.1.el7.x86_64
libselinux-python-2.5-14.1.el7.x86_64
libselinux-ruby-2.5-14.1.el7.x86_64
libselinux-utils-2.5-14.1.el7.x86_64
openstack-selinux-0.8.18-1.el7ost.noarch
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
pcp-selinux-4.1.0-4.el7.x86_64
selinux-policy-3.13.1-229.el7.noarch
selinux-policy-targeted-3.13.1-229.el7.noarch

Parsing again the audit log, the logrotate issue seems related to some PAM error:

type=USER_AVC msg=audit(10/20/2019 03:43:06.273:12713) : pid=17720 uid=root auid=root ses=337 subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 msg='avc:  denied  { passwd } for  scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tclass=passwd  exe=/usr/bin/su sauid=root hostname=? addr=? terminal=?' 
----
type=USER_AUTH msg=audit(10/20/2019 03:43:06.279:12714) : pid=17720 uid=root auid=root ses=337 subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct=rabbitmq exe=/usr/bin/su hostname=? addr=? terminal=? res=failed' 

Note the 'res=failed' at the end here. I also found other reports of rabbitmq log rotation failing *despite* a rule explicitly allowing the AVC being already present, e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1426600#c1 . We see the same error mentioned in the mail for root:

-----
Subject: Anacron job 'cron.daily' on undercloud-0.redhat.local
Date: Sun, 20 Oct 2019 03:43:09 -0400 (EDT)
Status: R

/etc/cron.daily/logrotate:

Password: su: Authentication failure
error: error running shared postrotate script for '/var/log/rabbitmq/*.log '
-----

Also in /var/log/secure-20191020:

Oct 20 03:43:06 undercloud-0 su: pam_unix(su:auth): auth could not identify password for [rabbitmq]
Oct 20 03:43:06 undercloud-0 su: pam_succeed_if(su:auth): requirement "uid >= 1000" not met by user "rabbitmq"

And /var/log/messages-20191020:

Oct 20 03:43:07 undercloud-0 su: FAILED SU (to rabbitmq) root on none

It seems related to this issue: https://github.com/rabbitmq/rabbitmq-server-release/issues/39 . Apparently some commands work correctly when called with sudo but not with su. There are fixes available in rabbitmq-server for this (see https://github.com/rabbitmq/rabbitmq-server-release/pull/40 ), however looking at the tags on the commit they are only present from rabbitmq 3.7 onwards and in the OSP13 system provided we use an earlier version:

# rpm -q rabbitmq-server
rabbitmq-server-3.6.15-4.el7ost.noarch

The patch involves calling commands with runuser instead of su, which seems to play better with SELinux. I am not sure how easy the patches would be to backport, especially since it seems follow-ups were necessary (e.g. https://github.com/rabbitmq/rabbitmq-server-release/pull/45 ).

--- Additional comment from Julie Pichon on 2019-10-29 17:50:39 UTC ---

Hi Petr,

I am wondering if you would have thoughts on comment 44 above (there is also a test system available, see comment 42). This seems to be a follow-up to the issue you helped with resolving in bug 1413775. The rule added to allow this AVC is already present in openstack-selinux, and we can see in the tests for your patch (L10 - https://github.com/redhat-openstack/openstack-selinux/pull/17/files#diff-5103669ece6ce89d45c68eb4a411fee4R10 ) that the very same denial should be resolved, however that doesn't seem to be sufficient due to hitting https://github.com/rabbitmq/rabbitmq-server-release/issues/39 which is only resolved with RabbitMQ 3.7, not available to OSP13 as far as I understand. As you helped with the previous SELinux bug, and I see you have patches on the downstream RabbitMQ package as well I am wondering if you would have any input on this. Thank you!

--- Additional comment from Peter Lemenkov on 2019-11-19 07:39:13 UTC ---

(In reply to Julie Pichon from comment #44)

> It seems related to this issue:
> https://github.com/rabbitmq/rabbitmq-server-release/issues/39 . Apparently
> some commands work correctly when called with sudo but not with su. There
> are fixes available in rabbitmq-server for this (see
> https://github.com/rabbitmq/rabbitmq-server-release/pull/40 ), however
> looking at the tags on the commit they are only present from rabbitmq 3.7
> onwards and in the OSP13 system provided we use an earlier version:
> 
> # rpm -q rabbitmq-server
> rabbitmq-server-3.6.15-4.el7ost.noarch
> 
> The patch involves calling commands with runuser instead of su, which seems
> to play better with SELinux. I am not sure how easy the patches would be to
> backport, especially since it seems follow-ups were necessary (e.g.
> https://github.com/rabbitmq/rabbitmq-server-release/pull/45 ).

Julie, thanks for the digging this up. I'll backport these patches soon.

--- Additional comment from Keigo Noha on 2019-11-25 08:41:05 UTC ---

Hi Peter,

If we have a test build, I'll check the behavior on my machine.
Would you be able to share the test package for this issue?

Thanks,
Keigo

--- Additional comment from Peter Lemenkov on 2020-01-28 16:21:19 UTC ---

Please try rabbitmq-server-3.6.15-6.el7ost

--- Additional comment from Keigo Noha on 2020-01-29 06:09:53 UTC ---

Hi Peter,

Thank you for your update!
I installed rabbitmq-server-3.6.15-6.el7ost on my lab environment.
I leave the undercloud for several days and confirm the fix.

If I get a result with the package, I'll update this bugzilla.

Thank you,
Keigo Noha