Bug 1473349

Summary: Logrotate cannot access candlepin logs
Product: [Community] Candlepin Reporter: Kevin Howell <khowell>
Component: candlepinAssignee: candlepin-bugs
Status: CLOSED CURRENTRELEASE QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: urgent    
Version: 0.9.54CC: ajoseph, awood, bcourt, ben.argyle, candlepin-bugs, cduryee, cisley, ehelms, gapatil, hajek, hartsjc, hmore, hshukla, jbubeck, jdickers, johan.bergstrom, katello-qa-list, khowell, lpramuk, lzap, michiel.smit, mmccune, mstead, nathan.t.mcgarvey, peter.vreman, pghadge, ramsingh, redakkan, riehecky, robert.sprockeels, skallesh, tkarlsso, victor.andreasson, zpytela
Target Milestone: ---Keywords: Triaged
Target Release: 2.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: candlepin-0.9.54.25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1473348 Environment:
Last Closed: 2019-04-04 13:07:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1520357, 1520502, 1533259    

Comment 1 Alex Wood 2017-07-25 19:14:24 UTC
commit 9357aa4761c78b19a0ddd47816cba7fb92b4891d
Author: Alex Wood <awood>
Date:   Mon Jul 24 16:23:53 2017 -0400

    1473349: Fix SELinux error when logrotate runs on candlepin logs

Comment 2 Chris Duryee 2017-08-04 14:29:26 UTC
Is there an selinux boolean or file context we could set as a workaround?

Comment 3 Alex Wood 2017-08-04 15:01:21 UTC
You could change the file context back to var_log_t I think.

Comment 4 Jason Dickerson 2017-08-04 15:09:43 UTC
Alex can we not just make the change in commit 1655 to /usr/share/doc/candlepin-selinux-0.9.54.24/candlepin.te and then run /usr/share/doc/candlepin-selinux-0.9.54.24/candlepin.sh --update?  Would that work?

Comment 5 Alex Wood 2017-08-04 15:27:59 UTC
Yes, that would update the policy with the denials found in the audit log and then you could install the new policy module.

Comment 6 Jason Dickerson 2017-08-04 15:38:12 UTC
the candlepin.sh requires /usr/share/selinux/devel/Makefile which is not on a satellite. not sure I should install selinux-policy-devel on a satellite.

Comment 9 Peter Vreman 2017-08-17 12:15:55 UTC
I the KB it is written RHEL7.4, but it also applies to RHEL7.3 were we have seen the same issue.

Comment 10 Peter Vreman 2017-08-17 12:17:04 UTC
I think the RHEKL7.3 shall be added.
Because based on the current KB makes me as a customer think i if i stay on RHEL7.3 then i do not have the problem.

Comment 12 James Hartsock 2017-08-17 20:59:08 UTC
So I have no issue when I run this as root and force log rotation..
  # /usr/sbin/logrotate -f /etc/logrotate.d/candlepin

But the nightly /etc/cron.daily/logrotate generates the error, which results in e-mail to root user (which for many enterprise environments results in admin notification)

Also confirmed today that this happens on fresh OS & satellite 6.2 install...
# aureport -a
<snip>
20. 08/17/2017 14:21:14 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 841
21. 08/17/2017 14:21:14 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 842
22. 08/17/2017 14:21:14 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 843
23. 08/17/2017 14:21:14 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 844
24. 08/17/2017 14:21:14 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 845
25. 08/17/2017 14:21:14 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 846


What are exact steps we can provide customer's to correct this?

Comment 13 James Hartsock 2017-08-17 23:20:11 UTC
Able to reproduce it...

Hack logrotate so doesn't know did candlepin already
~~~
# cp -ap /var/lib/logrotate/logrotate.status /var/lib/logrotate/logrotate.status.save
# grep -v /var/log/candlepin/ /var/lib/logrotate/logrotate.status.save > /var/lib/logrotate/logrotate.status
~~~

Now remove cron.daily, so it runs next check
~~~
# rm /var/spool/anacron/cron.daily
~~~


Wait for anacron to run it for us at top of hour...
~~~
# tail -f /var/log/cron
Aug 17 17:01:01 sat6 anacron[22714]: Will run job `cron.daily' in 7 min.
Aug 17 17:08:17 sat6 run-parts(/etc/cron.daily)[23163]: starting logrotate
Aug 17 17:08:17 sat6 run-parts(/etc/cron.daily)[23380]: finished logrotate

# aureport -a
27. 08/17/2017 17:08:17 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 643
28. 08/17/2017 17:08:17 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 644
29. 08/17/2017 17:08:17 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 645
30. 08/17/2017 17:08:17 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 646
31. 08/17/2017 17:08:17 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 647
32. 08/17/2017 17:08:17 logrotate system_u:system_r:logrotate_t:s0-s0:c0.c1023 6 file getattr system_u:object_r:candlepin_var_log_t:s0 denied 648
~~~

Comment 19 Ben 2017-10-06 10:16:17 UTC
Additional (superfluous?) datapoint.

RHEL7.4 fully updated as of 2017-10-06
Satellite 6.2.12 fully updated as of same date

Seems I can't even fix this manually:

# grep logrotate /var/log/audit/audit.log | audit2why
[...]
type=AVC msg=audit(1507255941.668:4582): avc:  denied  { getattr } for  pid=64893 comm="logrotate" path="/var/log/candlepin/error.log" dev="dm-2" ino=402770458 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:candlepin_var_log_t:s0 tclass=file

        Was caused by:
                Missing type enforcement (TE) allow rule.

                You can use audit2allow to generate a loadable module to allow this access.
[...]
# audit2allow -a -M logrotate_t
# semodule -i logrotate_t.pp
# chcon -R -t logrotate_t /var/log/candlepin/*.log
chcon: failed to change context of '/var/log/candlepin/audit.log' to 'system_u:object_r:logrotate_t:s0': Permission denied
chcon: failed to change context of '/var/log/candlepin/candlepin.log' to 'system_u:object_r:logrotate_t:s0': Permission denied
chcon: failed to change context of '/var/log/candlepin/cpdb.log' to 'unconfined_u:object_r:logrotate_t:s0': Permission denied
chcon: failed to change context of '/var/log/candlepin/cpinit.log' to 'unconfined_u:object_r:logrotate_t:s0': Permission denied
chcon: failed to change context of '/var/log/candlepin/error.log' to 'system_u:object_r:logrotate_t:s0': Permission denied

Any chance this can make it into 6.2.12 as a hotfix?

Comment 23 Ben 2017-10-18 08:27:26 UTC
Additionally, I'm also getting this email every morning:

/etc/cron.daily/logrotate:

error: error renaming /var/log/candlepin/audit.log.52.gz to /var/log/candlepin/audit.log.53.gz: Permission denied
error: error renaming /var/log/candlepin/candlepin.log.52.gz to /var/log/candlepin/candlepin.log.53.gz: Permission denied
error: error renaming /var/log/candlepin/error.log.52.gz to /var/log/candlepin/error.log.53.gz: Permission denied

Comment 27 Mike McCune 2017-11-14 18:36:33 UTC
*** WORKAROUND until released in 6.2.Z ***

1) edit policy

~~~
# yum install selinux-policy-devel

# cd /usr/share/doc/candlepin-selinux-0.9.54.23/
# cp -p candlepin.pp candlepin.pp.orig #if not present, no worries
# sed -ie 's/^files_type\((candlepin_var_log_t)\)$/logging_log_file\1/' candlepin.te 
# grep '(candlepin_var_log_t)' candlepin.te
# make NAME=targeted -f /usr/share/selinux/devel/Makefile

# semodule -i candlepin.pp
~~~


2) verify resolution

~~~
# tail -n0 -f /var/log/cron
Aug 18 09:01:01 sat6 run-parts(/etc/cron.hourly)[9988]: starting 0anacron
Aug 18 09:01:01 sat6 anacron[9997]: Will run job `cron.daily' in 5 min.
Aug 18 09:06:13 sat6 run-parts(/etc/cron.daily)[10379]: starting logrotate
Aug 18 09:06:13 sat6 run-parts(/etc/cron.daily)[10529]: finished logrotate

# aureport -a -ts 08/18/2017 09:00:00
AVC Report
<no events of interest were found>
~~~

Comment 28 Ben 2017-11-15 09:48:33 UTC
Selected output from doing the above:

# cp -p candlepin.pp candlepin.pp.orig 
cp: cannot stat ‘candlepin.pp’: No such file or directory
(no worries)

# cp candlepin.te candlepin.te.orig
(because I'm paranoid)

# sed -ie 's/^files_type\((candlepin_var_log_t)\)$/logging_log_file\1/' candlepin.te
(no errors)

# diff candlepin.te candlepin.te.orig
30c30
< logging_log_file(candlepin_var_log_t)
---
> files_type(candlepin_var_log_t)

# grep '(candlepin_var_log_t)' candlepin.te
logging_log_file(candlepin_var_log_t)
(duh!)

# make NAME=targeted -f /usr/share/selinux/devel/Makefile
Compiling targeted candlepin module
/usr/bin/checkmodule:  loading policy configuration from tmp/candlepin.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 17) to tmp/candlepin.mod
Creating targeted candlepin.pp policy package
rm tmp/candlepin.mod.fc tmp/candlepin.mod

# semodule -i candlepin.pp
(no errors)


I'll let you know how it goes tomorrow morning when the logrotate runs again.  Thanks again!

Comment 29 Chris "Ceiu" Rog 2018-01-31 18:59:29 UTC
*** Bug 1520357 has been marked as a duplicate of this bug. ***

Comment 30 Chris "Ceiu" Rog 2018-02-01 14:08:36 UTC
*** Bug 1520499 has been marked as a duplicate of this bug. ***