Bug 1639381

Summary: Drop the selinux domain check or use system_cronjob_t domain for the system crontab files
Product: [Fedora] Fedora Reporter: Lukas Vrabec <lvrabec>
Component: cronieAssignee: Tomas Mraz <tmraz>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 29CC: awilliam, bcotton, bojan, dan, dwalsh, extras-qa, fidencio, gmarr, jmontleo, jsullivan3, kparal, lvrabec, mailings, mgrepl, mmaslano, nphilipp, plambri, plautrba, pmarciniak, rmy, robatino, robin.bjorklin, tmraz
Target Milestone: ---Keywords: CommonBugs, Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: https://fedoraproject.org/wiki/Common_F29_bugs#crond-selinux RejectedBlocker AcceptedFreezeException PrioritizedBug
Fixed In Version: cronie-1.5.2-3.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1625645 Environment:
Last Closed: 2018-11-03 00:01:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1625645    
Bug Blocks: 1517014    

Description Lukas Vrabec 2018-10-15 15:22:32 UTC
+++ This bug was initially created as a clone of Bug #1625645 +++

Description of problem:
On a clean F29 install, selinux blocks everything inside /etc/cron.d and it seems cron is not able to load those files. The journal for cron service says:

Sep 05 14:09:43 localhost-live.default systemd[1]: Started Command Scheduler.
Sep 05 14:09:43 localhost-live.default crond[3275]: (CRON) STARTUP (1.5.2)
Sep 05 14:09:43 localhost-live.default crond[3275]: (CRON) INFO (Syslog will be used instead of sendmail.)
Sep 05 14:09:43 localhost-live.default crond[3275]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 88% if used.)
Sep 05 14:09:44 localhost-live.default crond[3275]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/crontab)
Sep 05 14:09:44 localhost-live.default crond[3275]: (root) FAILED (loading cron table)
Sep 05 14:09:44 localhost-live.default crond[3275]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/raid-check)
Sep 05 14:09:44 localhost-live.default crond[3275]: (root) FAILED (loading cron table)
Sep 05 14:09:44 localhost-live.default crond[3275]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/0hourly)
Sep 05 14:09:44 localhost-live.default crond[3275]: (root) FAILED (loading cron table)
Sep 05 14:09:44 localhost-live.default crond[3275]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=unconfined_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/test)
Sep 05 14:09:44 localhost-live.default crond[3275]: (root) FAILED (loading cron table)
Sep 05 14:09:44 localhost-live.default crond[3275]: (CRON) INFO (running with inotify support)
Sep 05 14:09:44 localhost-live.default crond[3275]: (CRON) INFO (@reboot jobs will be run at computer's startup.)

I have created one extra file in /etc/cron.d/test, as you can see in the output (it has slightly different selinux labels, I have no idea whether that's important), and it's blocked as well. I have verified that the command inside that test job is not executed.

When I switch to selinux permissive and restart cron service, it loads the jobs fine, and my test job is getting executed:

Sep 05 14:07:50 localhost-live.default crond[3190]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/crontab)
Sep 05 14:07:50 localhost-live.default crond[3190]: ((null)) SELinux in permissive mode, continuing (/etc/crontab)
Sep 05 14:07:50 localhost-live.default crond[3190]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/raid-check)
Sep 05 14:07:50 localhost-live.default crond[3190]: ((null)) SELinux in permissive mode, continuing (/etc/cron.d/raid-check)
Sep 05 14:07:50 localhost-live.default crond[3190]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/0hourly)
Sep 05 14:07:50 localhost-live.default crond[3190]: ((null)) SELinux in permissive mode, continuing (/etc/cron.d/0hourly)
Sep 05 14:07:50 localhost-live.default crond[3190]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=unconfined_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/test)
Sep 05 14:07:50 localhost-live.default crond[3190]: ((null)) SELinux in permissive mode, continuing (/etc/cron.d/test)

When I switch to enforcing afterwards, but don't restart cron job, cron is still trying to execute my test job, but fails with:

Sep 05 14:09:01 localhost-live.default crond[3253]: (*system*) NULL security context for user ()
Sep 05 14:09:01 localhost-live.default crond[3253]: (root) ERROR (failed to change SELinux context)

`ausearch -m AVC -ts boot` doesn't print anything related to cron, nor seapplet shows anything related. But it is a clean F29 Workstation install, so it's very easy to reproduce it.


Version-Release number of selected component (if applicable):
cronie-1.5.2-2.fc29.x86_64
selinux-policy-3.14.2-32.fc29.noarch

How reproducible:
always

Steps to Reproduce:
1. cleanly install F29
2. see `journalctl -b -u crond.service` and look for errors

Additional info:
This seems very similar to bug 1547368, but the reported selinux context is different (system_cronjob_t vs system_cron_spool_t).

--- Additional comment from Kamil Páral on 2018-09-05 08:32:52 EDT ---

I'm not sure we have a criterion for this, but this could be a big issue for Server (of course not only that), so I'm proposing it for a blocker discussion.

--- Additional comment from Lukas Vrabec on 2018-09-05 11:45:25 EDT ---



--- Additional comment from Lukas Vrabec on 2018-09-05 11:52:53 EDT ---

Kamil,

Could you boot in permissive and attach here audit.log? (/var/log/audit/audit.log)

THanks,
Lukas.

--- Additional comment from Kamil Páral on 2018-09-06 08:00 EDT ---



--- Additional comment from Lukas Vrabec on 2018-09-07 12:18:12 EDT ---

I believe this is some issue in cron. Guys did you change something SELinux related in conde? 

Thanks,
Lukas.

--- Additional comment from Tomas Mraz on 2018-09-10 03:22:14 EDT ---

There was no change in cronie recently.

--- Additional comment from Tomas Mraz on 2018-09-10 03:23:54 EDT ---

What is the SELinux domain of crond? ps axZ | grep crond

--- Additional comment from Kamil Páral on 2018-09-10 09:07:04 EDT ---

(In reply to Tomas Mraz from comment #7)
> What is the SELinux domain of crond? ps axZ | grep crond

system_u:system_r:crond_t:s0-s0:c0.c1023 958 ? Ss     0:00 /usr/sbin/crond -n

--- Additional comment from Tomas Mraz on 2018-09-10 09:16:06 EDT ---

And does the issue still happen? The reason I am asking is because in the log above I see unconfined_t domain where your crond clearly runs with crond_t.

--- Additional comment from Kamil Páral on 2018-09-10 09:51:37 EDT ---

Yes, it still happens, the log is still the same. I have no idea why the domain in the log doesn't match the process domain.

--- Additional comment from Geoffrey Marr on 2018-09-17 16:11:36 EDT ---

Discussed during the 2018-09-17 blocker review meeting: [1]

The decision to delay the classification of this as a blocker bug was made as this doesn't seem to be violating any existing criteria, but the group was fairly worried about it on merit. We will delay the decision to investigate further and to provide an opportunity for someone to propose a new criterion if they want to.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-09-17/f29-blocker-review.2018-09-17-16.02.txt

--- Additional comment from Jason Montleon on 2018-09-24 12:05:39 EDT ---

I am seeing this on three systems after upgrading from Fedora 28 to 29.

A slightly modified workaround taken from BZ 1298192 (there it was a similar error but for user crons) got things working for me until this is properly fixed:

cat custom_cron.cil:
(allow unconfined_t system_cron_spool_t( file ( entrypoint)))

semodule -i custom_cron.cil

Once the bug is fixed  use `semodule -r custom_cron` to remove it.

--- Additional comment from Geoffrey Marr on 2018-09-24 15:26:50 EDT ---

Discussed during the 2018-09-24 blocker review meeting: [1]

The decision to delay the classification of this as a blocker bug was made as no criterion has yet been proposed that would cover this, but we are still a bit concerned about it and willing to give it another week for further investigation and potential criteria proposals before rejecting.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-09-24/f29-blocker-review.2018-09-24-16.05.txt

--- Additional comment from Geoffrey Marr on 2018-10-01 15:42:26 EDT ---

Discussed during the 2018-10-01 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as it does not appear to violate any specific criterion, but it could be revisited if one is proposed. We accept it as a freeze exception issue as it'd obviously be for the best if this can be fixed as promptly as possible, and SELinux policy fixes are usually safe.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-10-01/f29-blocker-review.2018-10-01-16.00.txt

--- Additional comment from Kamil Páral on 2018-10-02 07:59:38 EDT ---

I think this is very severe especially for Server. Proposing as a prioritized bug.

--- Additional comment from Ben Cotton on 2018-10-10 15:36:19 EDT ---

This was accepted as a PrioritizedBug: https://meetbot.fedoraproject.org/fedora-meeting/2018-10-10/fedora_prioritized_bugs_and_issues.2018-10-10-15.02.log.html#l-83

--- Additional comment from Lukas Vrabec on 2018-10-15 11:21:53 EDT ---

I had discussion with cronie maintainer and we see bugs in selinux-policy and also in cronie. 

I'll add rules to allow system_cron_spool_t domain to be entrypoint for userdomains. 

Tomas could you please change behavior in cronie code that system jobs will end up with system_cronjob_t domain? (creating clone for cronie)

Thanks,
Lukas.

Comment 1 Bojan Smojver 2018-10-24 10:40:29 UTC
So, the selinux part has been closed. Does that mean the issue is fixed?

Comment 2 Tomas Mraz 2018-10-24 11:07:47 UTC
The symptoms are fixed however a proper fix requires modification of cronie otherwise further changes in SELinux policy could again regress this.

This can be done after F29 is released.

Comment 3 Bojan Smojver 2018-10-24 11:35:31 UTC
(In reply to Tomas Mraz from comment #2)
> The symptoms are fixed however a proper fix requires modification of cronie
> otherwise further changes in SELinux policy could again regress this.
> 
> This can be done after F29 is released.

OK, thanks for the info. I'm asking from a purely practical perspective, because cron jobs that won't run is a major problem for anyone running Fedora on a server (myself included).

Comment 4 John F Sullivan 2018-10-27 18:26:07 UTC
As noted in bug 1625645, the symptoms are still present in an updated version of Fedora 29 Beta:

cron jobs simply do not run in my Fedora 29 Beta installations without the workaround described in comment 12.

Output of 'rpm -qa | grep selinux | sort':
  container-selinux-2.74-1.gita62c2db.fc29.noarch
  libselinux-2.8-4.fc29.x86_64
  libselinux-utils-2.8-4.fc29.x86_64
  python2-libselinux-2.8-4.fc29.x86_64
  python3-libselinux-2.8-4.fc29.x86_64
  rpm-plugin-selinux-4.14.2-1.fc29.x86_64
  selinux-policy-3.14.2-40.fc29.noarch
  selinux-policy-devel-3.14.2-40.fc29.noarch
  selinux-policy-targeted-3.14.2-40.fc29.noarch

Output of 'rpm -qa | grep cron | sort':
  cronie-1.5.2-2.fc29.x86_64
  cronie-anacron-1.5.2-2.fc29.x86_64
  crontabs-1.11-17.20150630git.fc29.noarch

Output of 'ps -efZ | grep cron
  system_u:system_r:crond_t:s0-s0:c0.c1023 root 1574 1  0 Oct26 ?        00:00:00 /usr/sbin/atd -f
  system_u:system_r:crond_t:s0-s0:c0.c1023 root 10745 1  0 14:13 ?       00:00:00 /usr/sbin/crond -n
  unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 10871 9441  0 14:17 pts/0 00:00:00 grep --color=auto cron

After issuing "systemctl restart crond" the output of 'systemctl status crond --no-pager -l':
  ● crond.service - Command Scheduler
     Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2018-10-27 14:19:29 EDT; 9s ago
   Main PID: 11002 (crond)
      Tasks: 1 (limit: 1148)
     Memory: 616.0K
     CGroup: /system.slice/crond.service
             └─11002 /usr/sbin/crond -n
  
  Oct 27 14:19:29 test-rivendell crond[11002]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/0hourly)
  Oct 27 14:19:29 test-rivendell crond[11002]: (root) FAILED (loading cron table)
  Oct 27 14:19:29 test-rivendell crond[11002]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/raid-check)
  Oct 27 14:19:29 test-rivendell crond[11002]: (root) FAILED (loading cron table)
  Oct 27 14:19:29 test-rivendell crond[11002]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/clamav-update)
  Oct 27 14:19:29 test-rivendell crond[11002]: (root) FAILED (loading cron table)
  Oct 27 14:19:29 test-rivendell crond[11002]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/rsnapshot)
  Oct 27 14:19:29 test-rivendell crond[11002]: (root) FAILED (loading cron table)
  Oct 27 14:19:29 test-rivendell crond[11002]: (CRON) INFO (running with inotify support)
  Oct 27 14:19:29 test-rivendell crond[11002]: (CRON) INFO (@reboot jobs will be run at computer's startup.)

Comment 5 John F Sullivan 2018-10-27 18:27:11 UTC
Sorry - that's bug 1625645 comment 12...

Comment 6 Bojan Smojver 2018-10-27 22:07:44 UTC
(In reply to Tomas Mraz from comment #2)
> The symptoms are fixed however a proper fix requires modification of cronie
> otherwise further changes in SELinux policy could again regress this.
> 
> This can be done after F29 is released.

Definitely untrue. I can confirm that cron jobs from /etc/cron.d are definitely not running on F29 final (with all the testing updates applied as well). Example:

Oct 27 09:25:53 <system> crond[904]: ((null)) Unauthorized SELinux context=system_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 file_context=system_u:object_r:system_cron_spool_t:s0 (/etc/cron.d/clamav-update)

This should have been a blocker. A bit too late now, but let's hope it gets fixed soon. Otherwise, everyone and their dog is going to pile on here...

Comment 7 Nils Philippsen 2018-10-28 20:28:00 UTC
Can confirm that crond only loads tabs after applying the workaround linked from comment #5 -- https://bugzilla.redhat.com/show_bug.cgi?id=1625645#c12

Comment 8 Fedora Update System 2018-10-31 14:21:19 UTC
cronie-1.5.2-3.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-fade853384

Comment 9 Tomas Mraz 2018-11-01 07:52:57 UTC
*** Bug 1625645 has been marked as a duplicate of this bug. ***

Comment 10 Fedora Update System 2018-11-01 15:40:37 UTC
cronie-1.5.2-3.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-fade853384

Comment 11 John F Sullivan 2018-11-02 00:34:57 UTC
The fix is looking good to me in my environment.  I removed the workaround listed earlier.

Output of 'rpm -qa | grep cron | sort':
  cronie-1.5.2-3.fc29.x86_64
  cronie-anacron-1.5.2-3.fc29.x86_64
  crontabs-1.11-17.20150630git.fc29.noarch

Output of 'ps -efZ | grep cron
  system_u:system_r:crond_t:s0-s0:c0.c1023 root 1121 1  0 20:22 ?        00:00:00 /usr/sbin/atd -f
  system_u:system_r:crond_t:s0-s0:c0.c1023 root 1135 1  0 20:22 ?        00:00:00 /usr/sbin/crond -n
  unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 7696 1704  0 20:31 pts/1 00:00:00 grep --color=auto cron

Output of 'systemctl status crond --no-pager':
  ● crond.service - Command Scheduler
     Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2018-11-01 20:22:29 EDT; 4min 29s ago
   Main PID: 1135 (crond)
      Tasks: 1 (limit: 2358)
     Memory: 1.1M
     CGroup: /system.slice/crond.service
             └─1135 /usr/sbin/crond -n
  
  Nov 01 20:22:29 localhost.localdomain systemd[1]: Started Command Scheduler.
  Nov 01 20:22:29 localhost.localdomain crond[1135]: (CRON) STARTUP (1.5.2)
  Nov 01 20:22:29 localhost.localdomain crond[1135]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 79% if used.)
  Nov 01 20:22:29 localhost.localdomain crond[1135]: (CRON) INFO (running with inotify support)

Test cron job:
 echo "* * * * * root date >> /tmp/cronstuffs.txt" > /etc/cron.d/test

And we have output from  'tail /tmp/cronstuffs.txt':
  Thu Nov  1 20:31:01 EDT 2018
  Thu Nov  1 20:32:01 EDT 2018

Thank you!

Comment 12 Tomas Mraz 2018-11-02 07:18:55 UTC
*** Bug 1645300 has been marked as a duplicate of this bug. ***

Comment 13 Fedora Update System 2018-11-03 00:01:38 UTC
cronie-1.5.2-3.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 14 Tomas Mraz 2018-11-05 07:59:13 UTC
*** Bug 1645825 has been marked as a duplicate of this bug. ***