Bug 1385167 - Httpd can't read files created by certbot when started from /etc/crontab
Summary: Httpd can't read files created by certbot when started from /etc/crontab
Alias: None
Product: Fedora
Classification: Fedora
Component: selinux-policy-targeted
Version: 25
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Lukas Vrabec
QA Contact: Ben Levenson
Depends On:
Blocks: 1377312
TreeView+ depends on / blocked
Reported: 2016-10-14 20:40 UTC by Göran Uddeborg
Modified: 2017-12-12 10:57 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-12-12 10:57:02 UTC
Type: Bug

Attachments (Terms of Use)

Description Göran Uddeborg 2016-10-14 20:40:31 UTC
Description of problem:
When using Letsencrypt certificates, the instructions (https://certbot.eff.org/all-instructions/#fedora-23-apache) suggests running certbot twice a day.  The Fedora certbot package doesn't provide any automatic way to do this (bug 1377312), and I chose to put a line in /etc/crontab.  When it was time for certificate renewal, it failed and triggered the following AVC:

type=AVC msg=audit(1476326226.546:62004): avc:  denied  { getattr } for  pid=25869 comm="httpd" path="/var/lib/letsencrypt/Qj0ahmMsp9sdCWYO_1XRyUQBYLfEb95rR45XWqPiKmA.crt" dev="dm-0" ino=100335648 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cron_var_lib_t:s0 tclass=file permissive=0

Version-Release number of selected component (if applicable):

How reproducible:
Every time (when the certificate needs renewal)

Steps to Reproduce:
1. Configure a Letsencrypt certificate
2. Put a "certbot renew" job in /etc/crontab
3. Wait two months :-)

Actual results:
The renewal fails.

Expected results:
The renewal should succeed.

Additional info:
I worked around this problem with an additional SELinux module with the following rule:

read_files_pattern(httpd_t, cron_var_lib_t, cron_var_lib_t)

This may or may not be the correct fix.  As I understand it, different ways to start scheduled jobs put them in different domains.  If started from /etc/crontab they run in system_cronjob_t while if started from a crontab in /var/spool/cron (created with the "crontab" command) they run in unconfined_t.  I haven't checked others.  Thus, another option might be to coordinate with the certbot packager to start it in a way where the files created in /var/lib/letsencrypt gets a type httpd_t already can read.  Perhaps the certbot should have a special type so it can do a transition when started from cron and/or systemd.

As long as the certbot package doesn't contain any default way to automate the renewal, having a line in /etc/crontab seems like something that should be supported.

Comment 1 Dominik 'Rathann' Mierzejewski 2016-12-14 00:49:21 UTC
Reproducible on F24 as well:
$ rpm -qa \*certbot\* selinux-policy\* |sort

Comment 2 Dominik 'Rathann' Mierzejewski 2017-01-12 10:32:50 UTC
FYI this is the local policy module that I'm running to work around this:

# cat certbot-local.cil
(typeattributeset cil_gen_require httpd_t)
(typeattributeset cil_gen_require cron_var_lib_t)
(allow httpd_t cron_var_lib_t (file (getattr open read)))

I guess it's equivalent to what the original reporter posted.

Comment 3 James Hogarth 2017-03-06 13:04:06 UTC
certbot stores the certificates that have been signed in /etc/letsencrypt/live/ with archives of previous certificates in /etc/letsencrypt/archive/

You really should be using the /etc/.../live/ directory to point httpd to.

This doesn't have the right selinux label yet, but a PR is in place (and the certbot package will be updated to handle this as well if need be soon) to label everything in those directories as cert_t ... which httpd can read with no additional selinux module needed.

On my personal server I have:

/etc/letsencrypt/(live|archive)(/.*)?  all files system_u:object_r:cert_t:s0 

This works fine for a systemd timer based renewal (coming soon to a certbot package near you!)

Comment 4 Edward Rudd 2017-04-13 18:18:38 UTC

I am also having this issue on CentOS 7..  It is a stock install from EPEL.. 

It seems that /var/lib/letsencrypt is used by default as it's the "working directory" that letencrypt is using.  So the issue is not for the final "live" location of the certs, but the temporary location while things are being negotiated w/ letencrypt.

Now, by me adding this context to my selinux setup the issue is fully resolved.

semanage fcontext -a '/var/lib/letsencrypt(/.*)?'  -t cert_t

Perhaps the default setup of certbot RPM should be configure to either A) include that fcontext or B) use a different folder for it's working directory?

Also, I tried the timer service, and it just simply doesn't work. there are issues with how it's passing the command line parameters in so it NEVER runs.

I've been trying `systemctrl start certbot-renew.service` and this is the output that spews forth.

Apr 13 14:08:16 iis systemd: Starting This service automatically renews any certbot certificates found...
Apr 13 14:08:17 iis certbot: usage:
Apr 13 14:08:17 iis certbot: certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...
Apr 13 14:08:17 iis certbot: Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
Apr 13 14:08:17 iis certbot: it will attempt to use a webserver both for obtaining and installing the
Apr 13 14:08:17 iis certbot: cert.
Apr 13 14:08:17 iis certbot: certbot: error: argument --pre-hook: expected one argument
Apr 13 14:08:17 iis systemd: certbot-renew.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 13 14:08:17 iis systemd: Failed to start This service automatically renews any certbot certificates found.
Apr 13 14:08:17 iis systemd: Unit certbot-renew.service entered failed state.
Apr 13 14:08:17 iis systemd: certbot-renew.service failed.

Comment 5 Milan Kerslager 2017-07-24 21:13:20 UTC
The problem is still there. I have in /var/log/audit/audit.log denied write for /etc/letsencrypt/.certbot.lock and /var/log/letsencrypt/.certbot.lock

This is possible to run "certbot renew" as root, but it is not possible to run this command by cronjob as suggested by certbot page when SELinux is enforcing.

Comment 6 Göran Uddeborg 2017-07-24 21:25:24 UTC
But you don't need to run it as a cronjob any more.  The package maintainer worked around this problem with the systemd style certbot-renew.timer instead.  It will trigger the certbot renew task regularly.  At least it works fine for me.   Certbot is run once each night, and successfully renewed my certificates some weeks ago.

Comment 7 Milan Kerslager 2017-07-24 21:28:04 UTC
Ah, thanx. So the workaround is:

systemctl enable certbot-renew
systemctl start certbot-renew

Comment 8 Göran Uddeborg 2017-07-24 21:36:29 UTC
Now, I'm not 100% sure about the systemd motions I did, but I believe the important step is to enable the timer:

systemctl enable certbot-renew.timer

On my machine, it looks like this currently:

23:32 mimmi$ systemctl status certbot-renew.service certbot-renew.timer
● certbot-renew.service - This service automatically renews any certbot certificates found
   Loaded: loaded (/usr/lib/systemd/system/certbot-renew.service; static; vendor preset: disabled)
   Active: inactive (dead)

● certbot-renew.timer - This is the timer to set the schedule for automated renewals
   Loaded: loaded (/usr/lib/systemd/system/certbot-renew.timer; enabled; vendor preset: disabled)
   Active: inactive (dead)

Do a "journalctl -u certbot-renew" the next day and verify.

Comment 9 Fedora End Of Life 2017-11-16 19:09:01 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 10 Fedora End Of Life 2017-12-12 10:57:02 UTC
Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.