Bug 1701018

Summary: certbot apache fails to issue/renew certificate on new apache 2.4.39
Product: [Fedora] Fedora Reporter: Michal Ambroz <rebus>
Component: python-certbot-apacheAssignee: James Hogarth <james.hogarth>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: alex.ploumistos, anon.amish, cegolf, dominik, elyscape, gscott, itamar, james.hogarth, nb, nick, rbu, redhat-bugzilla, rkudyba, sergio.pasra
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-certbot-apache-0.34.2-1.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-01 00:51:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michal Ambroz 2019-04-17 19:25:41 UTC
Description of problem:
The Apache apachectl from version 2.4.39 (which came with updates now) stopped to pass all the unknown attributes to the httpd command. The certbot was relying on this feature.
As result it is not possible to issue/renew letsencrypt certificates in the current stable Fedora 29.

Version-Release number of selected component (if applicable):
python-certbot-apache-0.31.0-1.fc29
httpd-2.4.39-2.fc29

How reproducible:
100%

Steps to Reproduce:
1. have certbot installed
dnf install certbot certbot-apache

2. update F29 with the recent apache updates
dnf -y update httpd

3. try renew with the certbot
certbot renew

Actual results:
# certbot renew                                                                                                                                                                          
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/example.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Error while running apachectl -v.

apachectl: The "-v" option is not supported.

Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.                                                                                    
The error was: PluginError("Unable to run ['apachectl', '-v'] -v")
Attempting to renew cert (example.com) from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.               
The error was: PluginError("Unable to run ['apachectl', '-v'] -v"). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/example.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/example.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

Expected results:
certificates should renew without errors


Workaround:
dnf -y downgrade httpd
certbot renew
dnf -y upgrade httpd

Additional info:
https://github.com/certbot/certbot/issues/6940

Comment 1 Kevin R. Page 2019-05-23 15:35:26 UTC
certbot also fails on Fedora 30 for this reason, but with no httpd package to downgrade to.

Is there any alternative workaround?

Comment 2 Greg Scott 2019-05-26 16:45:55 UTC
How irritating. I should have https-ified my website a long time ago, but I'm just now getting to it. I spent the past several days reading up on certbot and LetsEncrypt and these look like the best tools to do it. I was ready to try it today and...

*****************
[root@www ~]# certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Error while running apachectl -v.

apachectl: The "-v" option is not supported.


How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): c
Could not choose appropriate plugin: authenticator could not be determined or is not installed
authenticator could not be determined or is not installed
[root@www ~]# 
*****************

According to this comment thread from https://github.com/certbot/certbot/issues/6940

***************** 

 ROV28 commented 7 days ago

It is solved (certbot-0.31.0-2.fc30.noarch.rpm).
@Barborica-Alexandru
Barborica-Alexandru commented 6 days ago •

Sorry but I have the same package as you and it is not yet solved.

I believe the fix is in the package python3-certbot-apache-0.31.0-2.fc30.noarch which is not yet pushed.
******************

But it looks like the latest with Fedora 30 right now is python3-certbot-apache-0.31.0-1.fc30.noarch. Hopefully the update comes soon. Meantime, I guess we wait.

Comment 3 Kevin R. Page 2019-05-26 22:26:39 UTC
As a temporary hack this worked for me, following the suggestion from:
https://github.com/certbot/certbot/issues/6940#issuecomment-483089957

Referring to:
/usr/lib/python3.7/site-packages/certbot_apache/override_centos.py
/usr/lib/python3.7/site-packages/certbot_apache/configurator.py

1: In override_centos.py, I changed line 21 to ctl="httpd"
2: In override_centos.py, I changed line 22 to version_cmd=['httpd', '-v']
3: In configurator.py, I commented out lines 135, 136, and 137

(i.e. modify the scripts to use httpd rather than apachectl)

Comment 4 RobbieTheK 2019-05-28 03:31:20 UTC
(In reply to Kevin R. Page from comment #3)
> As a temporary hack this worked for me, following the suggestion from:
> https://github.com/certbot/certbot/issues/6940#issuecomment-483089957
> 
> Referring to:
> /usr/lib/python3.7/site-packages/certbot_apache/override_centos.py
> /usr/lib/python3.7/site-packages/certbot_apache/configurator.py
> 
> 1: In override_centos.py, I changed line 21 to ctl="httpd"
> 2: In override_centos.py, I changed line 22 to version_cmd=['httpd', '-v']
> 3: In configurator.py, I commented out lines 135, 136, and 137
> 
> (i.e. modify the scripts to use httpd rather than apachectl)

I'm getting:
An unexpected error occurred:
  File "/usr/lib/python3.7/site-packages/certbot_apache/override_centos.py", line 25
    restart_cmd=['apachectl', 'graceful'],

Looks like it's the next line. I tried:
version_cmd=['apachectl', '-v'],

Same error.

Comment 5 RobbieTheK 2019-05-28 03:32:22 UTC
Sorry I meant I tried:
version_cmd=['httpd', '-k', 'restart '],

Comment 6 Greg Scott 2019-05-28 04:16:53 UTC
I also tinkered with that workaround today with Fedora 30, but certbot kept blowing up with various syntax errors.

One example - 

[root@www ~]# apachectl configtest
Syntax OK
[root@www ~]# httpd configtest
Usage: httpd [-D name] [-d directory] [-f file]
             [-C "directive"] [-c "directive"]
             [-k start|restart|graceful|graceful-stop|stop]
             [-v] [-V] [-h] [-l] [-L] [-t] [-T] [-S] [-X]
.
.
.

So it's not as simple as swapping "httpd" for "apachectl" everywhere.

If there truly is an upstream fix, would it be possible to quickly package it in a Fedora update?

Comment 7 Eli Young 2019-05-28 06:59:42 UTC
Apologies for the delay. I'll get the updates into testing tomorrow.

Comment 8 Fedora Update System 2019-05-29 01:42:07 UTC
FEDORA-2019-2361aca4af has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-2361aca4af

Comment 9 Fedora Update System 2019-05-29 01:43:14 UTC
FEDORA-2019-d81f439a8d has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-d81f439a8d

Comment 10 RobbieTheK 2019-05-29 02:35:03 UTC
Still getting this:
/usr/bin/certbot -auto renew
An unexpected error occurred:
  File "/usr/lib/python3.7/site-packages/certbot_apache/override_centos.py", line 25
    restart_cmd=['apachectl', 'graceful'],
              ^
SyntaxError: invalid syntax

Do the override_* files get upgraded here? Doesn't look like it.

Comment 11 Eli Young 2019-05-29 03:02:50 UTC
(In reply to RobbieTheK from comment #10)
> Still getting this:
> /usr/bin/certbot -auto renew
> An unexpected error occurred:
>   File "/usr/lib/python3.7/site-packages/certbot_apache/override_centos.py",
> line 25
>     restart_cmd=['apachectl', 'graceful'],
>               ^
> SyntaxError: invalid syntax
> 
> Do the override_* files get upgraded here? Doesn't look like it.

Have you upgraded to 0.34.2? I don’t believe the builds have hit the testing repos yet, so this would have to have been done manually.

Comment 12 RobbieTheK 2019-05-29 03:10:18 UTC
(In reply to Eli Young from comment #11)
> (In reply to RobbieTheK from comment #10)
> > Still getting this:
> > /usr/bin/certbot -auto renew
> > An unexpected error occurred:
> >   File "/usr/lib/python3.7/site-packages/certbot_apache/override_centos.py",
> > line 25
> >     restart_cmd=['apachectl', 'graceful'],
> >               ^
> > SyntaxError: invalid syntax
> > 
> > Do the override_* files get upgraded here? Doesn't look like it.
> 
> Have you upgraded to 0.34.2? I don’t believe the builds have hit the testing
> repos yet, so this would have to have been done manually.

Well I think so have a look:

dnf reinstall https://kojipkgs.fedoraproject.org//packages/certbot/0.34.2/3.fc29/noarch/certbot-0.34.2-3.fc29.noarch.rpm
Last metadata expiration check: 0:22:17 ago on Tue 28 May 2019 10:45:35 PM EDT.
certbot-0.34.2-3.fc29.noarch.rpm                                                                                      71 kB/s |  36 kB     00:00    
Dependencies resolved.
=====================================================================================================================================================
 Package                          Architecture                    Version                                Repository                             Size
=====================================================================================================================================================
Reinstalling:
 certbot                          noarch                          0.34.2-3.fc29                          @commandline                           36 k

Transaction Summary
=====================================================================================================================================================

Total size: 36 k
Installed size: 85 k
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                             1/1 
  Running scriptlet: certbot-0.34.2-3.fc29.noarch                                                                                                1/1 
  Reinstalling     : certbot-0.34.2-3.fc29.noarch                                                                                                1/2 
  Running scriptlet: certbot-0.34.2-3.fc29.noarch                                                                                                1/2 
  Cleanup          : certbot-0.34.2-3.fc29.noarch                                                                                                2/2 
  Running scriptlet: certbot-0.34.2-3.fc29.noarch                                                                                                2/2 
  Verifying        : certbot-0.34.2-3.fc29.noarch                                                                                                1/2 
  Verifying        : certbot-0.34.2-3.fc29.noarch                                                                                                2/2 

Reinstalled:
  certbot-0.34.2-3.fc29.noarch                                                                                                                       

Complete!
[root@dsm kudyba]# rpm -q certbot
certbot-0.34.2-3.fc29.noarch

/usr/bin/certbot -auto renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ourdomain.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Error while running apachectl -v.

apachectl: The "-v" option is not supported.

Could not choose appropriate plugin: The requested uto plugin does not appear to be installed
Attempting to renew cert (ourdomain) from /etc/letsencrypt/renewal/ourdomain.conf produced an unexpected error: The requested uto plugin does not appear to be installed. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ourdomain/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ourdomain/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

Comment 13 Eli Young 2019-05-29 20:55:55 UTC
(In reply to RobbieTheK from comment #12)
> Well I think so have a look:
> 
> dnf reinstall
> https://kojipkgs.fedoraproject.org//packages/certbot/0.34.2/3.fc29/noarch/
> certbot-0.34.2-3.fc29.noarch.rpm

Did you also update the appropriate python-acme, python-certbot, and python-certbot-apache packages? Those do the actual work. If not, determine if you're using the python2 or python3 subpackages (you're probably on python3) by running:

    dnf repoquery --installed --whatprovides certbot-apache

Once you've done that, select and run the appropriate command from the following, then try again:

Fedora 29, Python 2:

    dnf install https://kojipkgs.fedoraproject.org/packages/python-configargparse/0.14.0/2.fc29/noarch/python2-configargparse-0.14.0-2.fc29.noarch.rpm https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc29/noarch/certbot-0.34.2-3.fc29.noarch.rpm https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc29/noarch/python2-certbot-0.34.2-3.fc29.noarch.rpm https://kojipkgs.fedoraproject.org/packages/python-acme/0.34.2/1.fc29/noarch/python2-acme-0.34.2-1.fc29.noarch.rpm https://kojipkgs.fedoraproject.org/packages/python-certbot-apache/0.34.2/1.fc29/noarch/python2-certbot-apache-0.34.2-1.fc29.noarch.rpm

Fedora 29, Python 3:

    dnf install https://kojipkgs.fedoraproject.org/packages/python-configargparse/0.14.0/2.fc29/noarch/python3-configargparse-0.14.0-2.fc29.noarch.rpm https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc29/noarch/certbot-0.34.2-3.fc29.noarch.rpm https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc29/noarch/python3-certbot-0.34.2-3.fc29.noarch.rpm https://kojipkgs.fedoraproject.org/packages/python-acme/0.34.2/1.fc29/noarch/python3-acme-0.34.2-1.fc29.noarch.rpm https://kojipkgs.fedoraproject.org/packages/python-certbot-apache/0.34.2/1.fc29/noarch/python3-certbot-apache-0.34.2-1.fc29.noarch.rpm

Fedora 30:

    dnf install https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc30/noarch/certbot-0.34.2-3.fc30.noarch.rpm https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc30/noarch/python3-certbot-0.34.2-3.fc30.noarch.rpm https://kojipkgs.fedoraproject.org/packages/python-acme/0.34.2/1.fc30/noarch/python3-acme-0.34.2-1.fc30.noarch.rpm https://kojipkgs.fedoraproject.org/packages/python-certbot-apache/0.34.2/1.fc30/noarch/python3-certbot-apache-0.34.2-1.fc30.noarch.rpm

Comment 14 RobbieTheK 2019-05-29 21:09:15 UTC
(In reply to Eli Young from comment #13)
> (In reply to RobbieTheK from comment #12)
> > Well I think so have a look:
> > 
> > dnf reinstall
> > https://kojipkgs.fedoraproject.org//packages/certbot/0.34.2/3.fc29/noarch/
> > certbot-0.34.2-3.fc29.noarch.rpm
> 
> Did you also update the appropriate python-acme, python-certbot, and
> python-certbot-apache packages? Those do the actual work. If not, determine
> if you're using the python2 or python3 subpackages (you're probably on
> python3) by running:
> 
>     dnf repoquery --installed --whatprovides certbot-apache
> 
> Once you've done that, select and run the appropriate command from the
> following, then try again:
> 
> Fedora 29, Python 2:
> 
>     dnf install
> https://kojipkgs.fedoraproject.org/packages/python-configargparse/0.14.0/2.
> fc29/noarch/python2-configargparse-0.14.0-2.fc29.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc29/noarch/
> certbot-0.34.2-3.fc29.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc29/noarch/
> python2-certbot-0.34.2-3.fc29.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/python-acme/0.34.2/1.fc29/noarch/
> python2-acme-0.34.2-1.fc29.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/python-certbot-apache/0.34.2/1.
> fc29/noarch/python2-certbot-apache-0.34.2-1.fc29.noarch.rpm
> 
> Fedora 29, Python 3:
> 
>     dnf install
> https://kojipkgs.fedoraproject.org/packages/python-configargparse/0.14.0/2.
> fc29/noarch/python3-configargparse-0.14.0-2.fc29.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc29/noarch/
> certbot-0.34.2-3.fc29.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc29/noarch/
> python3-certbot-0.34.2-3.fc29.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/python-acme/0.34.2/1.fc29/noarch/
> python3-acme-0.34.2-1.fc29.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/python-certbot-apache/0.34.2/1.
> fc29/noarch/python3-certbot-apache-0.34.2-1.fc29.noarch.rpm
> 
> Fedora 30:
> 
>     dnf install
> https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc30/noarch/
> certbot-0.34.2-3.fc30.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/certbot/0.34.2/3.fc30/noarch/
> python3-certbot-0.34.2-3.fc30.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/python-acme/0.34.2/1.fc30/noarch/
> python3-acme-0.34.2-1.fc30.noarch.rpm
> https://kojipkgs.fedoraproject.org/packages/python-certbot-apache/0.34.2/1.
> fc30/noarch/python3-certbot-apache-0.34.2-1.fc30.noarch.rpm

Yes indeed that worked, just had to remember to use a double dash before auto as copy/paste sometimes uses an em long dash: /usr/bin/certbot --auto 
renew

Thanks!

Comment 15 Fedora Update System 2019-05-30 13:57:41 UTC
certbot-0.34.2-3.fc30, python-acme-0.34.2-1.fc30, python-certbot-apache-0.34.2-1.fc30, python-certbot-dns-cloudflare-0.34.2-1.fc30, python-certbot-dns-cloudxns-0.34.2-1.fc30, python-certbot-dns-digitalocean-0.34.2-1.fc30, python-certbot-dns-dnsimple-0.34.2-1.fc30, python-certbot-dns-dnsmadeeasy-0.34.2-1.fc30, python-certbot-dns-gehirn-0.34.2-1.fc30, python-certbot-dns-google-0.34.2-1.fc30, python-certbot-dns-linode-0.34.2-1.fc30, python-certbot-dns-luadns-0.34.2-1.fc30, python-certbot-dns-nsone-0.34.2-1.fc30, python-certbot-dns-ovh-0.34.2-1.fc30, python-certbot-dns-rfc2136-0.34.2-1.fc30, python-certbot-dns-route53-0.34.2-1.fc30, python-certbot-dns-sakuracloud-0.34.2-1.fc30, python-certbot-nginx-0.34.2-1.fc30, python-dns-lexicon-3.2.6-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d81f439a8d

Comment 16 Fedora Update System 2019-05-30 15:34:54 UTC
certbot-0.34.2-3.fc29, python-acme-0.34.2-1.fc29, python-certbot-apache-0.34.2-1.fc29, python-certbot-dns-cloudflare-0.34.2-1.fc29, python-certbot-dns-cloudxns-0.34.2-1.fc29, python-certbot-dns-digitalocean-0.34.2-1.fc29, python-certbot-dns-dnsimple-0.34.2-1.fc29, python-certbot-dns-dnsmadeeasy-0.34.2-1.fc29, python-certbot-dns-gehirn-0.34.2-1.fc29, python-certbot-dns-google-0.34.2-1.fc29, python-certbot-dns-linode-0.34.2-1.fc29, python-certbot-dns-luadns-0.34.2-1.fc29, python-certbot-dns-nsone-0.34.2-1.fc29, python-certbot-dns-ovh-0.34.2-1.fc29, python-certbot-dns-rfc2136-0.34.2-1.fc29, python-certbot-dns-route53-0.34.2-1.fc29, python-certbot-dns-sakuracloud-0.34.2-1.fc29, python-certbot-nginx-0.34.2-1.fc29, python-configargparse-0.14.0-2.fc29, python-dns-lexicon-3.2.6-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2361aca4af

Comment 17 Fedora Update System 2019-06-01 00:51:59 UTC
certbot-0.34.2-3.fc30, python-acme-0.34.2-1.fc30, python-certbot-apache-0.34.2-1.fc30, python-certbot-dns-cloudflare-0.34.2-1.fc30, python-certbot-dns-cloudxns-0.34.2-1.fc30, python-certbot-dns-digitalocean-0.34.2-1.fc30, python-certbot-dns-dnsimple-0.34.2-1.fc30, python-certbot-dns-dnsmadeeasy-0.34.2-1.fc30, python-certbot-dns-gehirn-0.34.2-1.fc30, python-certbot-dns-google-0.34.2-1.fc30, python-certbot-dns-linode-0.34.2-1.fc30, python-certbot-dns-luadns-0.34.2-1.fc30, python-certbot-dns-nsone-0.34.2-1.fc30, python-certbot-dns-ovh-0.34.2-1.fc30, python-certbot-dns-rfc2136-0.34.2-1.fc30, python-certbot-dns-route53-0.34.2-1.fc30, python-certbot-dns-sakuracloud-0.34.2-1.fc30, python-certbot-nginx-0.34.2-1.fc30, python-dns-lexicon-3.2.6-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2019-06-08 02:12:49 UTC
certbot-0.34.2-3.fc29, python-acme-0.34.2-1.fc29, python-certbot-apache-0.34.2-1.fc29, python-certbot-dns-cloudflare-0.34.2-1.fc29, python-certbot-dns-cloudxns-0.34.2-1.fc29, python-certbot-dns-digitalocean-0.34.2-1.fc29, python-certbot-dns-dnsimple-0.34.2-1.fc29, python-certbot-dns-dnsmadeeasy-0.34.2-1.fc29, python-certbot-dns-gehirn-0.34.2-1.fc29, python-certbot-dns-google-0.34.2-1.fc29, python-certbot-dns-linode-0.34.2-1.fc29, python-certbot-dns-luadns-0.34.2-1.fc29, python-certbot-dns-nsone-0.34.2-1.fc29, python-certbot-dns-ovh-0.34.2-1.fc29, python-certbot-dns-rfc2136-0.34.2-1.fc29, python-certbot-dns-route53-0.34.2-1.fc29, python-certbot-dns-sakuracloud-0.34.2-1.fc29, python-certbot-nginx-0.34.2-1.fc29, python-configargparse-0.14.0-2.fc29, python-dns-lexicon-3.2.6-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.