Bug 1832292 - EPEL 8 Modular repo not working if crypto policy set to FUTURE
Summary: EPEL 8 Modular repo not working if crypto policy set to FUTURE
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: distribution
Version: rawhide
Hardware: x86_64
OS: Linux
medium
unspecified
Target Milestone: ---
Assignee: Mohan Boddu
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2086194 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-06 13:35 UTC by Rob
Modified: 2023-12-15 21:05 UTC (History)
14 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2022-07-20 04:13:30 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Fedora Pagure fedora-infrastructure issue 10737 0 None None None 2022-06-01 07:33:54 UTC
Fedora Pagure releng issue 9450 0 None None None 2020-05-12 18:27:46 UTC

Description Rob 2020-05-06 13:35:23 UTC
Description of problem:
When setting crypto policies to FUTURE on CentOS 8.1.1911 or 8.0.1905, the EPEL 8 repo is not accesible during a "dnf update" or "yum update"

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. On a machine that has crypto policies set to default and the EPEL repo enabled, perform a yum/dnf update and observe correct response.
2. Execute "update-crypto-policies --set FUTURE" and reboot machine.
3. Perform a "dnf/yum update" and observe failure of epel-modular repo. A "dnf/yum clean all" may be required to prevent machine solely using locally cached files.

Actual results:
$ sudo  dnf update
CentOS-8 - AppStream                                                                                                  16 kB/s | 4.3 kB     00:00
CentOS-8 - Base                                                                                                       54 kB/s | 3.9 kB     00:00
CentOS-8 - Extras                                                                                                     18 kB/s | 1.5 kB     00:00
CentOS-8 - Plus                                                                                                      8.8 kB/s | 3.0 kB     00:00
Extra Packages for Enterprise Linux Modular 8 - x86_64                                                               0.0  B/s |   0  B     00:00
Failed to download metadata for repo 'epel-modular'
Error: Failed to download metadata for repo 'epel-modular'


Expected results:
$ sudo dnf update
CentOS-8 - AppStream                                                                                                 8.8 kB/s | 4.3 kB     00:00
CentOS-8 - Base                                                                                                       49 kB/s | 3.9 kB     00:00
CentOS-8 - Extras                                                                                                    4.8 kB/s | 1.5 kB     00:00
CentOS-8 - Plus                                                                                                       12 kB/s | 3.0 kB     00:00
Extra Packages for Enterprise Linux Modular 8 - x86_64                                                                56 kB/s |  34 kB     00:00
Extra Packages for Enterprise Linux 8 - x86_64                                                                        28 kB/s |  16 kB     00:00
Dependencies resolved.
Nothing to do.
Complete!

OR any pending packages to be upgraded.

Additional info:

Comment 1 Rob 2020-05-09 22:55:27 UTC
Added 20200509:-

works if .repo file changed to use BaseURL instead of default metalink entry;

ie, default config:

[epel-modular]
name=Extra Packages for Enterprise Linux Modular $releasever - $basearch
#baseurl=https://download.fedoraproject.org/pub/epel/$releasever/Modular/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-modular-$releasever&arch=$basearch&infra=$infra&content=$contentdir
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8
...

doesnt work and yields above error
but modified config:

[epel-modular]
name=Extra Packages for Enterprise Linux Modular $releasever - $basearch
baseurl=https://download.fedoraproject.org/pub/epel/$releasever/Modular/$basearch
#metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-modular-$releasever&arch=$basearch&infra=$infra&content=$contentdir
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8
...

seems to work correctly.

Comment 2 Rob 2020-05-09 22:58:02 UTC
scrap the above comment regarding .repo file I forgot my own notes about a "dnf/yum clean all" to prevent machine from solely using cached files. Apologies. Original issue as described still stands.

Comment 3 Marek Blaha 2020-05-12 12:11:42 UTC
If I run the command in verbose mode I could see more descriptive error message:

# dnf makecache -v
error: Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://mirrors.fedoraproject.org/metalink?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos [SSL certificate problem: CA certificate key too weak] (https://mirrors.fedoraproject.org/metalink?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos).
Extra Packages for Enterprise Linux Modular 8 - x86_64    0.0  B/s |   0  B     00:00    
Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos': Cannot prepare internal mirrorlist: Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://mirrors.fedoraproject.org/metalink?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos [SSL certificate problem: CA certificate key too weak].
Failed to download metadata for repo 'epel-modular'
Error: Failed to download metadata for repo 'epel-modular'

You can reproduce the issue using only curl:
# curl -v 'https://mirrors.fedoraproject.org/metalink?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos'
*   Trying 2620:52:3:1:dead:beef:cafe:fed7...
* TCP_NODELAY set
*   Trying 209.132.181.16...
* TCP_NODELAY set
* Connected to mirrors.fedoraproject.org (209.132.181.16) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, bad certificate (554):
* SSL certificate problem: CA certificate key too weak
* Closing connection 0
curl: (60) SSL certificate problem: CA certificate key too weak
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.


The problem is with SSL certificate of epel-modular repo, not in dnf so 'not a bug' from dnf point of view. But I'm not sure what component reassign the bug to.

Comment 4 Marek Blaha 2020-05-12 12:21:35 UTC
I didn't find suitable component on `Fedora EPEL` product. Reassigning to Fedora / distibution component for certificate for investigation.

Comment 5 Ben Cotton 2020-05-12 18:27:46 UTC
The Fedora Release Engineering team will look into this: https://pagure.io/releng/issue/9450

Comment 6 Rob 2020-05-12 20:01:38 UTC
Thanks to all who have investigated, grateful for confirmation of symptoms and further diagnosis as to cause. Stay safe!

Comment 7 Kevin Fenzi 2020-05-16 22:34:50 UTC
So, we can't fix this. 

The problem is that the FUTURE setting requires rsa certs be > 3072 bits. 

The *.fedoraproject.org cert is 4096, so thats fine. 

However, the intermediate and root certs from digicert (our cert provider), are still 2048, and fail this test. 

We can ask digicert to issue larger bit CA certs, but thats something they would have to choose to do. I can ask via our rep if there are any plans to do this.

Comment 8 Rob 2020-06-10 22:36:31 UTC
I would like to reopen this based upon own research, email convo with closer of ticket and further conversation with Digicert.

I had asked the ticket closer in an offline email about Digicert ECC certificates, was told that was not available. I dispute this as the holder of several ECC 256 certificates issued by Digicert.

According to the redhat notes here 
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening

FUTURE complies RSA and DH parameters if they are 3072 bits long.
This in turn suggests that an ECC-cert would be acceptable with a bit length of 256 as that has an equivalent length of RSA 3072.

Digicert have confirmed to me that they do issue ECC-256 bit certs and I consider this a valid and future-proof fix to the issue at hand.

Please discuss accordingly.

Comment 9 Rob 2020-06-10 22:46:07 UTC
Addition to the above, Digicert have further confirmed that they can issue certs from a 4096 RSA root however I might suggest that a more modern ECC cert would be the preferred future-proof path.

Comment 10 Kevin Fenzi 2020-06-15 23:33:36 UTC
We can definitely look into getting a new cert, but it's not going to be any kind of high priority for us. :)

Comment 11 Fedora Program Management 2021-04-29 17:14:44 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 12 Robert Scheck 2021-05-02 18:25:59 UTC
IMHO this issue still exists: F32 -> Rawhide

Comment 13 Petr Menšík 2021-06-03 17:09:38 UTC
This issue is not related to EPEL in any way. I set FUTURE crypto-policy on f34 and dnf is unusable.

Error: Error downloading packages:
  Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirrors.fedoraproject.org/metalink?repo=fedora-34&arch=x86_64 [SSL certificate problem: CA certificate key too weak]

Comment 14 Ben Cotton 2021-08-10 13:45:37 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 35 development cycle.
Changing version to 35.

Comment 15 Kevin Fenzi 2022-05-20 00:07:38 UTC
*** Bug 2086194 has been marked as a duplicate of this bug. ***

Comment 16 Fhiss 2022-06-14 21:02:29 UTC
Good afternoon. It's been two years. Have you thought about this problem? I think we should move on to EdDSA or ECDSA (DigiCert provide this opportunity), because until now, Fedora can't be updated with FIPS или FUTURE crypto policy.

Please, let's finally solve this annoying problem.

Comment 17 Kevin Fenzi 2022-06-14 23:09:13 UTC
Ah, if it were only so easy. :) 

I got a ECC wildcard cert for our stg env last time hoping it might fix this also. (ie, https://accounts.stg.fedoraproject.org ). 
It does not. Digicert still issues those with it's 2048bit CA. 

I'll try submitting a support request to see if there's any way I can get a cert from the newer CA, but I fear that those are only reserved for the 'extra validation' certs, which require a bunch of information about your business which they verify. The Fedora project is not a legal entity, so we can't provide any of that kind of information. 

As a side note, does anyone know if letsencrypt certs can meet the criteria here?

Comment 18 Alexander Sosedkin 2022-06-15 07:46:33 UTC
Let's Encrypt should support up to 4096 bit RSA key sizes, it's just the ACME client defaults being stuck at 2048.

Comment 19 Fhiss 2022-06-16 19:15:15 UTC
It seems to me that many certificate authorities can issue a certificate for the Fedora Project, in exchange for the opportunity to specify it as partner. I will write, for example, Asseco Data Systems S.A., they just have a suitable Certum EC-384 CA (ECDSA with SHA-384), if you don't mind.

Comment 20 Rob 2022-06-16 21:20:56 UTC
So, 2 years later and we are at a standstill.
And the Linux community wants to advertise fullstream adoption... with appropriate lockdown..

Its not hard to conceive that independent benchmark products will call out potential security issues with delinquent configurations and/or set users up for future support.

Seeing previous comments above its not the client side, it would be the server side that needs fixing per previous notes.
Its not really that difficult in this day and age to apply a modern TLS certificate to a fundamental process.
Quite likely to turn users off particular distro as I don't see the same issue on <doesnt matter other> distros.

Again request remediation of this.

Comment 21 Kevin Fenzi 2022-07-20 04:13:30 UTC
I'd like to thank everyone for their patience. 

We worked with digicert and they were able to enable our account to request certs using their G4 trust root CA instead of the default. 

I've reissued our *.fedoraproject.org cert using that CA and now it's 4096bit the entire way. 

Do note that this is only the wildcard cert, it's not 100% of all certs we use. We have a number of certs using letsencrypt, and those are not using a 4096bit CA yet. 

However, it covers mirrors.fedoraproject.org. It's not going to solve this issue 100% however, as now you can get the metalink from mirrors.fedoraproject.org just fine, but there's not anything saying the mirror you get from that list also has a FUTURE compatible ssl cert chain. :(

Comment 22 jarrod.harch 2023-12-06 23:59:18 UTC
With recent (30-Nov-2023) certificate renewal for *.fedoraproject.org, the new certificate is once again signed by intermediate CA with 2048bit key.

This bug should be re-opened, until certificate can be reissued from 4096 bit CA again.



        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
        Validity
            Not Before: Mar 30 00:00:00 2021 GMT
            Not After : Mar 29 23:59:59 2031 GMT
        Subject: C=US, O=DigiCert Inc, CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)

...

        Issuer: C=US, O=DigiCert Inc, CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
        Validity
            Not Before: Nov 30 00:00:00 2023 GMT
            Not After : Nov 29 23:59:59 2024 GMT
        Subject: C=US, ST=North Carolina, L=Raleigh, O=Red Hat, Inc., CN=*.fedoraproject.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

Comment 23 Kevin Fenzi 2023-12-07 19:38:06 UTC
Sigh. I'll try and see if I can reissue it. 

For those of you setting this, out of curiosity... how are you handling mirrors? I suspect much of our volenteer mirror network doesn't have certs that work with FUTURE either?

Comment 24 Kevin Fenzi 2023-12-07 21:57:01 UTC
ok. I reissued it against the G3 root. Can you check it now?

Comment 25 jarrod.harch 2023-12-08 01:58:31 UTC
Thanks, looking better now, showing ecdsa intermediate ca.

        Serial Number:
            0b:00:e9:2d:4d:6d:73:1f:ca:30:59:c7:cb:1e:18:86
    Signature Algorithm: ecdsa-with-SHA384
        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G3
        Validity
            Not Before: Apr 14 00:00:00 2021 GMT
            Not After : Apr 13 23:59:59 2031 GMT
        Subject: C=US, O=DigiCert Inc, CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:78:a9:9c:75:ae:88:5d:63:a4:ad:5d:86:d8:10:
                    49:d6:af:92:59:63:43:23:85:f4:48:65:30:cd:4a:
                    34:95:a6:0e:3e:d9:7c:08:d7:57:05:28:48:9e:0b:
                    ab:eb:c2:d3:96:9e:ed:45:d2:8b:8a:ce:01:4b:17:
                    43:e1:73:cf:6d:73:48:34:dc:00:46:09:b5:56:54:
                    c9:5f:7a:c7:13:07:d0:6c:18:17:6c:ca:db:c7:0b:
                    26:56:2e:8d:07:f5:67
                ASN1 OID: secp384r1
                NIST CURVE: P-384


Re: mirrors, yes many fail due to same issue. Eventually yum finds one that works.

Comment 26 Rob 2023-12-14 01:10:30 UTC
The gift that keeps on giving...
Thank you to all above for notes on originally reported issue and subsequent work.

To remain constructive do we have now another issue...

..if we now believe, on the gold repo(s), we have correct modern certs applied and it works (to be honest not tested myself as I had to keep the lights on so I may not have FUTURE enabled on my internet-facing machines) is it a different thing we face?

If we are confident that we can now support modern certs on the gold repo, do we now have some onboarding or verification issue of mirrors; ie should there be a minimum specifcation for mirrors to support a specified TLS configuration before they are permitted inclusion in the mirror pool?

To quote the above - "eventually yum [or dnf] finds one that works so some mirrors are compliant. So that suggests that some but not all are compliant hence "minimum standards".

I would hate for this to be closed as fixed unless we can show another related bugfix for standards on mirror onboarding etc.

Hoping this makes sense.

Comment 27 Kevin Fenzi 2023-12-15 20:45:06 UTC
That makes sense, but don't really agree we need to do more here. Consider: 

* the FUTURE setting is... not expected to be fully supported yet. If it was, it's settings would be merged with DEFAULT. 

* People setting this are likely a very small percentage of all users. I don't have any hard data of course, but consider that all the following sites don't work (at least here, setting FUTURE in Fedora rawhide): redhat.com, cnn.com, google.com, etc

* Our mirror network are volunteering to host our content. We can surely urge them to update when this is closer to becoming DEFAULT, but dropping mirrors who don't support this right now seems counterproductive and would likely result in a large reduction in mirrors for not much gain, IMHO. 

Perhaps crypto-policy folks could chime in here?

Comment 28 Alexander Sosedkin 2023-12-15 21:05:49 UTC
> We can surely urge them to update when this is closer to becoming DEFAULT, but dropping mirrors who don't support this right now seems counterproductive and would likely result in a large reduction in mirrors for not much gain, IMHO. 

> Perhaps crypto-policy folks could chime in here?

There's no immediate plan to bump DEFAULT to 3072 keys, that'd be rather unreasonable. Even if Let's Encrypt switches to ECDSA by default overnight, we'd still be ~ a year away from 3072 RSA mandated in DEFAULT.
You could anticipate the next crypto-policies tightening proposal to be something like the rejected https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3 and not much more ambitious than that.


Note You need to log in before you can comment on or make changes to this bug.