This is a tracking bug for Change: Dropping of cert.pem file For more details, see: https://fedoraproject.org/wiki/Changes/dropingOfCertPemFile In order to increase the performance of OpenSSL by default using directory-hash format we need to drop the /etc/pki/tls/cert.pem file to prevent it from being loaded by default. This also includes the certificate bundles in /etc/pki/tls/certs/ folder(ca-certificates.crt, ca-bundle.crt). If you encounter a bug related to this Change, please do not comment here. Instead create a new bug and set it to block this bug.
Frantisek, please let me know when the change is approaching. Thanks.
The change is going to be deployed during the fedora mass rebuild starting 23th of June 2025. Make the necessary modifications before this date to ensure smooth transition.
"The change is going to be deployed during the fedora mass rebuild starting 23th of June 2025" *PLEASE* don't do this. This is the worst possible way to land a significant change. Mass rebuild builds bypass Bodhi and automated testing and are tagged directly into 'stable'. This change broke Qt, and it took me all morning to figure out that this was the cause, because the suspect pool was 2000 packages. If this change had been submitted as a regular update, the test would have failed on the update. The update would have been gated, so no other systems or updates would have been affected. As it is, landing it this way has wasted several hours of my time and caused several other unrelated updates to be gated because the test fails on them too.
Also, it seems questionable to go ahead and land this despite six known cases where the migration has not yet happened, beyond the qt6 case which apparently was not detected.
Also, somehow updating to the new ca-certificates in a container and then downgrading to the old one causes the trust system to break entirely. Everything in /etc/pki/ca-trust/extracted is 0 length, and re-running update-ca-trust extract does not fix it.
Hey Adam, > Mass rebuild builds bypass Bodhi and automated testing and are tagged directly into 'stable'. I somehow presumed the mass-rebuilds are tested even more, like the whole system together but not leaving out bodhi and the automated testing... I was terribly wrong at that... What would be the best course of action now? Revert the change and ping someone from release engineering ?
Untagging it would probably be the safest option, yeah. I am testing a patch to fix Qt, which should probably cause it to pass all the openQA update tests, though.
Hmm, unfortunately, the package made it into a compose already. Given my experience with downgrading it in a container, untagging it might actually be a bad idea. I was assuming it hadn't reached a compose yet.
Luckily, I couldn't reproduce the downgrade problem in a clean mock chroot, so there may have been something specific about the container I hit it in.
Please rename (ie move) the wiki page to DroppingOfCertPemFile to fix the typo in the title. Though even that is too terse (short) in my opinion.
Found another affected thing: perl-HTTP-Tiny. Sent https://github.com/Perl-Toolchain-Gang/HTTP-Tiny/pull/18 and https://src.fedoraproject.org/rpms/perl-HTTP-Tiny/pull-request/4 . Go has a directory list with only /etc/ssl/certs and /etc/pki/tls/certs in it. Are we OK considering /etc/pki/tls/certs as a supported hashed directory now or should I send a patch to add /etc/pki/ca-trust/extracted/pem/directory-hash ?
Oh, hmm, perl-HTTP-Tiny already rejected it in https://github.com/Perl-Toolchain-Gang/HTTP-Tiny/pull/17 .
The folder /etc/pki/tls/certs is valid when using openssl, there is no need to use /etc/pki/ca-trust/extracted/pem/directory-hash. the perl-HTTP-tiny got rejected with a valid response, there really isn't need to add it as they use openssl defaults (/etc/pki/tls).
A few points: - Frantisek did rename the Change page to https://fedoraproject.org/wiki/Changes/droppingOfCertPemFile - For Haskell I had to build https://bodhi.fedoraproject.org/updates/FEDORA-2025-78cdd2c8eb (24 packages) https://bodhi.fedoraproject.org/updates/FEDORA-2025-b888a66bb8 (4 packages) and one more build at short notice, and the needed path changes are still not merged or reviewed upstream: so Fedora users using Haskell tooling will still get the broken behavior until that happens. - I still feel this is a risky change as it will cause less tested packages including 3rd party ones to fail after GA. - A safer or rather softer approach would be to have a Phase 1 Change for package migration - and next a Phase 2 Change for the actual removal for a following release (eg F44)
Dear change owner, this is a reminder that your change is required to be 100% code complete by August 26, which is the start of beta freeze. Please provide a status update on your change in the Incomplete Changes Report if you are not able to move your change to 'ON_QA' before this date. If you need to defer your change to the next Fedora release, please let me know and I will reassign this bug and the change page. Thank you kindly.
Jens, Thanks for the updates. Do you find the following: https://fedoraproject.org/wiki/Changes/droppingOfCertPemFile#Contingency_Plan as a sufficient replacement to postponing the change? or would you still rather postpone?
(In reply to Frantisek Krenzelok from comment #16) > Do you find the following: > https://fedoraproject.org/wiki/Changes/ > droppingOfCertPemFile#Contingency_Plan as a sufficient replacement to > postponing the change? I think it helps - I haven't tried it yet though: I guess normal users may not be accustomed to running update-ca-trust. Sorry if I seem to be pushing back - I just worry about any potential backlash or complaints from users after GA when certain of their programs may stop working. (Actually the Contingency Plan section is to explain how a Change could be reverted before release: I think the workaround correctly belongs in https://fedoraproject.org/wiki/Changes/droppingOfCertPemFile#Upgrade/compatibility_impact or possibly also https://fedoraproject.org/wiki/Changes/droppingOfCertPemFile#User_Experience) I still believe at least providing an optional subpackage with the old symlinks would be a clearer/simpler more transparent way to offer backward compatibility at least for a release or two.
> I think it helps - I haven't tried it yet though: > I guess normal users may not be accustomed to running update-ca-trust. In that case, I don't think running `update-ca-trust extract --rhbz...` would be any different than running `dnf install ca-certificates-compat`. > Sorry if I seem to be pushing back - I just worry about any potential > backlash or > complaints from users after GA when certain of their programs may stop > working. Thank you! for doing just that, others raising concerns is the best way to mitigate potential issues. > (Actually the Contingency Plan section is to explain how a Change could be > reverted before release: > I think the workaround correctly belongs in > https://fedoraproject.org/wiki/Changes/droppingOfCertPemFile#Upgrade/ > compatibility_impact > or possibly also > https://fedoraproject.org/wiki/Changes/droppingOfCertPemFile#User_Experience) Thanks, I'll update the change proposal to reflect that. > I still believe at least providing an optional subpackage with the old > symlinks would be > a clearer/simpler more transparent way to offer backward compatibility at > least for a release or two. The problem with subpackages is that, as far as I know, we can't prevent maintainers from adding them as dependencies. I'm concerned that maintainers will simply add this subpackage as a dependency instead of fixing the root cause, which is usually a trivial path change. This would likely result in the compatibility package being installed on nearly every system, which reduces visibility on affected packages and prolongs this mess. Rather than create a subpackage, I would prefer to revert the change in F43 and postpone it to F44. I'm open to this, especially since I mishandled the communication for this change.
Ok, let's postpone to f44, so there is more time for readjustment.
(In reply to Frantisek Krenzelok from comment #18) > Rather than create a subpackage, I would prefer to revert the change in F43 > and postpone it to F44. I'm open to this, especially since I mishandled the > communication for this change. Thank you for the consideration and suggesting this possibility. That should allow more time for more smaller projects to adapt hopefully. :) I wonder if it might be worth keeping and tweaking the current F43 Change into some form of a general "deprecation heads-up" announcement ("Phase 1"). eg It could be renamed to say "Deprecation of cert.pem", encouraging packages to stop using the deprecated symlinks. Anyway it could be discussed with Fesco. Or the deprecation could also be mentioned in the F43 release notes perhaps. (At least quite a good number of packages have adopted the change now so I think the effort has not been completely wasted and it should be easier to complete for F44.:) But I would say please only revert the change in the f43 branch so that it can continue to be tested and adopted in Rawhide.
> Ok, let's postpone to f44, so there is more time for readjustment. So you're going to revert the drop in f43, where it already happened? Please be aware the Beta freeze is on 2025-08-26. If you can't get the revert through by then, it will need a Beta freeze exception.
Yes, I will do that, Thanks for the reminder, it is a small patch, will have it on Monday.
Since the build missed the cut-off I opened bug 2391224 requesting a Beta Freeze Exception.
BTW I think the Change page should be updated to target Fedora 44? https://fedoraproject.org/wiki/Changes/droppingOfCertPemFile