Bug 2372978 - packagekitd shows "failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-XX-secondary" errors every second and consumes a lot of CPU if google-chrome repository is enabled
Summary: packagekitd shows "failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fed...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libdnf
Version: 43
Hardware: Unspecified
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Petr Pisar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://discussion.fedoraproject.org/...
: 2361189 2364932 2373014 2396626 2397087 (view as bug list)
Depends On:
Blocks: F43FinalBlocker, FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2025-06-16 08:06 UTC by Vít Ondruch
Modified: 2025-10-04 00:37 UTC (History)
36 users (show)

Fixed In Version: libdnf-0.74.0-9.fc43
Clone Of:
Environment:
Last Closed: 2025-09-28 01:03:03 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
strace.log (570.77 KB, text/plain)
2025-06-16 08:07 UTC, Vít Ondruch
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github rpm-software-management libdnf pull 1724 0 None open Stop importing subkeys to RPM >= 4.19.0 2025-09-18 16:46:07 UTC

Description Vít Ondruch 2025-06-16 08:06:16 UTC
Gnome Software / packagekitd consumes a lot of CPU. It seems that packagekitd gets stuck in some infinte loop, or if nothing else, `systemctl restart packagekit.service` fixes the issue for a day. Please see attached `strace` if that reveals something.

Reproducible: Always

Comment 1 Vít Ondruch 2025-06-16 08:07:09 UTC
Created attachment 2094130 [details]
strace.log

A few seconds of `$ sudo strace -p 1862782 -o strace.log`

Comment 2 Vít Ondruch 2025-06-16 08:09:34 UTC
Actually, there are also associated repetitive journal entries:

~~~
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-primary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-secondary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-7-primary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-primary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-primary-original to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-primary to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-primary-original to rpmdb
čen 16 10:08:20 fedora packagekitd[1862782]: failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-secondary to rpmdb
čen 16 10:08:20 fedora PackageKit[1862782]: update-packages transaction /347982_aecaddab from uid 1000 finished with need-untrusted after 354ms
čen 16 10:08:20 fedora PackageKit[1862782]: new update-packages transaction /347983_dddadabb scheduled from uid 1000
~~~

Comment 3 Vít Ondruch 2025-06-16 08:13:49 UTC
This might or might not be related to bug 2364932

Comment 4 Michael Schwendt 2025-09-12 09:04:13 UTC
I posted about this to devel@ list on May 1st asking whether anyone has seen it, too.

Yesterday I upgraded from F42 to F43 Beta once again, and the upgraded installation shows the same symptoms on reboot as if it's 100% reproducible. PackageKit just doesn't stop.

$ sudo journalctl -xb|grep -i packagekit|grep "failed to add
subkeys"|wc -l
336126
$ rpm -qf /usr/libexec/packagekitd
PackageKit-1.2.8-13.fc43.x86_64

Comment 5 Fedora Blocker Bugs Application 2025-09-14 17:45:03 UTC
Proposed as a Blocker for 43-final by Fedora user mschwendt using the blocker tracking app because:

 This bug has plagued Fedora 43 devel for months. It would be bad, if it made it into the final release.

Comment 6 Lukas Ruzicka 2025-09-15 19:06:30 UTC
Discussed at the 2025-09-15 (blocker / freeze exception) review meeting:

We weren't able to reproduce this on a clean F43 Beta install in a quick test, so we want to punt to try to get more detail on when and why this issue happens.

https://meetbot-raw.fedoraproject.org//blocker-review_matrix_fedoraproject-org/2025-09-15/f43-blocker-review.2025-09-15-16.00.txt

Comment 7 Vít Ondruch 2025-09-16 09:29:34 UTC
I have not provided too much details about versions or what not, but I don't suffer this issue anymore. Looking into `dnf history`, I believe I have observed the issue starting from 2025-06-02 with update to PackageKit-0:1.2.8-9.fc42.x86_64 / gnome-software-0:48.1-1.fc43.x86_64 and I have likely stopped to observe the issue after this manual update:

~~~
$ sudo dnf history info 122
Transaction ID : 122
Begin time     : 2025-06-29 09:30:10
Begin rpmdb    : b307b12b1e8c54e3d81af41f39a5c2f97077e400981068ca83f4478e61408e1d
End time       : 2025-06-29 09:30:58
End rpmdb      : fe84255d74615c340a26790dfad8317cecd8b8d89f70fa8012847fdfd4452b03
User           : 1000 Vít Ondruch <vondruch>
Status         : Ok
Releasever     : rawhide
Description    : dnf update https://kojipkgs.fedoraproject.org//packages/gnome-software/49~alpha/1.fc43/data/signed/31645531/x86_64/gnome-software-49~alpha-1.fc43.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/gnome-software/49~alpha/1.fc43/data/signed/31645531/x86_64/gnome-software-fedora-langpacks-49~alpha-1.fc43.x86_64.rpm
Comment        : 
Packages altered:
  Action   Package                                                        Reason          Repository
  Install  dnf5daemon-server-0:5.2.14.0-3.fc43.x86_64                     Dependency      rawhide
  Install  dnf5daemon-server-polkit-0:5.2.14.0-3.fc43.noarch              Dependency      rawhide
  Install  libdnf5-plugin-appstream-0:5.2.14.0-3.fc43.x86_64              Dependency      rawhide
  Install  rpm-plugin-dbus-announce-0:5.99.90-6.fc43.x86_64               Dependency      rawhide
  Install  fedpkg-completion-0:1.46-4.fc43.noarch                         Weak Dependency rawhide
  Upgrade  gnome-software-0:49~alpha-1.fc43.x86_64                        Group           @commandline
  Upgrade  libdnf5-0:5.2.14.0-3.fc43.x86_64                               Dependency      rawhide
  Upgrade  libdnf5-cli-0:5.2.14.0-3.fc43.x86_64                           Dependency      rawhide
  Upgrade  rpm-libs-0:5.99.90-6.fc43.x86_64                               Dependency      rawhide
  Upgrade  gnome-software-fedora-langpacks-0:49~alpha-1.fc43.x86_64       Weak Dependency @commandline
  Upgrade  librepo-0:1.20.0-1.fc43.x86_64                                 Dependency      rawhide
  Upgrade  rpm-0:5.99.90-6.fc43.x86_64                                    Group           rawhide
  Upgrade  libdnf5-plugin-expired-pgp-keys-0:5.2.14.0-3.fc43.x86_64       Weak Dependency rawhide
  Upgrade  dnf5-plugins-0:5.2.14.0-3.fc43.x86_64                          Group           rawhide
  Upgrade  dnf5-0:5.2.14.0-3.fc43.x86_64                                  Group           rawhide

... snip ...

$ sudo dnf history info 122 | grep software
Description    : dnf update https://kojipkgs.fedoraproject.org//packages/gnome-software/49~alpha/1.fc43/data/signed/31645531/x86_64/gnome-software-49~alpha-1.fc43.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/gnome-software/49~alpha/1.fc43/data/signed/31645531/x86_64/gnome-software-fedora-langpacks-49~alpha-1.fc43.x86_64.rpm
  Upgrade  gnome-software-0:49~alpha-1.fc43.x86_64                        Group           @commandline
  Upgrade  gnome-software-fedora-langpacks-0:49~alpha-1.fc43.x86_64       Weak Dependency @commandline
  Replaced gnome-software-0:48.2-1.fc42.x86_64                            Group           @System
  Replaced gnome-software-fedora-langpacks-0:48.2-1.fc42.x86_64           Weak Dependency @System
~~~

If I remember correctly, at that moment, G-S switched from PackageKit to DNF backend. But I might be wrong.

I can imagine that if the PacakageKit backend is still in use for some reason, the issue was not fixed.

Comment 8 Michael Schwendt 2025-09-16 10:37:27 UTC
(In reply to Lukas Ruzicka from comment #6)
> 
> We weren't able to reproduce this on a clean F43 Beta install in a quick
> test,

Dunno whether a clean install is affected, too. Might be that I will try a clean installation of F43 Beta once available.


> Upgrade  gnome-software-0:49~alpha-1.fc43.x86_64                        Group           @commandline

That's a rather old one. Here, nothing has changed with current gnome-software. As it is another upgraded F42, PackageKit remains active/enabled, and if it starts working on something after a (re)boot, it just doesn't stop.

$ rpm -qa|grep gnome-software
gnome-software-49.0-1.fc43.x86_64
gnome-software-fedora-langpacks-49.0-1.fc43.x86_64

Comment 9 Chris Murphy 2025-09-16 14:27:17 UTC
Possibly related: I notice on upgraded systems that /var/cache/dnf still exists, and with plenty of crusty bits. For some reason this didn't get removed with the dnf5 upgrade. dnf5 uses /var/cache/libdnf5.

Comment 11 Adam Williamson 2025-09-17 22:58:48 UTC
OK, so after playing about a bit I think I have a fairly consistent reproducer for this.

1. Do a clean Fedora 43 Beta Workstation install
2. Boot the installed system, and during gnome-initial-setup, choose to enable third-party repositories
3. Complete g-i-s, and from the desktop, run GNOME Software
4. Try and install anything

You'll get a warning dialog "Install Unsigned Software?", with an error message:

failed to add subkeys for /var/cache/PackageKit/43/metadata/google-chrome-43-x86_64/linux_signing_key.pub to rpmdb

If you click "Install", it will show Authentication Required and prompt you for your password. Entering the password just brings back the same Install Unsigned Software? dialog.

If you now check the journal, you'll see the once-per-second "failed to add subkeys for /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-XX-secondary to rpmdb" message flood.

The only way to recover seems to be to disable the google-chrome repository *and then wipe the /var/cache/PackageKit/43/metadata/google-chrome-43-x86_64/ directory*. If you don't also wipe the cache directory, the broken state persists.

If anyone is able to reproduce this *without* ever enabling the google-chrome repo, please speak up...for now, it seems to be the trigger.

Comment 12 Adam Williamson 2025-09-17 23:01:43 UTC
Note, for me, this blocks installing *anything* via GNOME Software (and presumably other PackageKit-based routes), once you enable the google-chrome repo, until you disable it and wipe the cache. *As well as* the resource usage from the constant errors. That makes it a pretty bad bug.

Comment 13 Adam Williamson 2025-09-17 23:46:45 UTC
So I've poked around a bit here and I have a suspicion, at least.

The error message here comes from libdnf: https://github.com/rpm-software-management/libdnf/blob/91a0bf9aada36a722855051526f012e0b5ab1af9/libdnf/dnf-keyring.cpp#L142

That code hasn't changed in years, so it's not the problem here. *But*, the code it calls in RPM - rpmKeyringAddKey() and rpmGetSubkeys() - *did* change quite a lot in RPM 6, which we have in F43 (well, pre-releases of it). There's a couple of PRs in RPM 6 with multiple related commits:

https://github.com/rpm-software-management/rpm/pull/3083
https://github.com/rpm-software-management/rpm/pull/3397 (the "Split keystore handling to a separate source" commit)
https://github.com/rpm-software-management/rpm/pull/3401
https://github.com/rpm-software-management/rpm/pull/3403

In particular, c0e43f81f "Handle subkeys in rpmKeyringModify" from #3403 looks like it makes rpmKeyringModify() add subkeys when adding a main key, which is...exactly what libdnf is also trying to do here, I think. Note that rpmKeyringAddKey, in RPM 6, is just a light wrapper around rpmKeyringModify():
```
int rpmKeyringAddKey(rpmKeyring keyring, rpmPubkey key)
{
    return rpmKeyringModify(keyring, key, RPMKEYRING_ADD);
}
```
so when libdnf does `rc = rpmKeyringAddKey(keyring, pubkey);`, that's going to call `rpmKeyringModify()`, and I *think* that's going to go ahead and try to import the subkeys before libdnf then tries to do it itself.

Also, libdnf tries to import the subkeys by calling `rpmKeyringAddKey()` - i.e. this new `rpmKeyringModify()` - on them, and it's possible it just...doesn't work with subkeys, while the previous `rpmKeyringAddKey` did. The old `rpmKeyringAddKey()` was just this:

int rpmKeyringAddKey(rpmKeyring keyring, rpmPubkey key)
{
    int rc = 1; /* assume already seen key */
    if (keyring == NULL || key == NULL)
	return -1;

    /* check if we already have this key, but always wrlock for simplicity */
    pthread_rwlock_wrlock(&keyring->lock);
    if (!rpmKeyringFindKeyid(keyring, key)) {
	keyring->keys = xrealloc(keyring->keys,
				 (keyring->numkeys + 1) * sizeof(rpmPubkey));
	keyring->keys[keyring->numkeys] = rpmPubkeyLink(key);
	keyring->numkeys++;
	qsort(keyring->keys, keyring->numkeys, sizeof(*keyring->keys),
		keyidcmp);
	rc = 0;
    }
    pthread_rwlock_unlock(&keyring->lock);

    return rc;
}

the new `rpmKeyringModify` is obviously pretty different from that. My knowledge of C, rpm internals and GPG keys are all insufficient to make a guess as to whether or why the new one might fail if called on a subkey, but it sure seems *possible*.

CCing the RPM folks who might be able to help with this.

Comment 14 Adam Williamson 2025-09-17 23:49:45 UTC
oh, sorry, one more thing from my investigations: it looks like the files we see the error messages for are all the files which contain subkeys. Among the Fedora key files, the 'secondary' files all have subkeys, but they stop existing after RPM-GPG-KEY-fedora-26-secondary ; we get errors for all the ones that exist. The 'primary' files only have subkeys up to RPM-GPG-KEY-fedora-10-primary, and that's indeed the highest number we get the error for. And of course, the google key has subkeys.

Comment 15 Adam Williamson 2025-09-17 23:53:49 UTC
Also CCing ppisar and amatej for libdnf. As I said libdnf didn't change here so it's not libdnf's "fault", but given that it looks like RPM 6 imports subkeys automatically, we might want to make libdnf *not* try and do it too, at least when building for / running against RPM 6? Not sure if there's a good way to check that.

Comment 16 Petr Pisar 2025-09-18 10:10:14 UTC
I'm debugging a similar issue in microdnf and so far got to a conclusion that pgpPubkeyMerge() of rpm-sequoia-1.9.0-2.fc43.x86_64 fails to merge some subkeys.

If packagekit uses libdnf's dnf_keyring_add_public_keys() (or indirectly via dnf_transaction_import_keys()), then the messages about failed import of historical keys from /etc/pki/rpm-gpg are not errors. Those are warnings.

> given that it looks like RPM 6 imports subkeys automatically, we might want to make libdnf *not* try and do it too

This has to be confirmed by RPM maintainers. libdnf cannot unilaterally stop processing subkeys without RPM telling that it is intended change.

Moreover, iterating over subkeys has a security aspect because it's a subkey what signs a package and it's a subkey's ID which is presented in a list if imported keys in the RPM database. I worry that working around this sequoia's bug on libdnf level would not be the right approach.

Comment 17 Michal Domonkos 2025-09-18 10:52:11 UTC
This seems to be related to the following two changes in RPM:

1) rpmKeyringAddKey() is now handling subkeys as well (https://github.com/rpm-software-management/rpm/issues/3350)
2) rpmKeyringAddKey() is now automatically merging keys on addition (https://github.com/rpm-software-management/rpm/pull/3526)

Meanwhile, libdnf is still handling the addition of subkeys by itself (https://github.com/rpm-software-management/libdnf/blob/91a0bf9aada36a722855051526f012e0b5ab1af9/libdnf/dnf-keyring.cpp#L142) and thus, when it tries to import a subkey via rpmKeyringAddKey() (which has been imported in the previous rpmKeyringAddKey() call), RPM attempts to merge it with the existing key, and fails (returns -1). At least that's what seems to be happening here.

Subkey handling was never documented for the rpmKeyringAddKey() API function but it is a contract change, in a way, one that libdnf (and any other users) should probably be adapted to.

@pmatilai, any thoughts?

Comment 18 Michal Domonkos 2025-09-18 11:07:17 UTC
Just to elaborate on this a bit:

(In reply to Michal Domonkos from comment #17)
> [...] RPM attempts to merge it with the existing key, and fails (returns -1).

What I meant was, passing a key (certificate) that's a subkey to rpmKeyringAddKey() (which passes it to pgpPubkeyMerge()) is not something that the (latter) function expects, which is why it most likely just fails. But then, I'm not an expert in PGP and/or the sequoia API.

Comment 19 Panu Matilainen 2025-09-18 13:37:07 UTC
Yes, it's very much intentional that rpm now automatically handles subkeys, and mostly hides their existence as we were adviced by the Sequoia folks: they are an implementation detail that should not be exposed to users where avoidable, only main key IDs are shown through eg rpmkeys --list.

API users could use eg the presence of rpmKeyringModify() function as a clue that subkeys are handled automatically.

I guess ideally rpm would just note that this subkey is already there and ignore instead of returning an error for what is really a no-op.

Comment 20 Petr Pisar 2025-09-18 14:50:49 UTC
Thanks for the confirmation. These changes exist in RPM since rpm-4.19.0-alpha2. I will update libdnf not to pass individual subkeys to rpmKeyringAddKey() on recent RPM.

I tested the current libdnf also in Fedora 42 (rpm-libs-4.20.1-1.fc42.x86_64, rpm-sequoia-1.7.0-5.fc42.x86_64) and there it does not report any error. It even does not report any error after upgrading rpm-sequoia to 1.9.0-1.fc43.x86_64. However, it starts reporting an error after upgrading rpm to 5.99.90-2.fc43.x86_64.

Comment 21 Adam Williamson 2025-09-18 15:23:21 UTC
Thanks for the fast confirmation, folks.

As the planned libdnf change should resolve this, I'll re-assign the bug to libdnf. I'll file an RPM issue upstream about not erroring out when `rpmKeyringModify` is called on a subkey.

Comment 22 Michal Domonkos 2025-09-18 16:07:43 UTC
(In reply to Adam Williamson from comment #21)
> Thanks for the fast confirmation, folks.

You're welcome! And thanks for the initial analysis (that helped a lot).

> As the planned libdnf change should resolve this, I'll re-assign the bug to
> libdnf. I'll file an RPM issue upstream about not erroring out when
> `rpmKeyringModify` is called on a subkey.

Ack, sounds good to me.

Comment 23 Adam Williamson 2025-09-18 17:25:50 UTC
RPM issue: https://github.com/rpm-software-management/rpm/issues/3954

Setting POST as there's a PR linked.

Comment 24 Adam Williamson 2025-09-18 19:11:51 UTC
Scratch build for testing: https://koji.fedoraproject.org/koji/taskinfo?taskID=137233930 . Looks to be good here in initial testing.

Comment 25 Adam Williamson 2025-09-18 19:28:30 UTC
Yeah. I tested two ways. First I tested updating my 'broken' VM with the new libdnf. After restarting packagekit and gnome-software, I can install packages with the google-chrome repo enabled, and the journal messages have gone away. Also, I tested my fresh-start reproducer using a live image built with the new libdnf - install, enable third-party repos, try and install a package with GNOME Software. It worked fine, and no log messages about subkeys. Finally I tested installing Chrome, in case that got broken because we didn't import the subkey at all, or anything. Weirdly I can't find the packaged Chrome in GNOME Software (only flathub's flatpak Chrome), but I can do `pkcon install google-chrome-stable` and it works.

So, the fix looks good here.

Comment 26 Adam Williamson 2025-09-18 23:12:58 UTC
+4 in https://pagure.io/fedora-qa/blocker-review/issue/1927 , marking accepted Final blocker: to justify it, it's a violation of "The installed system must be able appropriately to install, remove, and update software with the default tool for the relevant software type in all release-blocking desktops (e.g. default graphical package manager). This includes downloading of packages to be installed/updated" in the case that you enable third-party repos, which is very common.

Comment 27 Panu Matilainen 2025-09-19 06:09:58 UTC
> Thanks for the confirmation. These changes exist in RPM since rpm-4.19.0-alpha2.

No! These automatic subkey handling and key merges are rpm 6.0 only!

If there's a behavior difference in rpm-4.19.0-alpha2, that's something entirely different.

Comment 28 Adam Williamson 2025-09-19 06:19:54 UTC
There's no behaviour difference in or since rpm-4.19.0-alpha2 that I'm aware of. This is definitely apparent only in Fedora 43, i.e., it appeared between 4.20.1 and 5.99. I'm not sure how Petr got confused.

Comment 29 Panu Matilainen 2025-09-19 07:10:42 UTC
> I'm debugging a similar issue in microdnf and so far got to a conclusion that pgpPubkeyMerge() of rpm-sequoia-1.9.0-2.fc43.x86_64 fails to merge some subkeys.
>
> If packagekit uses libdnf's dnf_keyring_add_public_keys() (or indirectly via dnf_transaction_import_keys()), then the messages about failed import of historical keys from /etc/pki/rpm-gpg are not errors. Those are warnings.

I missed this little piece. Looking at comment #2, the failing keys are indeed historic and would have things like SHA1 based keys that are no longer considered acceptable. 

Why on earth are those ancient keys (Fedora 9? For deitys sake!) being imported? This makes no sense.

Comment 30 Milan Crha 2025-09-19 07:16:23 UTC
If I recall correctly, PackageKit imports simply everything in the directory, not thinking at all what makes and what not any sense to import.

Comment 31 Panu Matilainen 2025-09-19 07:27:41 UTC
So it would appear, but why? Also, why are we shipping keys to distros that EOL'ed years, many of them well over 10 years obsolete?

Comment 32 Panu Matilainen 2025-09-19 07:29:16 UTC
I mean, in rpm, import implies trust, and we really SHOULD NOT trust keys for long gone distros, not by default anyhow.

Comment 33 Milan Crha 2025-09-19 07:40:13 UTC
I do not know why it does that, their devs might know better. I guess, and only guess, the PackageKit expects the keys in the directory being up-to-date and accurate and generally considered trusted by the user/admin (kinda silly expectation, I know), then it imports everything and is happy when a package from any repo is installed, that the RPM can verify its signature, because the keys are there.

In contrast, the keys imported by the dnf(5) itself are imported persistently and on question, not every time the daemon/process runs or installs anything.

And why those ancient keys are still available? That's a question for the `fedora-gpg-keys` maintainers.

Comment 34 Petr Pisar 2025-09-19 07:45:07 UTC
(In reply to Adam Williamson from comment #28)
> There's no behaviour difference in or since rpm-4.19.0-alpha2 that I'm aware
> of. This is definitely apparent only in Fedora 43, i.e., it appeared between
> 4.20.1 and 5.99. I'm not sure how Petr got confused.

I got confused by non-linear history of RPM git repository. I will correct the minimal version.

Comment 35 Petr Pisar 2025-09-19 08:34:49 UTC
(In reply to Panu Matilainen from comment #29)
> I missed this little piece. Looking at comment #2, the failing keys are
> indeed historic and would have things like SHA1 based keys that are no
> longer considered acceptable. 
> 
I will add that the SHA1 problem is independent from problem with merging subkeys. I experienced the merging problem even with recently created subkeys signed with SHA-512.

> Why on earth are those ancient keys (Fedora 9? For deitys sake!) being
> imported? This makes no sense.

The code is in libdnf almost from the very beginning without any explanation (commit 47f81dd4b8093b27c7476e5dbecf6df4084e878e by Richard Hughes). I guess some people want to keep installing historical packages, e.g. when Fedora removes them from a distribution. Or people want packagekit without any questions. DNF5 actually has recent bug reports from people like that. For some reason packagekit uses this feature. Contrary neither DNF4 or DNF5 do that.

Comment 36 Panu Matilainen 2025-09-19 09:24:33 UTC
> I will add that the SHA1 problem is independent from problem with merging subkeys. I experienced the merging problem even with recently created subkeys signed with SHA-512.

Ack.

> The code is in libdnf almost from the very beginning without any explanation (commit 47f81dd4b8093b27c7476e5dbecf6df4084e878e by Richard Hughes). I guess some people want to keep installing historical packages, e.g. when Fedora removes them from a distribution. Or people want packagekit without any questions. DNF5 actually has recent bug reports from people like that. For some reason packagekit uses this feature. Contrary neither DNF4 or DNF5 do that.

The people who want to install from old distros can do so by manually importing those keys, it should NOT be a default action. Anyway, this is kinda outside the scope here, we should probably open new ticket(s) for the issue of historic keys.

Comment 37 Adam Williamson 2025-09-19 15:47:26 UTC
Yeah. Practically speaking, per my testing, the current libdnf patch does seem to fix the observed issues. And I can install Chrome at least, which implies the subkey import does actually happen (if anyone has a more reliable way to check it's working, let me know).

Comment 38 Adam Williamson 2025-09-19 15:52:19 UTC
Oh, one reason the old keys are present is likely that, in theory, old *packages* can remain in Fedora more or less indefinitely. It's not *common* due to mass rebuilds and so on, but there's no mechanism that really 100% guarantees something like "there will only be packages from maximum N-2 in N" or anything like that. Current Rawhide has several fc40 packages in it, all Go for some reason:

[adamwill@compose-x86-01 Packages][PROD-RDU3]$ find -name *fc40*
./c/compat-golang-github-linkedin-goavro-2-devel-2.10.0-10.fc40.noarch.rpm
./c/compat-golang-github-seborama-govcr-devel-4.5.0-3.fc40.noarch.rpm
./c/compat-golang-github-transip-gotransip-6-devel-6.5.0-9.fc40.noarch.rpm
./g/golang-github-fogleman-gg-devel-1.3.0-15.20200726gitad4d1ea.fc40.noarch.rpm
./g/golang-github-linkedin-goavro-devel-2.10.0-10.fc40.noarch.rpm
./g/golang-github-transip-gotransip-devel-6.5.0-9.fc40.noarch.rpm
./g/golang-github-grpc-ecosystem-prometheus-devel-1.2.0-16.20200727git9abf3eb.fc40.noarch.rpm
./g/golang-github-segmentio-encoding-devel-0.3.6-4.fc40.noarch.rpm
./g/golang-github-wi2l-jettison-devel-0.7.4-3.fc40.noarch.rpm
./g/golang-github-deepmap-oapi-codegen-1.13.0-1.fc40.x86_64.rpm
./g/golang-github-yvasiyarov-metrics-devel-0-0.16.20190626gitc25f46c.fc40.noarch.rpm
./g/golang-github-robertkrimen-otto-devel-0-0.21.20210110gitef014fd.fc40.noarch.rpm
./g/golang-gopkg-seborama-govcr-4-devel-4.5.0-3.fc40.noarch.rpm
./g/golang-github-xlab-treeprint-devel-1.1.0-7.fc40.noarch.rpm
./g/golang-honnef-tools-devel-2023.1.3-3.20230802git0e3cc29.fc40.noarch.rpm
./g/golang-k8s-apiserver-devel-1.22.0-12.fc40.noarch.rpm
./g/golang-k8s-component-base-devel-1.22.0-9.fc40.noarch.rpm
./g/golang-mongodb-mongo-driver-devel-1.4.5-12.fc40.noarch.rpm
./g/golang-github-deepmap-oapi-codegen-devel-1.13.0-1.fc40.noarch.rpm
./g/golang-github-xaionaro-metrics-devel-0-0.8.20210706git68050b3.fc40.noarch.rpm
./g/golang-honnef-tools-2023.1.3-3.20230802git0e3cc29.fc40.x86_64.rpm
./g/golang-mongodb-mongo-driver-1.4.5-12.fc40.x86_64.rpm

Comment 39 Adam Williamson 2025-09-19 15:54:56 UTC
Can we get a build with the fix for Rawhide and F43 soon? It'd be great to fix this ASAP as people are picking up the F43 Beta and it's pretty easy to encounter this, and it looks pretty bad if you do.

I can backport the fix if need be, just don't want to step on any toes.

Comment 40 Kevin Fenzi 2025-09-20 21:43:39 UTC
Late to this party, but just fyi... 

(In reply to Adam Williamson from comment #38)
> Oh, one reason the old keys are present is likely that, in theory, old
> *packages* can remain in Fedora more or less indefinitely. It's not *common*
> due to mass rebuilds and so on, but there's no mechanism that really 100%
> guarantees something like "there will only be packages from maximum N-2 in
> N" or anything like that. Current Rawhide has several fc40 packages in it,
> all Go for some reason:

While we still have packages that didn't for whatever rebuild in a mass rebuild, they are all signed by the current release key.
ie, a f40 package in f43 repos is in fact signed by the f43 key since we resign everything when we branch.

So, the only reason I can think of we keep the old keys around is in case someone wanted for some historical reason get say a f10 package and wanted to verify it was signed correctly with the old f10 key.

Comment 41 Adam Williamson 2025-09-21 06:22:30 UTC
*** Bug 2364932 has been marked as a duplicate of this bug. ***

Comment 42 Adam Williamson 2025-09-21 06:23:01 UTC
*** Bug 2397087 has been marked as a duplicate of this bug. ***

Comment 43 Adam Williamson 2025-09-21 06:24:40 UTC
*** Bug 2361189 has been marked as a duplicate of this bug. ***

Comment 44 Adam Williamson 2025-09-21 06:25:12 UTC
*** Bug 2396626 has been marked as a duplicate of this bug. ***

Comment 45 Michael Schwendt 2025-09-22 23:15:37 UTC
(In reply to Adam Williamson from comment #24)
> Scratch build for testing:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=137233930 . Looks to be
> good here in initial testing.

Been running with this for a few days, and so far it works, and the high load symptoms haven't reoccurred. Also verified that GNOME Software can install Google Chrome e.g.

Comment 46 Fedora Update System 2025-09-23 01:32:13 UTC
FEDORA-2025-3ebabcddab (libdnf-0.74.0-8.fc43) has been submitted as an update to Fedora 43.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-3ebabcddab

Comment 47 Fedora Update System 2025-09-24 15:50:39 UTC
FEDORA-2025-3ebabcddab has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-3ebabcddab`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-3ebabcddab

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 48 Kamil Páral 2025-09-25 12:39:06 UTC
(In reply to Fedora Update System from comment #46)
> FEDORA-2025-3ebabcddab (libdnf-0.74.0-8.fc43) has been submitted as an
> update to Fedora 43.
> https://bodhi.fedoraproject.org/updates/FEDORA-2025-3ebabcddab

For some reason, I can no longer reproduce the journal spam (I was able to before). But I can confirm that with this update, gnome-software can now correctly install packages and isn't stuck in the "trust key?" loop.

Comment 49 Fedora Update System 2025-09-26 01:29:14 UTC
FEDORA-2025-3ebabcddab has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-3ebabcddab`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-3ebabcddab

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 50 Colin Walters 2025-09-26 17:26:51 UTC
> The code is in libdnf almost from the very beginning without any explanation (commit 47f81dd4b8093b27c7476e5dbecf6df4084e878e by Richard Hughes).

See https://github.com/rpm-software-management/libdnf/issues/43 - I did some research on this then (man it's been 10 years, amazing).

The biggest thing is that the "do you trust this key" imposes an interactivity requirement and I think the PK folks didn't want to do that. Also of course the "do you trust <sha checksum>" is generally just a broken UI.

Comment 51 Geraldo Simião 2025-09-27 22:19:01 UTC
(In reply to Fedora Update System from comment #49)
> FEDORA-2025-3ebabcddab has been pushed to the Fedora 43 testing repository.
> Soon you'll be able to install the update with the following command:
> `sudo dnf upgrade --enablerepo=updates-testing --refresh
> --advisory=FEDORA-2025-3ebabcddab`
> You can provide feedback for this update here:
> https://bodhi.fedoraproject.org/updates/FEDORA-2025-3ebabcddab
> 
> See also https://fedoraproject.org/wiki/QA:Updates_Testing for more
> information on how to test updates.


With this upgrade Discover managed to install new packages without the error messages.

Comment 52 Fedora Update System 2025-09-28 01:03:03 UTC
FEDORA-2025-3ebabcddab (libdnf-0.74.0-9.fc43) has been pushed to the Fedora 43 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 53 Adam Williamson 2025-10-04 00:37:50 UTC
*** Bug 2373014 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.