Bug 2529804

Upstream report: https://github.com/PackageKit/PackageKit/issues/1012

Plugin initialization is done by libdnf5, not PackageKit. So the issue belongs with dnf5. Moving.

> Steps to reproduce:
> 
> 1. Have a Fedora 44 KDE installation with DNF5/libdnf5 5.4.3.0-2.fc44.
> 2. Install the current Fedora updates through KDE Discover.

I don't know what this step does. Does it install something, or not?

> 3. Allow Discover to perform its offline update during reboot.
> 4. The transaction upgrades DNF5/libdnf5 to 5.4.4.0-1.fc44 and installs
>    libdnf5-plugin-systemd-inhibit as a weak dependency.
> 5. After returning to the desktop, Discover reports that the offline update
>   failed with the systemd-inhibit.conf plugin-loading error.

What do you mean with "returning to the desktop"? There is no desktop to be returned to. An off-line update reboots the machine. There is a new desktop running with a new Discover process with a new libdnf5 and new libdnf5 plugins.

I frankly do not know what Discover does and why it reports a failure. That needs to be explained by a Discover maintainer.

Thank you.  I can clarify the reproduction steps.

In step 2, by “Install the updates through KDE Discover” I mean using Discover to download/prepare the available updates and choosing its restart/install-offline-update workflow. The packages are not actually installed at the desktop at that point.

The actual installation then occurs during the offline boot through PackageKit’s packagekit-offline-update.service / pk-offline-update.

By “after returning to the desktop” in step 5, I mean after the offline updater has completed and rebooted the machine back into the normal graphical session. I understand that this is then a new Discover process using the newly installed libraries.

However, the failure itself is recorded before that reboot, during the offline update. The journal from a clean Fedora 44 KDE reproduction shows:

percentage 100%
...
status finished
...
update-packages transaction ... finished with failed
writing failed results
failed to update system: Cannot load libdnf plugin enabled from:
/etc/dnf/libdnf5-plugins/00-systemd-inhibit.conf
...
rebooting

So Discover appears to be displaying the persisted PackageKit offline-update failure result after login, rather than generating a new plugin-loading failure from the new Discover process.

I also reproduced this on a completely fresh stock Fedora 44 KDE installation. That installation started with libdnf5 5.4.1 rather than 5.4.3, so the reproduction does not appear to depend specifically on 5.4.3 → 5.4.4.

On that clean KDE reproduction, the journal also shows the new plugin itself being successfully installed:


op=install sw="libdnf5-plugin-systemd-inhibit-5.4.4.0-1.fc44.x86_64"
...
res=success

followed afterward by the plugin-loading failure above.

For comparison, I also performed fresh Fedora 44 GNOME installations that ended up with the same dnf5, libdnf5, and libdnf5-plugin-systemd-inhibit 5.4.4 packages. Those updates completed through dnf5daemon-server without this failure.

I think the issue is here:

https://github.com/PackageKit/PackageKit/blob/cf89311412a109098d186b238793368907c48d95/backends/dnf5/dnf5-backend-thread.cpp#L565

dnf5_setup_base() runs after trans.run() and creates a new libdnf5::Base. Its Base::setup() rescans enabled plugin configuration and loads plugins from disk.

In this transaction, libdnf5 was upgraded to 5.4.4 and the new systemd-inhibit plugin, including its enabled configuration file, was installed as a weak dependency. The plugin uses IPlugin2_2, an API introduced by libdnf5 5.4.4.

However, PackageKit is still the same process and still has the pre-5.4.4 libdnf5.so.2 loaded in memory. Recreating Base does not reload that shared library. Consequently, the new plugin is loaded against the old in-memory libdnf5 and fails because IPlugin2_2 is unavailable.

Therefore, reinitializing plugins after a transaction that replaced libdnf5 or installed libdnf5 plugins is unsafe unless the backend process is restarted first.

That's a good explanation, but I still don't understand how that can happen during off-line update. At off-line update there is (should be) no PackageKit process running. Only DNF5. I suspect that the reporter actually installs new libdnf5 with the new plugin on-line, not off-line. Or the PackageKit has preinstalled some libdnf5 plugin which is loaded anytime when DNF5 is executed, including an off-line update.

Nevertheless, libdnf5 should not reload plugins exactly for the problem Marek described.

The answer might be `/usr/lib/systemd/system/packagekit-offline-update.service`.
PackageKit seems to have its own pk-offline-update process that handles offline upgrades.

(In reply to Marek Blaha from comment #5)
> I think the issue is here:
> 
> https://github.com/PackageKit/PackageKit/blob/
> cf89311412a109098d186b238793368907c48d95/backends/dnf5/dnf5-backend-thread.
> cpp#L565
> 
> dnf5_setup_base() runs after trans.run() and creates a new libdnf5::Base.
> Its Base::setup() rescans enabled plugin configuration and loads plugins
> from disk.
> 
> In this transaction, libdnf5 was upgraded to 5.4.4 and the new
> systemd-inhibit plugin, including its enabled configuration file, was
> installed as a weak dependency. The plugin uses IPlugin2_2, an API
> introduced by libdnf5 5.4.4.
> 
> However, PackageKit is still the same process and still has the pre-5.4.4
> libdnf5.so.2 loaded in memory. Recreating Base does not reload that shared
> library. Consequently, the new plugin is loaded against the old in-memory
> libdnf5 and fails because IPlugin2_2 is unavailable.
> 
> Therefore, reinitializing plugins after a transaction that replaced libdnf5
> or installed libdnf5 plugins is unsafe unless the backend process is
> restarted first.

Would it be reasonable to just drop that part?

(In reply to Neal Gompa from comment #8)
> > Therefore, reinitializing plugins after a transaction that replaced libdnf5
> > or installed libdnf5 plugins is unsafe unless the backend process is
> > restarted first.
> 
> Would it be reasonable to just drop that part?

Either drop the post-transaction Base reinitialization, or treat failures during the Base refresh separately from failures of the transaction itself.
  
Could PackageKit mark the DNF5 backend as requiring a service restart before accepting another request?
  
I also considered handling this selectively in libdnf5, but there is a chicken-and-egg problem. The failure can occur during dlopen(plugin.so), before libdnf5 can call the plugin’s exported libdnf_plugin_get_api_version() function. Therefore, libdnf5 cannot reliably determine that this particular loading failure means “the plugin requires a newer API.”
  
Supporting that would require advertising the required API version before dlopen(), e.g. "required_api_version=2.2"
  
This could be stored in the plugin configuration or, preferably, in package-owned plugin metadata. An older libdnf5 could then skip only plugins requiring a newer API while keeping other plugin-loading failures fatal. Of course it would address only API version incompatibility and could not make arbitrary in-process plugin reloads after updating libdnf5 generally safe.

> I also considered handling this selectively in libdnf5, but there is a chicken-and-egg problem.

My idea was adding a global variable to libdnf5, allowing loading the plugins only once. Effectively enabling plugins only in the first instance of libdnf5::Base in a process.
Alternatively there could be a new Base::setup(, libdnf5::Base &old_base) which PackageKit would use to copy the iplugins from old_base into this Base instead of loading the plugins again.

(In reply to Marek Blaha from comment #9)
>   
> Could PackageKit mark the DNF5 backend as requiring a service restart before
> accepting another request?
>   

Doing this would make PackageKit extremely slow, because it would do it for every request of any kind. I would really like to avoid that.

At least, I don't think it's possible to only trigger a restart for transaction requests and not require it for everything else.