Bug 1674504

Summary: python3-dnf-plugin-system-upgrade causes pk offline-updates to fail
Product: [Fedora] Fedora Reporter: Hans de Goede <hdegoede>
Component: dnf-plugins-extrasAssignee: rpm-software-management
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: high    
Version: 29CC: dmach, jkadlcik, jmracek, mblaha, pkratoch, rpm-software-management, vmukhame
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-04 06:51:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hans de Goede 2019-02-11 13:49:35 UTC
When trying to use offline-updates from within gnome-software, rather then doing "dnf update" as I usually do, I hit the following backtrace in dnf-system-upgrade.service:

feb 11 12:35:01 shalem.localdomain dnf[880]: Traceback (most recent call last):
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/bin/dnf", line 58, in <module>
feb 11 12:35:01 shalem.localdomain dnf[880]:     main.user_main(sys.argv[1:], exit_code=True)
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 179, in user_main
feb 11 12:35:01 shalem.localdomain dnf[880]:     errcode = main(args)
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main
feb 11 12:35:01 shalem.localdomain dnf[880]:     return _main(base, args, cli_class, option_parser_class)
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 95, in _main
feb 11 12:35:01 shalem.localdomain dnf[880]:     cli.configure(list(map(ucd, args)), option_parser())
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 959, in configure
feb 11 12:35:01 shalem.localdomain dnf[880]:     self.command.configure()
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/lib/python3.7/site-packages/dnf-plugins/system_upgrade.py", line 320, in configure
feb 11 12:35:01 shalem.localdomain dnf[880]:     self._call_sub("configure")
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/lib/python3.7/site-packages/dnf-plugins/system_upgrade.py", line 332, in _call_sub
feb 11 12:35:01 shalem.localdomain dnf[880]:     subfunc()
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/lib/python3.7/site-packages/dnf-plugins/system_upgrade.py", line 395, in configure_upgrade
feb 11 12:35:01 shalem.localdomain dnf[880]:     self.base.conf.exclude = libdnf.conf.VectorString(self.state.exclude)
feb 11 12:35:01 shalem.localdomain dnf[880]:   File "/usr/lib64/python3.7/site-packages/libdnf/conf.py", line 1197, in __init__
feb 11 12:35:01 shalem.localdomain dnf[880]:     this = _conf.new_VectorString(*args)
feb 11 12:35:01 shalem.localdomain dnf[880]: ValueError: invalid null reference in method 'new_VectorString', argument 1 of type 'std::vector< std::string > const &'
feb 11 12:35:01 shalem.localdomain systemd[1]: dnf-system-upgrade.service: Main process exited, code=exited, status=1/FAILURE
feb 11 12:35:01 shalem.localdomain systemd[1]: dnf-system-upgrade.service: Failed with result 'exit-code'.
feb 11 12:35:01 shalem.localdomain systemd[1]: Failed to start System Upgrade using DNF.
feb 11 12:35:01 shalem.localdomain systemd[1]: dnf-system-upgrade.service: Triggering OnFailure= dependencies.

dnf-system-upgrade.service exiting with a service causes systemd to immediately reboot the system, killing packagekit-offline-update.service:

feb 11 12:35:01 shalem.localdomain systemd[1]: packagekit-offline-update.service: Main process exited, code=killed, status=15/TERM
feb 11 12:35:01 shalem.localdomain systemd[1]: packagekit-offline-update.service: Failed with result 'signal'.
feb 11 12:35:01 shalem.localdomain systemd[1]: Stopped Update the operating system whilst offline.

The problem here is that dnf-system-upgrade.service does not correctly implement the systemd offline system update spec:
https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.html

Specifically section 5 of "The Logic" says:

"As the first step, an update service should check if the /system-update symlink points to the location used by that update service. In case it does not exist or points to a different location, the service must exit without error. It is possible for multiple update services to be installed, and for multiple update services to be launched in parallel, and only the one that corresponds to the tool that created the symlink before reboot should perform any actions. It is unsafe to run multiple updates in parallel."

It seems that dnf-system-upgrade.service tries to do its thing regardless if the
/system-update symlink was created by its and then shortly afterwards errors out with a backtrace because of what I think might be a missing /var/lib/dnf/system-upgrade.json file.

Note I'm not the first person to hit this bug, searching the web for similar issues I found:
https://forums.fedoraforum.org/showthread.php?320121-F29-offline-update

Comment 1 Hans de Goede 2019-02-11 13:51:33 UTC
Note as a workaround one can do: "sudo rpm -e python3-dnf-plugin-system-upgrade" after that gnome-software / packagekit offline updates work again.

Comment 2 Marek Blaha 2019-02-18 12:16:56 UTC
Please, can you specify which versions of dnf components are you using? (rpm -q dnf libdnf python3-dnf-plugin-system-upgrade)

Comment 3 Hans de Goede 2019-02-18 12:20:05 UTC
[hans@shalem ~]$ rpm -q dnf libdnf python3-dnf-plugin-system-upgrade
dnf-4.0.9-2.fc29.noarch
libdnf-0.22.3-1.fc29.x86_64
python3-dnf-plugin-system-upgrade-4.0.0-1.fc29.noarch

Comment 4 Jaroslav Mracek 2019-07-04 06:51:58 UTC
The problem (TraceBack) is resolved by https://github.com/rpm-software-management/dnf-plugins-extras/commit/6c89b394226929e6d6cefaea2619c4a018573399. It is part of dnf-plugins-extras-4.0.4.