Bug 1519543

Summary: Fedora 27 offline updates (gnome-software intitiated) are failing
Product: [Fedora] Fedora Reporter: Chris Murphy <bugzilla>
Component: dnf-plugins-extrasAssignee: Jaroslav Mracek <jmracek>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 27CC: afranke, andrea.vai, bugzilla, daniell1, dmach, ignatenko, jkadlcik, jmracek, jpoimboe, juha.heljoranta, klember, lnykryn, msekleta, pdwyer, rpm-software-management, sirweazel, ssahani, s, systemd-maint, vmukhame, zbyszek
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: dnf-plugins-extras-2.0.5-1.fc27 dnf-plugins-extras-2.0.5-1.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-16 12:33:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
journal.log
none
journaldebug.log
none
Journal none

Description Chris Murphy 2017-11-30 21:12:09 UTC
Created attachment 1361241 [details]
journal.log

Description of problem:


This is on a system upgraded a couple weeks ago with 'dnf system-upgrade' from Fedora 26 to Fedora 27.

I've receive a notification there are updatess, so I go into Gnome Software, choose Restart & Update from within Gnome-Software, it reboots and plymouth gives me the updating indication and status, and pretty much immediately I get a reboot.

Once rebooted, it's clear the updates have not been applied.

Version-Release number of selected component (if applicable):



How reproducible:


Steps to Reproduce:
1. Launch Gnome-Software and click Restart & Update
2. System reboots seemingly normally, and then:

Actual results:

Immediately reboots as the installing updates status appears in plymouth; does not apply updates.

Expected results:

Should apply updates.


Additional info:

Attaching complete journal.log for the prior boot (-b -1) which would be the pkofflineupdate boot that fails. This seems to be the relevant part but it's not enough information. Why does dnf and systemd want dnf-system-upgrade.service for a routine (weekly) non-upgrade update?

[    6.218141] f27h.localdomain pk-offline-update[815]: percentage 4%
[    6.254059] f27h.localdomain pk-offline-update[815]: percentage 6%
[    6.325118] f27h.localdomain dnf[813]: Error: use 'dnf system-upgrade reboot' to begin the upgrade
[    6.352470] f27h.localdomain systemd[1]: dnf-system-upgrade.service: Main process exited, code=exited, status=1/FAILURE
[    6.356239] f27h.localdomain systemd[1]: Failed to start System Upgrade using DNF.
[    6.356570] f27h.localdomain systemd[1]: dnf-system-upgrade.service: Unit entered failed state.
[    6.356777] f27h.localdomain systemd[1]: dnf-system-upgrade.service: Failed with result 'exit-code'.
[    6.356983] f27h.localdomain systemd[1]: Rebooting: service failed

Comment 1 Chris Murphy 2017-11-30 21:15:55 UTC
systemd-234-9.fc27.x86_64
gnome-software-3.26.3-1.fc27.x86_64
dnf-2.7.5-1.fc27.noarch

Comment 2 Chris Murphy 2017-11-30 22:21:55 UTC
Created attachment 1361323 [details]
journaldebug.log

Replace the useless log with one using systemd.log_level=debug.


[    6.876190] f27h.localdomain dnf[823]: Error: use 'dnf system-upgrade reboot' to begin the upgrade
[    6.904437] f27h.localdomain systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 53 (stored), closing.
[    6.905020] f27h.localdomain systemd[1]: Received SIGCHLD from PID 823 (dnf).
[    6.905189] f27h.localdomain systemd[1]: Child 823 (dnf) died (code=exited, status=1/FAILURE)
[    6.905328] f27h.localdomain systemd[1]: dnf-system-upgrade.service: Child 823 belongs to dnf-system-upgrade.service
[    6.906227] f27h.localdomain systemd[1]: dnf-system-upgrade.service: Main process exited, code=exited, status=1/FAILURE
[    6.906471] f27h.localdomain systemd[1]: dnf-system-upgrade.service: About to execute: /usr/bin/rm -fv /system-update
[    6.906792] f27h.localdomain systemd[1]: dnf-system-upgrade.service: Forked /usr/bin/rm as 917
[    6.906984] f27h.localdomain systemd[1]: dnf-system-upgrade.service: Changed start -> stop-post



Well it looks to me like there's a conflict between dnf-system-upgrade.service and packagekit-offline-update.service. They are both running, but why?

[    6.292645] f27h.localdomain systemd[1]: dnf-system-upgrade.service: ConditionPathExists=/system-update succeeded.



Are both of these update methods using the same trigger? Very clearly gnome-software is setting /system-update symlink when I click Restart & Update. Huh...

Comment 3 Chris Murphy 2017-11-30 22:28:51 UTC
To be clear, gnome-software upon selecting and confirming Restart & Update, sets this symlink

/system-update -> /var/lib/PackageKit/prepared-update

This is intended for packagekit-offline-update.service, which does in fact start the update, as indicated by the progress in the log where it gets to 6%, at which point dnf injects itself into the update, and I guess it gets pissed because the symlink points to what it sees as a malformed payload, since the payload pointed to by the symlink is not a dnf system-upgrade download based payload.

Comment 4 Jaroslav Mracek 2017-12-05 11:35:13 UTC
Please can you provide dnf-plugins-extras version on your system?

Comment 5 Chris Murphy 2017-12-06 03:42:29 UTC
$ rpm -qa | grep dnf
python3-dnf-plugins-core-2.1.5-1.fc27.noarch
dnf-yum-2.7.5-1.fc27.noarch
dnf-conf-2.7.5-1.fc27.noarch
libdnf-0.11.1-1.fc27.x86_64
python3-dnf-plugins-extras-common-2.0.4-1.fc27.noarch
dnf-2.7.5-1.fc27.noarch
python3-dnf-2.7.5-1.fc27.noarch
dnf-plugins-core-2.1.5-1.fc27.noarch
python3-dnf-plugin-system-upgrade-2.0.4-1.fc27.noarch

Comment 6 Alexandre Franke 2017-12-12 21:31:41 UTC
Created attachment 1366975 [details]
Journal

Same problem here, attaching my journal output as asked on IRC by Kalev.

FWIW only one of my three machines has this issue and it’s a laptop, unlike the others.

Comment 7 Alexandre Franke 2017-12-12 21:32:15 UTC
*** Bug 1524459 has been marked as a duplicate of this bug. ***

Comment 8 Jaroslav Mracek 2017-12-13 08:45:16 UTC
I believe that the problem can be reproduce only with python3-dnf-plugin-system-upgrade-2.0.4-1.fc27.noarch . The commit https://github.com/rpm-software-management/dnf-plugins-extras/commit/cf33132f305fddf0f283e6227007f7fb7b18f44f should solve the issue.

Comment 9 Chris Murphy 2017-12-13 09:21:23 UTC
That version of python3-dnf-plugin-system-upgrade is stable on F26 and F27, so is seems like it'd affect Fedora 26 as well. If I understand this bug correctly, everyone who did their major version upgrade with dnf system-upgrade, cannot apply minor version updates using Gnome Software. Presumably they could work around this in the meantime by removing python3-dnf-plugin-system-upgrade?

Comment 10 Kalev Lember 2017-12-13 09:35:07 UTC
We're getting reports from multiple users that this has regressed in F26 as well now and broken updates done through gnome-software, e.g. https://mail.gnome.org/archives/gnome-software-list/2017-December/msg00000.html

Comment 11 Andrea Vai 2017-12-13 09:46:35 UTC
I am experiencing the same problem on Fedora 26: see also

https://mail.gnome.org/archives/gnome-software-list/2017-December/msg00000.html

gnome-software-3.24.3-3.fc26.x86_64

$ rpm -qa | grep dnf
python2-dnf-plugin-system-upgrade-2.0.4-1.fc26.noarch
dnf-conf-2.7.5-2.fc26.noarch
python3-dnf-2.7.5-2.fc26.noarch
python2-dnf-plugin-migrate-2.1.5-1.fc26.noarch
python2-dnf-2.7.5-2.fc26.noarch
python3-dnf-plugins-core-2.1.5-1.fc26.noarch
python3-dnf-plugins-extras-common-2.0.4-1.fc26.noarch
python3-dnf-plugin-system-upgrade-2.0.4-1.fc26.noarch
python2-dnf-plugins-core-2.1.5-1.fc26.noarch
python2-dnf-plugins-extras-common-2.0.4-1.fc26.noarch
dnf-2.7.5-2.fc26.noarch
dnf-plugins-core-2.1.5-1.fc26.noarch
libdnf-0.11.1-1.fc26.x86_64
dnf-yum-2.7.5-2.fc26.noarch

Let me know if I have to provide any further info / test result that may help

Comment 12 Jaroslav Mracek 2017-12-13 16:57:29 UTC
Tomorrow I will make a release of dnf-plugins-extras (dnf-system-upgrade) with patch that should solve it.

Comment 13 Fedora Update System 2017-12-14 09:38:26 UTC
dnf-plugins-extras-2.0.5-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-86e209f3c1

Comment 14 Fedora Update System 2017-12-14 09:38:43 UTC
dnf-plugins-extras-2.0.5-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-48b29b741c

Comment 15 Kalev Lember 2017-12-14 09:44:52 UTC
Thanks Jaroslav!

Comment 16 Andrea Vai 2017-12-14 10:12:00 UTC
Thank you. How can I try it out on Fedora 26? 

dnf install dnf-plugins-extras --enablerepo=updates-testing

says no matching argument.

Sorry for my being unskilled.

Comment 17 Jaroslav Mracek 2017-12-14 10:29:07 UTC
RPMs can be downloaded from https://koji.fedoraproject.org/koji/buildinfo?buildID=1009394 or just wait till it appears in updates-testing (I guess 24 to 48 hours).

Comment 18 Fedora Update System 2017-12-14 13:02:42 UTC
dnf-plugins-extras-2.0.5-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-48b29b741c

Comment 19 Fedora Update System 2017-12-14 13:34:18 UTC
dnf-plugins-extras-2.0.5-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-86e209f3c1

Comment 20 Fedora Update System 2017-12-16 12:33:59 UTC
dnf-plugins-extras-2.0.5-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Andrea Vai 2017-12-18 08:08:17 UTC
(In reply to Jaroslav Mracek from comment #17)
> RPMs can be downloaded from
> https://koji.fedoraproject.org/koji/buildinfo?buildID=1009394 or just wait
> till it appears in updates-testing (I guess 24 to 48 hours).

I am sorry but I would ask for more instructions to test it, because still today the command

dnf install dnf-plugins-extras --enablerepo=updates-testing

says "no match for argument"

Is my configuration missing anything, or something else I don't undersand?

Comment 22 SirWeazel 2017-12-23 19:15:53 UTC
Is this getting pushed to the Fedora 26 Stable Repo also? I saw where it's in testing, and has been pushed to the 27 repo already.

Comment 23 Josh Poimboeuf 2018-01-01 16:46:25 UTC
(In reply to Andrea Vai from comment #21)
> I am sorry but I would ask for more instructions to test it, because still
> today the command
> 
> dnf install dnf-plugins-extras --enablerepo=updates-testing
> 
> says "no match for argument"
> 
> Is my configuration missing anything, or something else I don't undersand?

This worked for me:

dnf update --enablerepo=updates-testing python3-dnf-plugin-system-upgrade

Comment 24 Fedora Update System 2018-01-09 16:46:59 UTC
dnf-plugins-extras-2.0.5-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.