Bug 2350947 - dnf offline-upgrade reboot causes unclean shutdown after upgrade (fsck messages)
Summary: dnf offline-upgrade reboot causes unclean shutdown after upgrade (fsck messages)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf5
Version: 41
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Petr Pisar
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-03-09 10:56 UTC by 1gooddood1
Modified: 2025-07-31 02:03 UTC (History)
15 users (show)

Fixed In Version: dnf5-5.2.15.0-1.fc43 dnf5-5.2.15.0-2.fc42 dnf5-5.2.15.0-2.fc41
Clone Of:
Environment:
Last Closed: 2025-07-31 00:53:08 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
journalctl logs (247.85 KB, text/plain)
2025-03-09 10:56 UTC, 1gooddood1
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github rpm-software-management dnf5 pull 2328 0 None open Offline graceful reboot 2025-06-30 16:01:02 UTC

Description 1gooddood1 2025-03-09 10:56:27 UTC
I use the following commands to perform offline upgrades from CLI with dnf5:
$ dnf offline-upgrade download --refresh
$ dnf offline-upgrade reboot

I notice that when after the upgrades complete and the system reboots, I keep getting fsck messages like below:

Mar 09 11:25:47 oldhome systemd[1]: Starting systemd-fsck-root.service - File System Check on /dev/mapper/vg_root-lv_root...
Mar 09 11:25:47 oldhome systemd-fsck[1080]: /dev/mapper/vg_root-lv_root: recovering journal
Mar 09 11:25:48 oldhome systemd-fsck[1080]: /dev/mapper/vg_root-lv_root: 323267/2621440 files (0.4% non-contiguous), 7952549/10485760 blocks

This happens almost every time except for when only a very small amount of packages are being upraded. My root filesystem is ext4 on LVM.
I attached the journalctl logs from the boot in which the upgrades happen.
After dnf5 initiates the reboot, I see no indication that my filesystems are being unmounted before the system reboots.


Reproducible: Always

Comment 1 1gooddood1 2025-03-09 10:56:57 UTC
Created attachment 2079410 [details]
journalctl logs

Comment 2 Petr Pisar 2025-03-10 09:28:38 UTC
DNF5 does not hard-reset a system. It requests systemd with a "Reboot" method call to "org.freedesktop.systemd1.Manager" D-Bus interface. I guess systemd is unable to unmount your root file systemd before issuing a system reset. Moving to systemd.

I you would be great if you told us what systemd, kernel, and dracut packages you have installed. Also make sure that your initramdisk /boot/initramfs-* is successfully regenerated.

Comment 3 Zbigniew Jędrzejewski-Szmek 2025-03-10 13:20:57 UTC
Thank you for the log.
I filed https://github.com/rpm-software-management/dnf5/issues/2121
and https://github.com/rpm-software-management/dnf5/issues/2122
for some cosmetic issues.

The log shows an intermediate shutdown:
Mar 09 11:25:12 oldhome dnf5[1974]: Transaction complete! Cleaning up and rebooting...
Mar 09 11:25:12 oldhome systemd[1]: Reboot requested from client PID 1974 ('dnf5') (unit dnf5-offline-transaction.service)...
Mar 09 11:25:12 oldhome systemd[1]: Shutting down.
Mar 09 11:25:12 oldhome dnf5[1974]: Complete!
Mar 09 11:25:13 oldhome systemd-shutdown[1]: Syncing filesystems and block devices.
Mar 09 11:25:13 oldhome systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Mar 09 11:25:13 oldhome systemd-journald[2773]: Received SIGTERM from PID 1 (systemd-shutdow).
Mar 09 11:25:13 oldhome systemd-journald[2773]: Journal stopped

This is because reboot() in dnf5/commands/offline/offline.cpp requests an intermediate reboot:
  proxy->callMethod("Reboot").onInterface(SYSTEMD_MANAGER_INTERFACE);

The systemd.offline-updates(7) man page says:
   If your script succeeds you should trigger the reboot in your own code, for example
   by invoking logind's Reboot() call or calling systemctl reboot. See org.freedesktop.login1(5)
   for details about the logind D-Bus API.

The original dnf-system-upgrade plugin used the normal reboot interfae:
https://github.com/rpm-software-management/dnf-plugin-system-upgrade/blob/5835d930d76f04054ea1ee69e222f302eac5f4d8/system_upgrade.py#L105

Dnf shouldn't do immediate reboot. There is no reason not to do a normal
shutdown. Whatever services are running should be stopped cleanly. 
The offline upgrade mode makes an effort to not start unnecessary services,
but the ones that are started should be treated normally. This would also
give systemd to umount more of the disks.

Please switch the plugin to use 'systemctl reboot' or the logind interface.

(All that said, systemd *should* be able to unmount or remount-ro all mountpoints
during shutdown. systemd-shutdown binary which takes over from systemd during
late shutdown does a loop over all mounts trying to do that. Unfortunately this
is after the logging has been stopped, so we don't know why this wasn't successful.
Do you see any messages during late shutdown which might indicate why that isn't
happening? Is the reboot very quick, or does it seem to be doing some work or
waiting for something at shutdown?)

Comment 4 1gooddood1 2025-03-10 16:17:55 UTC
I have the following packages installed:
systemd.x86_64       256.12-1.fc41
dracut.x86_64        103-3.fc41
kernel-core.x86_64   6.13.5-200.fc41
dnf5.x86_64          5.2.10.0-2.fc41

nemu@oldhome:~$ uname -a
Linux oldhome 6.13.5-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 27 15:07:31 UTC 2025 x86_64 GNU/Linux

nemu@oldhome:~$ dnf --version
dnf5 version 5.2.10.0
dnf5 plugin API version 2.0
libdnf5 version 5.2.10.0
libdnf5 plugin API version 2.1

>Also make sure that your initramdisk /boot/initramfs-* is successfully regenerated.
How can I do this?

>Do you see any messages during late shutdown which might indicate why that isn't
>happening? Is the reboot very quick, or does it seem to be doing some work or
>waiting for something at shutdown?
I don't believe I see much more than what is in the logs.
I did always think that the reboot happens suspiciously fast.
After dnf finishes the last task in the upgrade process, it reboots near instantaneously.

Comment 5 Petr Pisar 2025-03-11 14:42:59 UTC
Thanks Zbigniew for the documentation. DNF5 needs to change the reboot method.

Comment 6 Petr Pisar 2025-06-27 16:23:33 UTC
I'm fixing this problem and in my testing system the reboot via D-Bus fails because there is no system-wide D-bus daemon running in the offline-updates environment. (I have installed dbus-broker.)

After adding dbus-broker as a dependency of dnf5-offline-transaction.service, the logind Reboot(true) works. However, I feel it's an overkill because it starts the whole logind subsystem just only to stop it later:

čen 27 18:00:46 fedora-43 dnf5[641]: HIT pre reboot
čen 27 18:00:46 fedora-43 dnf5[641]: Transaction complete! Cleaning up and rebooting...
čen 27 18:00:47 fedora-43 systemd[1]: Created slice user.slice - User and Session Slice.
čen 27 18:00:47 fedora-43 systemd[1]: Starting modprobe - Load Kernel Module drm...
čen 27 18:00:47 fedora-43 systemd[1]: modprobe: Deactivated successfully.
čen 27 18:00:47 fedora-43 systemd[1]: Finished modprobe - Load Kernel Module drm.
čen 27 18:00:47 fedora-43 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? add>
čen 27 18:00:47 fedora-43 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr>
čen 27 18:00:47 fedora-43 kernel: audit: type=1130 audit(1751040047.041:49): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/syst>
čen 27 18:00:47 fedora-43 kernel: audit: type=1131 audit(1751040047.041:50): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/syst>
čen 27 18:00:47 fedora-43 audit: BPF prog-id=35 op=LOAD
čen 27 18:00:47 fedora-43 audit: BPF prog-id=36 op=LOAD
čen 27 18:00:47 fedora-43 audit: BPF prog-id=37 op=LOAD
čen 27 18:00:47 fedora-43 kernel: audit: type=1334 audit(1751040047.045:51): prog-id=35 op=LOAD
čen 27 18:00:47 fedora-43 kernel: audit: type=1334 audit(1751040047.045:52): prog-id=36 op=LOAD
čen 27 18:00:47 fedora-43 kernel: audit: type=1334 audit(1751040047.045:53): prog-id=37 op=LOAD
čen 27 18:00:47 fedora-43 systemd[1]: Starting systemd-logind.service - User Login Management...
čen 27 18:00:47 fedora-43 systemd-logind[652]: New seat seat0.
čen 27 18:00:47 fedora-43 systemd-logind[652]: Watching system buttons on /dev/input/event0 (Power Button)
čen 27 18:00:47 fedora-43 systemd-logind[652]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard)
čen 27 18:00:47 fedora-43 systemd[1]: Started systemd-logind.service - User Login Management.
čen 27 18:00:47 fedora-43 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-logind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? a>
čen 27 18:00:47 fedora-43 systemd-logind[652]: The system will reboot now!
čen 27 18:00:47 fedora-43 systemd-logind[652]: System is rebooting.
čen 27 18:00:47 fedora-43 kernel: audit: type=1130 audit(1751040047.122:54): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-logind comm="systemd" exe="/usr/lib/sy>
čen 27 18:00:47 fedora-43 kernel: audit: type=1131 audit(1751040047.141:55): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udev-load-credentials comm="systemd" e>
čen 27 18:00:47 fedora-43 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udev-load-credentials comm="systemd" exe="/usr/lib/systemd/systemd>
čen 27 18:00:47 fedora-43 dnf5[641]: HIT post reboot

The "HIT pre/post reboot" message surround the Reboot D-Bus call.

I think executing "reboot" would simpler and thus less error-prone.

However, I don't like that a responsibility of the reboot is on DNF5. The reason is that when DNF5 invokes a reboot, there is no guarantee that the dnf5 process naturally terminates before systemd kills its service because of shutting down the system. And if the dnf5 process exits with non-zero exit code, then systemd will log that the update did not finish successfully.

Comment 7 Petr Pisar 2025-06-30 12:00:54 UTC
systemd's /usr/bin/shutdown also does not work without D-Bus :(

Comment 8 Petr Pisar 2025-07-01 06:27:17 UTC
In the end, I added a dependency on dbus.socket.

Comment 9 1gooddood1 2025-07-01 15:22:00 UTC
Many thanks!

Comment 10 Fedora Update System 2025-07-16 08:35:34 UTC
FEDORA-2025-a5e5885906 (dnf5-5.2.15.0-1.fc42 and librepo-1.20.0-1.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-a5e5885906

Comment 11 Fedora Update System 2025-07-16 08:36:09 UTC
FEDORA-2025-fb095dd283 (dnf5-5.2.15.0-1.fc41 and librepo-1.20.0-1.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-fb095dd283

Comment 12 Fedora Update System 2025-07-17 00:58:44 UTC
FEDORA-2025-a5e5885906 has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-a5e5885906`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-a5e5885906

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2025-07-17 01:07:39 UTC
FEDORA-2025-fb095dd283 has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-fb095dd283`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-fb095dd283

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2025-07-24 14:59:09 UTC
FEDORA-2025-9ae670b810 (dnf5-5.2.15.0-2.fc42 and librepo-1.20.0-1.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-9ae670b810

Comment 15 Fedora Update System 2025-07-24 15:00:13 UTC
FEDORA-2025-fdcda3af30 (dnf5-5.2.15.0-2.fc41 and librepo-1.20.0-1.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-fdcda3af30

Comment 16 Fedora Update System 2025-07-25 02:13:33 UTC
FEDORA-2025-9ae670b810 has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-9ae670b810`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-9ae670b810

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2025-07-25 02:26:15 UTC
FEDORA-2025-fdcda3af30 has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-fdcda3af30`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-fdcda3af30

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2025-07-31 00:53:08 UTC
FEDORA-2025-9ae670b810 (dnf5-5.2.15.0-2.fc42 and librepo-1.20.0-1.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 19 Fedora Update System 2025-07-31 02:03:59 UTC
FEDORA-2025-fdcda3af30 (dnf5-5.2.15.0-2.fc41 and librepo-1.20.0-1.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.