Description of problem: The upgrades of Fedora IoT 35 are rolled back every time because the device (Raspberry Pi 2) lacks a hardware clock and I have enabled DNSSEC in systemd-resolved. When booting in the new system image, the DNS checks fail due to DNSSEC rejecting the queries due an incorrect system time. Version-Release number of selected component (if applicable): fedora-iot:fedora/stable/armhfp/iot Version: 35.20220106.0 (2022-01-06T17:52:37Z) Commit: 0b874b32edc64b4a3e2500afbb3ef715ee8a7c75f323337439919448f2bc954d GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39 How reproducible: Always Steps to Reproduce: 1. Install Fedora 35 IoT in Raspberry Pi 2 2. Set DNSSEC=yes in /etc/systemd/resolved.conf 2. rpm-ostree upgrade 3. reboot Actual results: The upgrade is rolled back. These messages can be seen in the journal (note that the system time is incorrect, the correct time is 2022-01-19): Nov 15 01:00:57 pi2 00_required_scripts_start.sh[781]: Running greenboot Required Health Check Scripts Nov 15 01:00:57 pi2 greenboot[765]: Script '00_required_scripts_start.sh' SUCCESS Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question . IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question org IN DS: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question org IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question fedoraproject.org IN DS: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question fedoraproject.org IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question ostree.fedoraproject.org IN AAAA: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question . IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question org IN DS: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question org IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question fedoraproject.org IN DS: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question fedoraproject.org IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question ostree.fedoraproject.org IN AAAA: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question . IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question org IN DS: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question org IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question fedoraproject.org IN DS: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question fedoraproject.org IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question ostree.fedoraproject.org IN A: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question . IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question org IN DS: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question org IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 01_repository_dns_check.sh[784]: The following repository domains haven't responded properly to DNS queries: Nov 15 01:00:57 pi2 01_repository_dns_check.sh[784]: ostree.fedoraproject.org Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question fedoraproject.org IN DS: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question fedoraproject.org IN DNSKEY: signature-expired Nov 15 01:00:57 pi2 systemd-resolved[637]: [🡕] DNSSEC validation failed for question ostree.fedoraproject.org IN A: signature-expired Nov 15 01:00:57 pi2 greenboot[765]: Script '01_repository_dns_check.sh' FAILURE (exit code '1'). Continuing... Nov 15 01:00:57 pi2 02_watchdog.sh[797]: Watchdog check is disabled Nov 15 01:00:57 pi2 greenboot[765]: Script '02_watchdog.sh' SUCCESS Nov 15 01:00:57 pi2 systemd[1]: greenboot-healthcheck.service: Main process exited, code=exited, status=1/FAILURE Nov 15 01:00:57 pi2 systemd[1]: greenboot-healthcheck.service: Failed with result 'exit-code'. Nov 15 01:00:57 pi2 systemd[1]: Failed to start greenboot Health Checks Runner. Nov 15 01:00:57 pi2 systemd[1]: Dependency failed for Boot Completion Check. Nov 15 01:00:57 pi2 systemd[1]: Dependency failed for Mark boot as successful in grubenv. Nov 15 01:00:57 pi2 systemd[1]: Dependency failed for Multi-User System. Nov 15 01:00:57 pi2 systemd[1]: multi-user.target: Job multi-user.target/start failed with result 'dependency'. Nov 15 01:00:57 pi2 systemd[1]: greenboot-grub2-set-success.service: Job greenboot-grub2-set-success.service/start failed with result 'dependency'. Nov 15 01:00:57 pi2 systemd[1]: Dependency failed for greenboot Success Scripts Runner. Nov 15 01:00:57 pi2 systemd[1]: greenboot-task-runner.service: Job greenboot-task-runner.service/start failed with result 'dependency'. Nov 15 01:00:57 pi2 systemd[1]: boot-complete.target: Job boot-complete.target/start failed with result 'dependency'. Nov 15 01:00:57 pi2 systemd[1]: greenboot-healthcheck.service: Triggering OnFailure= dependencies. Nov 15 01:00:57 pi2 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=greenboot-healthcheck comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' Nov 15 01:00:57 pi2 kernel: audit: type=1130 audit(1636934457.860:164): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=greenboot-healthcheck comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=f> Nov 15 01:00:57 pi2 systemd[1]: Starting greenboot Failure Scripts Runner... Nov 15 01:00:57 pi2 greenboot[824]: Boot Status is RED - Health Check FAILURE! Nov 15 01:00:57 pi2 systemd[1]: Starting Record Runlevel Change in UTMP... Nov 15 01:00:57 pi2 greenboot[824]: Running Red Scripts... Nov 15 01:00:58 pi2 systemd[1]: Finished greenboot Failure Scripts Runner. Nov 15 01:00:58 pi2 NetworkManager[687]: <info> [1636934458.1064] dhcp6 (eth0): activation: beginning transaction (timeout in 45 seconds) Nov 15 01:00:58 pi2 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=redboot-task-runner comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Nov 15 01:00:58 pi2 kernel: audit: type=1130 audit(1636934458.113:165): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=redboot-task-runner comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=suc> Nov 15 01:00:58 pi2 systemd[1]: Reached target Generic red boot target. Nov 15 01:00:58 pi2 NetworkManager[687]: <info> [1636934458.1302] policy: set 'Wired connection 1' (eth0) as default for IPv6 routing and DNS Nov 15 01:00:58 pi2 NetworkManager[687]: <info> [1636934458.1656] dhcp6 (eth0): state changed unknown -> bound, address=2600:70ff:f0d0:7::7bf7 2600:70ff:f0d0:7::3 Nov 15 01:00:58 pi2 systemd-resolved[637]: eth0: Bus client set DNS server list to: 192.168.7.1, 2600:70ff:f0d0:7::1 Nov 15 01:00:58 pi2 systemd[1]: Starting greenboot MotD Generator... Nov 15 01:00:58 pi2 systemd[1]: Starting Reboot on red boot status... Nov 15 01:00:58 pi2 systemd-update-utmp[825]: Failed to get new runlevel, utmp update skipped. Nov 15 01:00:58 pi2 systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully. Nov 15 01:00:58 pi2 systemd[1]: Finished Record Runlevel Change in UTMP. Nov 15 01:00:58 pi2 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-update-utmp-runlevel comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Nov 15 01:00:58 pi2 kernel: audit: type=1130 audit(1636934458.369:166): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-update-utmp-runlevel comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=> Nov 15 01:00:58 pi2 kernel: audit: type=1131 audit(1636934458.372:167): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-update-utmp-runlevel comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=> Nov 15 01:00:58 pi2 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-update-utmp-runlevel comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Nov 15 01:00:58 pi2 redboot-auto-reboot[832]: SYSTEM is UNHEALTHY. Rebooting... Nov 15 01:00:59 pi2 systemd[1]: redboot-auto-reboot.service: Deactivated successfully. Nov 15 01:00:59 pi2 systemd[1]: Finished Reboot on red boot status. Expected results: Successful upgrade Additional info: Switching from chronyd to systemd-timesyncd fixes the issue, as the latter stores the timestamp of the last synchronization and restores it early on boot. I believe it's worth considering the switch to systemd-timesyncd by default in Fedora IoT.
TBH that's kind of working as expected because DNS queries would fail if the clock wasn't correct with DNSSEC enabled. This is one of the greenboot health checks so moving it there and will have a think how we can enhance this check.
I'm also unable to update (RPI4) due to `01_repository_dns_check.sh` causing rollbacks (failing on ostree.fedoraproject.org). I've not, however, enabled DNSSEC in `/etc/systemd/resolved.conf`, so I'm utterly clueless as to why it fails. Switching from chronyd to systemd-timesyncd did not solve the issue. Version I'm updating from: 35.20211212.1 Check passes on rollbacked version.
@vilgot could you provide the relevant journal logs for a boot where this occurred?
This message is a reminder that Fedora Linux 35 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '35'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 35 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13. Fedora Linux 35 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.