After upgrading from F41 to F42 I noticed after some days that I was no longer receiving emails. I found that postfix.service, which had been enabled before, was now disabled. I could just re-enable and restart it and now everything appears to be fine again. Reproducible: Didn't try Additional Information: Used the DNF method for upgrading.
I also experienced this bug. Upgrade should not disable a previously enabled service. Furthermore, postfix is now logging the following warnings - this would seem to be related to the /usr/sbin move: May 05 08:16:48 localhost postfix[1814434]: postfix/postlog: warning: not owned by group postdrop: /usr/sbin/postdrop May 05 08:16:48 localhost postfix/postfix-script[1814434]: warning: not owned by group postdrop: /usr/sbin/postdrop May 05 08:16:48 localhost postfix[1814436]: postfix/postlog: warning: not set-gid or not owner+group+world executable: /usr/sbin/postqueue May 05 08:16:48 localhost postfix/postfix-script[1814436]: warning: not set-gid or not owner+group+world executable: /usr/sbin/postqueue May 05 08:16:48 localhost postfix[1814437]: postfix/postlog: warning: not set-gid or not owner+group+world executable: /usr/sbin/postdrop May 05 08:16:48 localhost postfix/postfix-script[1814437]: warning: not set-gid or not owner+group+world executable: /usr/sbin/postdrop
Upgrading postfix once again disabled the systemd service. $ dnf history info last | grep postfix Upgrade postfix-2:3.9.1-5.fc42.x86_64 User updates Upgrade postfix-perl-scripts-2:3.9.1-5.fc42.x86_64 User updates Replaced postfix-2:3.9.1-3.fc42.x86_64 User @System Replaced postfix-perl-scripts-2:3.9.1-3.fc42.x86_64 User @System $ systemctl status postfix.service ○ postfix.service - Postfix Mail Transport Agent Loaded: loaded (/usr/lib/systemd/system/postfix.service; disabled; preset: disabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: inactive (dead)
This is more recent I think. I upgraded all of my fedora hosts and everything was fine. I have a loghost that was disabled I think on 5/15. I updated my mail server for my domain yesterday and it promptly disabled it. Hours later I realized I wasn't getting mail anymore. What's odd is the loghost was disabled and not the mail server. Looks like I updated them on 5/15/25. I use an ansible script and both hosts show updates on the 15th, both were rebooted.
Wanted to say I ran into the same issue. Upgrade to 42 disabled postfix and recent update also disabled postfix. There was also an issue with postfix and the consolidation of /usr/bin and /usr/sbin. Don't know if that's related or not.
After another postfix update, the service was disabled again.
After upgrading from F41 to F42 Postfix was disabled during upgrade, and again this week end during upgrade to postfix-3.9.1-5.fc42.x86_64
In the postinstall scriptlet there is if [ $1 -eq 1 ] && [ -x "/usr/lib/systemd/systemd-update-helper" ]; then # Initial installation /usr/lib/systemd/systemd-update-helper install-system-units postfix.service || : fi the install-system-units verb tells systemd-update-helper to call systemctl like this: systemctl --no-reload preset "$@" and the systemctl manpage says: preset UNIT... Reset the enable/disable status one or more unit files, as specified on the command line, to the defaults configured in the preset policy files. This has the same effect as disable or enable, depending how the unit is listed in the preset files. as postfix.service preset is disabled, the service is disabled during upgrade
(In reply to Laurent Jacquot from comment #7) > In the postinstall scriptlet there is > > if [ $1 -eq 1 ] && [ -x "/usr/lib/systemd/systemd-update-helper" ]; then > # Initial installation > /usr/lib/systemd/systemd-update-helper install-system-units > postfix.service || : > fi For an upgrade action however, this script should receive $1 == 2 so therefore this command should not run. https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/
Yes I've seen this is for Initial installation as stated in the script, but for some reason this is triggered.. When looking in the logs however, the transaction looks like a normal upgrade: 2025-05-18T11:07:28+0000 [1907198] DEBUG [librepo] Transfer finished: Packages/p/postfix-3.9.1-5.fc42.x86_64.rpm (Effective url: http://distrib-coffee.ipsl.jussieu.fr/pub/linux/fedora/linux/updates/42/Everything/x86_64/Packages/p/postfix-3.9.1-5.fc42.x86_64.rpm) 2025-05-18T11:07:30+0000 [1907198] DEBUG Implicitly added element postfix-2:3.9.1-3.fc42.x86_64 type remove package (caused by postfix-2:3.9.1-5.fc42.x86_64) 2025-05-18T11:07:32+0000 [1907198] INFO RPM callback open file "/var/cache/libdnf5/updates-79babcf8637033ce/packages/postfix-3.9.1-5.fc42.x86_64.rpm" 2025-05-18T11:07:33+0000 [1907198] INFO RPM callback open file "/var/cache/libdnf5/updates-79babcf8637033ce/packages/postfix-3.9.1-5.fc42.x86_64.rpm" 2025-05-18T11:07:34+0000 [1907198] INFO RPM callback open file "/var/cache/libdnf5/updates-79babcf8637033ce/packages/postfix-3.9.1-5.fc42.x86_64.rpm" 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback open file "/var/cache/libdnf5/updates-79babcf8637033ce/packages/postfix-3.9.1-5.fc42.x86_64.rpm" 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback start sysusers scriptlet "postfix-2:3.9.1-5.fc42.x86_64" 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback stop sysusers scriptlet "postfix-2:3.9.1-5.fc42.x86_64" return code 0 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback start pre-install scriptlet "postfix-2:3.9.1-5.fc42.x86_64" 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback stop pre-install scriptlet "postfix-2:3.9.1-5.fc42.x86_64" return code 0 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback install start "postfix-2:3.9.1-5.fc42.x86_64" total 4659656 2025-05-18T11:09:25+0000 [1907198] WARNING [rpm] /etc/postfix/main.cf créé en tant que /etc/postfix/main.cf.rpmnew 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback install stop "postfix-2:3.9.1-5.fc42.x86_64" amount 4659656 total 4659656 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback start post-install scriptlet "postfix-2:3.9.1-5.fc42.x86_64" 2025-05-18T11:09:25+0000 [1907198] INFO RPM callback stop post-install scriptlet "postfix-2:3.9.1-5.fc42.x86_64" return code 0 2025-05-18T11:09:58+0000 [1907198] INFO RPM callback start pre-uninstall scriptlet "postfix-2:3.9.1-3.fc42.x86_64" 2025-05-18T11:09:58+0000 [1907198] INFO RPM callback stop pre-uninstall scriptlet "postfix-2:3.9.1-3.fc42.x86_64" return code 0 2025-05-18T11:09:58+0000 [1907198] INFO RPM callback uninstall start "postfix-2:3.9.1-3.fc42.x86_64" total 347 2025-05-18T11:09:58+0000 [1907198] INFO RPM callback uninstall stop "postfix-2:3.9.1-3.fc42.x86_64" amount 347 total 347 2025-05-18T11:09:58+0000 [1907198] INFO RPM callback start post-uninstall scriptlet "postfix-2:3.9.1-3.fc42.x86_64" 2025-05-18T11:09:58+0000 [1907198] INFO RPM callback stop post-uninstall scriptlet "postfix-2:3.9.1-3.fc42.x86_64" return code 0
Just did "dnf --refresh upgrade" and it did not include any updated version of postfix. However, upon reboot postfix was disabled. I had to enable it and start it. I think this was the first time I've seen it disabled without a new version being installed.
Same here, noticed because a user on IRC mentioned it. Postfix was running for me, but I also have it in monit and it may have started it for me. But it was disabled by default, which it was not before as I run my own mail.
I noticed that randomly I get postfix disabled on all of my VMs. I checked ~20 VMs in total, and all but the Fedora 41 ones had postfix stopped and disabled. All of the VMs are automatically updated by dnf-automatic. The latest lot of breakage happened for me around 28-29 of last month. I had emails from cron / dnf-automatic on the 28th, but nothing from the 29th onwards. There were no postfix updates on any of the VMs on the 28th or 29th - so something is screwing up bad here.
Oh - I should mention that this isn't the first time over the last month that this has occurred. There have been multiple times where postfix has been stopped / disabled by some automated process over the last month.
Just a note to mention that this occurs again in a system-upgrade from F42 to F43 Beta. This time, it killed both postfix and nftables. It should not be acceptable to have security related services disabled on upgrades.
Proposed as a Blocker for 43-final by Fedora user crcinau using the blocker tracking app because: The upgrade from F41 -> F42 had this original bug. Upgrades from F42 to F43 beta currently cause multiple services to be reset / disabled. For me personally, it disabled postfix and nftables on an upgrade - completely opening up the system to the internet. This scope has worsened since the F41->F42, so should be addressed before F43 final.
Something similar has happened for a while with another MTA package, exim: https://bugzilla.redhat.com/show_bug.cgi?id=2355049 Perhaps the root cause is similar.
nftables only uses standard macros. I suspect something is busted in the systemd service macros. Re-assigning to systemd. For Steven's experience, systemd 258 went stable on Sept 21 - https://bodhi.fedoraproject.org/updates/FEDORA-2025-744415bdd8 - so I suspect that might be the update that caused the problem? Another possibility here is that it's *RPM* that's borked and not providing the right args to the scriptlets?
My problem with a similar disabling of the exim service here: https://bugzilla.redhat.com/show_bug.cgi?id=2355049 began in March 2025 so if the cause is common then it's not necessarily in systemd but as Adam suggests maybe something in rpm?
AGREED Delayed Decision (punt) Discussed at the 2025-10-06 (blocker / freeze exception) review meeting: This doesn't look like it violates the criteria so far, but more investigation is needed before making a final decision. https://meetbot-raw.fedoraproject.org//blocker-review_matrix_fedoraproject-org/2025-10-06/f43-blocker-review.2025-10-06-16.00.txt
Hmm, based on the multiple reports, it seems that there is an issue. But so far, nobody has thought to provide a full upgrade log. With just random snippets of logs, it's very hard to figure out what is going on. We don't even know which packages were part of the update.
So yeah, this is not reproducible just by installing postfix and upgrading as specified in comment#2. Maybe some special combination of packages in the upgrade is required. Please provide the info about the upgrade ('dnf history info NN') and the logs ('journalctl -b' from the affected boot.)
Created attachment 2109388 [details] dnf history Here is the dnf info from the upgrade. Unfortunately my journal logs are no longer available for this date.
Thanks, that is useful. I tried the upgrade postfix-2:3.9.0-8.fc41.x86_64 → postfix-2:3.9.1-3.fc42.x86_64, and then to 2:3.9.1-5.fc42, but it all went fine. Dunno, this is very strange. If the general mechanism was broken in either systemd or rpm, I think the issue would have to be very widespread and noticeable and it'd be widely reported. But so far this has been reported just for a handful of packages (postfix, exim, something-network-related). Comment #1 indicates some other breakage. But I don't think we're going to figure this out without either a reproducer or some logs that indicate what the issue is.
There is an exim-4.99 upgrade pending soon, it's currently at the RC3 state. Can you please advise on the best set of commands to capture the upgrade process and all the scriptlet and systemd activity to avoid something important being lost? Thanks.
My wild-ass guess here would be that this may involve a trigger from another installed package. Triggers are cool, but can cause unexpected effects that are a bit tricky to debug...
> Can you please advise on the best set of commands to capture the upgrade > process and all the scriptlet and systemd activity to avoid something > important being lost? I don't think any special preparation is necessary. Please check before the upgrade that the service is indeed enabled, and after the upgrade, if that changed. If the bug occurs, attach the logs ('dnf history info last' and 'journalctl -b XX' from the the boot where the upgrade happened).
This was an issue to me back in May when I reported it. So far I haven't had any more trouble with it turning Postfix off.
The XX is found where, in the dnf history output?
Whilst I've been moving my more critical services away from Fedora because of this bug, just about every single VM I had has shown this problem randomly since around March this year. It may help to reproduce the problem as all VMs are based off a kickstart install. The kickstart file I use is at: https://www.crc.id.au/files/fedora.ks or https://www.crc.id.au/files/fedora-docker.ks If you do an install with that as Fedora 42, and then system-upgrade to Fedora 43 - this may show the problem. If the repro works, then I'd expect postfix to be running at the end of the install and on the first reboot... Then after the system-upgrade, it'll probably be disabled and not running after the upgrade.
AGREED Delayed Decision (Punt) Discussed at the 2025-10-13 (blocker / freeze exception) review meeting: This bug is concerning, but it is too unclear at this moment to be considered as a blocker. We'll wait whether more solid understanding of the bug is made in the near future. https://meetbot-raw.fedoraproject.org//blocker-review_matrix_fedoraproject-org/2025-10-13/f43-blocker-review.2025-10-13-16.00.txt
I upgraded an F42 machine running postfix yesterday and it got disabled. Straight dnf upgrade - nothing fancy. Just FYI. PS. On that same machine, systemd could not find default target after upgrade and I was prompted to start an emergency shell. Probably unrelated, but I thought I'd mention it anyway. I do have some systemd drop-in overrides, if it makes any difference.
What overrides do you have?
Before I get you those, here is one of trigger scripts I found on my machine: ----- # rpm -q --triggers opendkim triggerun scriptlet (using /bin/sh) -- opendkim < 2.8.0-1 if [ $1 -eq 1 ] && [ -x "/usr/lib/systemd/systemd-update-helper" ]; then # Initial installation /usr/lib/systemd/systemd-update-helper install-system-units opendkim.service || : fi /usr/bin/chkconfig --del opendkim >/dev/null 2>&1 || : if [ $1 -ge 1 ] && [ -x "/usr/lib/systemd/systemd-update-helper" ]; then # Package upgrade, not uninstall /usr/lib/systemd/systemd-update-helper mark-restart-system-units opendkim.service || : fi -----
Sorry, wrong one: ----- # rpm -q --triggers postfix triggerun scriptlet (using /bin/sh) -- postfix < 2.8.12-2 /usr/bin/systemd-sysv-convert --save postfix >/dev/null 2>&1 ||: /usr/bin/systemd-sysv-convert --apply postfix >/dev/null 2>&1 ||: /sbin/chkconfig --del postfix >/dev/null 2>&1 || : ----- I mean, this should only trigger when postfix is older than 2.8.12-2, right?
My override: ----- amavisd.service.d ----- [Unit] Wants=sa-update.timer ----- clamd.d ----- [Service] TimeoutSec=600 ----- dhcpd6.service.d ----- [Unit] Wants=network-online.target After=network-online.target ----- dhcpd.service.d ----- [Unit] Wants=network-online.target After=network-online.target ----- dovecot.service.d ----- [Unit] Wants=network-online.target named-chroot.service After=network-online.target named-chroot.service [Service] Restart=on-failure ----- httpd.service.d ----- [Service] UMask=0027 ----- named.service.d ----- [Service] Restart=on-failure [Unit] Wants=network-online.target After=network-online.target ----- postfix.service.d ----- [Unit] Wants=network-online.target amavisd.service opendkim.service opendmarc.service named-chroot.service After=network-online.target amavisd.service opendkim.service opendmarc.service named-chroot.service ----- rsyslog.service.d ----- [Unit] Wants=network-online.target After=network-online.target ----- slapd.service.d ----- [Service] Environment="SLAPD_URLS=ldap:/// ldapi:///" "SLAPD_OPTIONS=" EnvironmentFile=/etc/sysconfig/slapd ExecStartPre= ExecStartPre=/usr/bin/rm -f /usr/share/openldap-servers/UPGRADE_INSTRUCTIONS ExecStartPre=/usr/libexec/openldap/check-config.sh ExecStart= ExecStart=/usr/sbin/slapd -u ldap -h ${SLAPD_URLS} $SLAPD_OPTIONS Restart=on-failure
(In reply to Bojan Smojver from comment #34) > # rpm -q --triggers postfix > triggerun scriptlet (using /bin/sh) -- postfix < 2.8.12-2 > /usr/bin/systemd-sysv-convert --save postfix >/dev/null 2>&1 ||: > /usr/bin/systemd-sysv-convert --apply postfix >/dev/null 2>&1 ||: > /sbin/chkconfig --del postfix >/dev/null 2>&1 || : systemd-sysv-convert was dropped in 2013! /sbin/chkconfig --del postfix just says: "error reading information on service postfix: No such file or directory" So this script, even if it did run, wouldn't have any effect. > I mean, this should only trigger when postfix is older than 2.8.12-2, right? Yes.
I looked through the journal to see whether there is any evidence of postfix getting disabled, but I can only see the service getting stopped during the upgrade. So, not quite sure what exactly is doing that.
I do have a reproducer, however: dnf reinstall postfix Just disabled the service. I'll dig a bit more into it.
Running this: rpm -Uvh --force --nopost postfix-3.10.3-3.fc43.x86_64.rpm Does not disable the service. Disabling triggers does not help. Still trying to find out what it actually is...
It's the call to alternatives in the postinstall. Probably that --initscript option doing it. If I move /usr/bin/alternatives to a different name, so it cannot get started, it does not happen.
exim is happily running. The service is showing as enabled: systemctl status exim ● exim.service - Exim Mail Transport Agent Loaded: loaded (/usr/lib/systemd/system/exim.service; enabled; preset: disabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: active (running) since Fri 2025-10-17 19:21:04 BST; 19h ago Invocation: c00e7a215dbd45f0ad4f15523e17b799 Main PID: 1537 (exim) Tasks: 1 (limit: 9005) Memory: 26.1M (peak: 35.6M) CPU: 18.391s CGroup: /system.slice/exim.service └─1537 /usr/sbin/exim -bd -q1h Oct 17 19:21:04 peterson.fenrir.org.uk systemd[1]: Starting exim.service - Exim Mail Transport Agent... Oct 17 19:21:04 peterson.fenrir.org.uk systemd[1]: Started exim.service - Exim Mail Transport Agent. sudo alternatives --config mta shows this: There is 3 program that provides 'mta'. Selection Command ----------------------------------------------- * 1 /usr/bin/sendmail.sendmail + 2 /usr/bin/sendmail.exim 3 /usr/bin/sendmail.exim Enter to keep the current selection[+], or type selection number: Hitting return results in this on checking the exim service status: systemctl status exim ● exim.service - Exim Mail Transport Agent Loaded: loaded (/usr/lib/systemd/system/exim.service; disabled; preset: disabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: active (running) since Fri 2025-10-17 19:21:04 BST; 19h ago Invocation: c00e7a215dbd45f0ad4f15523e17b799 Main PID: 1537 (exim) Tasks: 1 (limit: 9005) Memory: 26.1M (peak: 35.6M) CPU: 18.391s CGroup: /system.slice/exim.service └─1537 /usr/sbin/exim -bd -q1h Oct 17 19:21:04 peterson.fenrir.org.uk systemd[1]: Starting exim.service - Exim Mail Transport Agent... Oct 17 19:21:04 peterson.fenrir.org.uk systemd[1]: Started exim.service - Exim Mail Transport Agent. exim is now disabled, and I needed to manually enable it again. Looks like something to do with the alternatives package then.
awesome debugging folks, thanks.
This bug: https://bugzilla.redhat.com/show_bug.cgi?id=2338142 resulted in a chkconfig package change and this happened at about 11 days before I first noticed problems with exim upgrades failing to enable the service here: https://bugzilla.redhat.com/show_bug.cgi?id=2355049 Looks linked to me.
I tried to reproduce this but with no luck. If anyone has a minimal reproducer that would help a lot.
Following this guess: https://src.fedoraproject.org/rpms/postfix/pull-request/15
And also: https://src.fedoraproject.org/rpms/exim/pull-request/22 Both untested.
This part is likely the key to this problem: === There is 3 program that provides 'mta'. Selection Command ----------------------------------------------- * 1 /usr/bin/sendmail.sendmail + 2 /usr/bin/sendmail.exim 3 /usr/bin/sendmail.exim === Specifically, the way it has two /usr/bin/sendmail.exim entries for some reason. There's a loop in alternatives.c that (silently) disables the service for every alternative we think is *not* the current one, which matches what we're seeing pretty well: for (i = 0; i < set->numAlts; i++) { struct alternative *tmpalt = set->alts + i; if (tmpalt != alt && tmpalt->initscript) { if (isSystemd(tmpalt->initscript)) { asprintf(&path, "/bin/systemctl -q disable %s.service", tmpalt->initscript); so I suspect what's happening here is we're hitting that loop, it's seeing that the third alternative is not the 'current' one, and disabling it. We need to figure out why we get this duplication, I guess.
Brian, Bojan, could you see if you have individual executable file symlinks from /usr/bin to /usr/sbin or vice versa, or something like that, going on? ISTR that there's a thing where that can happen when you do the upgrade that implements the sbin merge, depending on the current state of the system. On a clean install, /usr/sbin is itself a symlink to /usr/bin , but what I'm remembering is that, on upgrade, it's possible to *not* wind up with that symlink but instead wind up with a bunch of: /usr/sbin/foo -> /usr/bin/foo /usr/sbin/bar -> /usr/bin/bar ...etc etc... symlinks.
Ah, yeah, the Change page for the sbin merge actually states that - https://fedoraproject.org/wiki/Changes/Unify_bin_and_sbin , under How To Test: "For upgraded systems, an attempt to merge the directories will be made during an upgrade. Package scriptlets will say: All files under /usr/sbin are symlinks; linking to ./bin... — the merge is done and the expected state is as above. /usr/sbin cannot be merged yet, found /usr/sbin/FILENAME — package providing /usr/sbin/FILENAME hasn't been updated. It is not an error, but the merge is not finished. /usr/sbin cannot be merged yet, /usr/sbin/FILENAME points to "..PATH — this is not expected to happen. A bug should be opened against the package providing /usr/sbin/FILENAME." so I rather suspect affected systems will be those that: 1. Have some updates-alternatives-enabled package which calls `alternatives --initscript` in a scriptlet 2. Got a /usr/sbin/foo -> /usr/bin/foo symlink for the relevant executable during upgrade to F42 I'll try and confirm that later, and see if I can come up with any kind of fix...
This might be stupid and I haven't tested this at all, but could this be a race between the update of the package and the update of alternatives? The alternatives in f41 consider the /bin/foo and /sbin/foo as two separate alternatives. In f42 and higher, it is considered to be the same file. Hence, if you first update a package to the 42 variant first and alternatives later you will get this result. exim is calling alternatives --install %{_sbindir}/sendmail mta %{_sbindir}/sendmail.exim 10 and from, the proposal Adjust %_sbindir in /usr/lib/rpm/macros (part of rpm package) to evaluate to %_bindir. Packages will be updated automatically during the mass rebuild.
My affected system, which has been upgraded to each new Fedora release since 2007 (oh it's been fun), has this: $ locate exim | grep /usr/bin /usr/bin/clamd.exim /usr/bin/exim /usr/bin/exim_checkaccess /usr/bin/exim_dbmbuild /usr/bin/exim_dumpdb /usr/bin/exim_fixdb /usr/bin/exim_lock /usr/bin/exim_tidydb /usr/bin/eximon /usr/bin/eximon.bin /usr/bin/eximstats /usr/bin/mailq.exim /usr/bin/newaliases.exim /usr/bin/rmail.exim /usr/bin/rsmtp.exim /usr/bin/runq.exim /usr/bin/sendmail.exim and $ locate exim | grep /usr/sbin /usr/sbin/clamd.exim /usr/sbin/exim /usr/sbin/exim_checkaccess /usr/sbin/exim_dbmbuild /usr/sbin/exim_dumpdb /usr/sbin/exim_fixdb /usr/sbin/exim_lock /usr/sbin/exim_tidydb /usr/sbin/eximon /usr/sbin/eximon.bin /usr/sbin/eximstats /usr/sbin/sendmail.exim however each of the above /usr/sbin/* which symlink to /usr/bin/ give this message with rpm -qf /usr/sbin/<filename>: file /usr/sbin/<filename> is not owned by any package I presume this means that the /usr/sbin directory will eventually disappear, as will /sbin which is symlinked to it, but until that happens the extra unwanted entry in the alternatives menu is going to persist unless action is taken to suppress it.
Thanks Brian! That goes with my theory so far... >This might be stupid and I haven't tested this at all, but could this be a race between the update of the package and the update of alternatives? With the reports of people hitting this on upgrade from F42 to F43, or just doing 'dnf reinstall', I don't think that's it, Lukas.
Discussed at the blocker review meeting on 20th Oct. 2025 AGREED RejectedFinalBlocker This is rejected as, based on further investigation, we're fairly sure it doesn't technically break the criteria, and isn't particularly specific to Fedora 43; at present we suspect a change to chkconfig's alternatives handling which has landed in both F42 and F43 interacts poorly with systems that were previously upgraded to Fedora 42 and have individual /usr/sbin/foo -> /usr/bin/foo symlinks rather than a /usr/sbin -> /usr/bin symlink. Accepted as an FE so we can land a fix during freeze if we find one and reduce the number of F43 upgrades/updates that may be affected. https://meetbot.fedoraproject.org/blocker-review_matrix_fedoraproject-org/2025-10-20/f43-blocker-review.2025-10-20-16.02.html
Here is what I have: # find /usr/sbin -type l /usr/sbin So, no individual symlinks.
Huh. Can we get your `alternatives --config` output?
]# alternatives --config mta There are 2 programs which provide 'mta'. Selection Command ----------------------------------------------- *+ 1 /usr/bin/sendmail.postfix 2 /usr/bin/sendmail.postfix
Which makes zero sense, of course. I haven't touched any of this by hand - I promise.
As a point of reference, my system has this: $ find /usr/sbin -type l | wc 1136 1136 23901 so there are a lot of links for it to get things wrong with.
Bojan: what are /bin and /sbin for you? Real directories or symlinks to /usr/bin and /usr/sbin ?
/usr/sbin is a symlink to /usr/bin
# ls -ld /usr/{s,}bin dr-xr-xr-x. 1 root root 34748 Oct 19 15:33 /usr/bin lrwxrwxrwx. 1 root root 3 May 19 18:34 /usr/sbin - > bin
yes, but that's not what I'm asking: I'm asking about /bin and /sbin (no /usr prefix). what are those? (i'm digging back to the /usr merge now)
# ls -ld /bin /sbin lrwxrwxrwx. 1 root root 7 Jul 30 10:00 /bin -> usr/bin lrwxrwxrwx. 1 root root 8 Jul 30 10:00 /sbin -> usr/sbin
From memory, merge of bin and sbin didn't go through the first time and there used to be a whole bunch of symlinks around. Again, from memory, I had to rerun some stuff (which I don't precisely remember) to get this done. This machine has been upgraded and cross-graded all the way from 32-bit Red Hat Linux 6.0, so the amount of cruft on it may be unusually high. 😁
Interesting to see that both Bojan and I have machines with a fairly long RedHat/Fedora history, mine I think initially had Fedora Core 7 on it but somewhere along the line it got a larger disk which had all the old partitions copied over to it and then /etc/fstab furgled to boot normally. No idea if this makes any difference. Adam, if you need more information from me, please ask, I can also provide the same for my other machine (which runs few services and has not apparently broken) that dates from 2020 and started out with Fedora 32. It also has a real /usr/sbin directory which is not a symlink to /usr/bin
well, the key data from any given system is the `alternatives --config <whatever>` output I guess. My working theory is that if that has multiple entries which are really the same thing (the one we want to be active), we're going to have problems; if not, we'll be fine. I'd obviously be interested in any cases which seem to disprove that theory. Assuming the theory is correct, the next step is figuring out exactly why we have those multiple entries (possibly multiple causes?) I did not get to this yet as I've got quite a lot of plates in the air today.
I can remove one of them and try the reinstall, if that test helps.
Edited /var/lib/alternatives/mta and removed duplicate postfix entries. Made sure mta alternative is in auto mode and that only one is remaining. Reinstall of postfix did not disable the service. Best guess, when bin and sbin got merged, the system ended up with two identical alternatives and that started disabling the service on each upgrade.
yeah, if the config is persistent then it could have got into a bad state when the dirs were not fully merged in the past, or during an upgrade (the kinda race Lukas was thinking about, in fact!) or something like that. I was going to look at the code for constructing the list of alternatives next, but hadn't got to it. I mean, I can see a way to fix this already - just have the 'service disable' loop skip entries that are 'the same' as the current entry, when adjusted for merged bin directories. But that's probably not the 'right' level to fix it. We probably want to get rid of entries that are duplicates from the list of alternatives itself. So I want to go look in that direction a bit, if nobody else beats me to it...
Hmm, before we spend too much time on the code in alternatives… What about the approach in the PRs from #c45 and #c46? There is a total of 4 packages which currently use 'alternatives --initscript' (postfix, exim, opensmtpd, sendmail), so it'd be trivial to patch them so. And in fact, I think that the '--initscript' support is not necessary for anything, since our Packaging Guidelines stipulate that systemd presets should be used to decide whether the package is enabled or not. And this is what is already happening through the standard scriptlets. So the whole secondary mechanism that is only used in a handful of packages and which clearly nobody understands should be dropped.
Looking at both my Fedora 43 systems, neither of them has /sbin symlinked to /usr/bin, there are quite a few installed packages that put binaries in /usr/sbin Is this all going to come out in the wash in the end or is it going to need manual intervention? I've just found all the files in /usr/sbin that are not owned by installed packages and deleted them but this still leaves some files there that mean I can't just delete /usr/sbin and symlink it to /usr/bin without needing to uninstall and reinstall those packages with uncertain consequences. I thought this was supposed to be handled automatically, or at least that packagers would upgrade their packages to fit in with the /bin /sbin merge.
Given nftables package seemed to do the same to me upgrading from F42 -> F43, is that the same root cause?
(In reply to Brian Morrison from comment #71) > Is this all going to come out in the wash in the end or is it going to need > manual intervention? In general, the conversion should finish automatically. But if there are non-packaged files there, the final conversion might need some manual intervention. But if that _doesn't_ happen, the system should remain fully functional. Currently, AFAIK, all packages have been updated to move the files from /usr/sbin. But some packages are FTBFS and there is not much we can do about those until they become buildable again. > I've just found all the files in /usr/sbin that are not owned by installed > packages and deleted them but this still leaves some files there that mean I > can't just delete /usr/sbin and symlink it to /usr/bin without needing to > uninstall and reinstall those packages with uncertain consequences. I > thought this was supposed to be handled automatically, or at least that > packagers would upgrade their packages to fit in with the /bin /sbin merge. It's hard to say without knowing the specifics. If you have a case where the merge is not finalized (i.e. /usr/sbin is not turned into a symlink), please report it is a bug, with the details of what files are in /usr/sbin. (In reply to Steven Haigh from comment #72) > Given nftables package seemed to do the same to me upgrading from F42 -> > F43, is that the same root cause? I suspect nftables has some different cause.
> Currently, AFAIK, all packages have been updated to move the files from > /usr/sbin. But some packages are FTBFS and there is not much we can do > about those until they become buildable again. If a package such as udisks (for example) is buildable but currently puts a file in /sbin then it seems to have slipped through the net somehow. Both my systems have it installed and the most recent build is from July 2025. >> I've just found all the files in /usr/sbin that are not owned by installed >> packages and deleted them but this still leaves some files there that mean I >> can't just delete /usr/sbin and symlink it to /usr/bin without needing to >> uninstall and reinstall those packages with uncertain consequences. I >> thought this was supposed to be handled automatically, or at least that >> packagers would upgrade their packages to fit in with the /bin /sbin merge. > It's hard to say without knowing the specifics. If you have a case where the > merge is not finalized (i.e. /usr/sbin is not turned into a symlink), please > report it is a bug, with the details of what files are in /usr/sbin. Which package do I file this as a bug against? I'm not sure how to do this for something general such as a directory merge.
> udisks This one slipped through the cracks. https://src.fedoraproject.org/rpms/udisks/pull-request/1 should fix the issue. > Which package do I file this as a bug against? Against the package that has the files in unexpected placed. But put me in CC.
> Hmm, before we spend too much time on the code in alternatives… > What about the approach in the PRs from #c45 and #c46? > There is a total of 4 packages which currently use 'alternatives --initscript' > (postfix, exim, opensmtpd, sendmail), so it'd be trivial to patch them so. Oh, if there are definitely only four packages which use it, then sure, we can just patch those. I was working under the assumption there might be many more packages we might run into this with. It's still a bug in alternatives, but we could probably just file it and leave it to the maintainers to decide if they care enough to fix it. > And in fact, I think that the '--initscript' support is not necessary for > anything, since our Packaging Guidelines stipulate that systemd presets should > be used to decide whether the package is enabled or not. And this is what is > already happening through the standard scriptlets. So the whole secondary > mechanism that is only used in a handful of packages and which clearly > nobody understands should be dropped. well, remember this mechanism exists outside of packaging. The *idea* is that you can swap between 'providers' at will. So this code won't only "fire" during package updates; if you decide to manually switch from postfix to exim, or whatever, it should kick in. And I guess if you had postfix service enabled, when you switch to exim, you want the postfix service disabled and the exim service enabled... I would be all in favor of dropping alternatives entirely, but I think the few times it's been proposed enough people have objected that it didn't happen.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #75) > > udisks > This one slipped through the cracks. > https://src.fedoraproject.org/rpms/udisks/pull-request/1 > should fix the issue. > > > Which package do I file this as a bug against? > Against the package that has the files in unexpected placed. > But put me in CC. Right, I have done this for all the /sbin binaries and their associated packages that I have here. I doubt it's an exhaustive list but it's a start.
Having looked into this a bit further, I don't think Zbigniew's proposal to drop the `--initscript` arg from the specs is correct. We really *do* want update-alternatives to handle service enablement for this case; if you use the alternatives system to switch from e.g. postfix to exim as your "system MTA", you *want* it to disable the postfix service and enable the exim service. If it doesn't do that, things will be inconsistent. The issue isn't exactly "alternatives somehow gets confused about enablement state" as Zbyszek wrote in the PRs. The name of the arg maybe makes it look 'old', but it's a misnomer, alternatives does use systemd where appropriate. The bug is really just this thing about us getting multiple entries for the same file somehow (likely to do with the sbin merge but we haven't traced *exactly* how it happens yet) interacting badly with the loop in alternatives that disables services for entries that aren't the 'current' one. The best fix would, I guess, be for alternatives to look for exact duplicate entries when parsing the config file, discard them, and print a warning that they shouldn't be there? (or just edit them out of the config file itself?) I'm currently trying to understand the source well enough to propose a fix for that (but it's very...C-ish C, so it's taking me a while). A quicker easier hack would be just to skip items in the 'service disablement' loop if they have the same effective binary path as the 'current' entry, not just if they *are* the current entry. I think I can probably write that, if I can't write the first thing. That should at least solve the biggest practical problem, though there are probably other odd consequences of having multiple identical entries like this.
Managed to come up with a small patch for alternatives that fixes it at least in my basic testing.
Oh, forgot to mention - there's a scratch build at https://koji.fedoraproject.org/koji/taskinfo?taskID=138410452 ; if you're affected by this, try it out and post here and/or on the github PR whether it works (and if you notice it introduces any problems).
Upgraded to the scratch build. Then: $ sudo alternatives --config mta There is 2 program that provides 'mta'. Selection Command ----------------------------------------------- * 1 /usr/bin/sendmail.sendmail + 2 /usr/bin/sendmail.exim Enter to keep the current selection[+], or type selection number: bdm@peterson:~$ systemctl status exim ● exim.service - Exim Mail Transport Agent Loaded: loaded (/usr/lib/systemd/system/exim.service; enabled; preset: disabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: active (running) since Sun 2025-10-19 22:37:50 BST; 5 days ago Invocation: 80c6585507ee4b3c866a3da05dfed3b2 Main PID: 1557 (exim) Tasks: 1 (limit: 9005) Memory: 46.6M (peak: 83.2M) CPU: 2min 26.049s CGroup: /system.slice/exim.service └─1557 /usr/sbin/exim -bd -q1h Oct 19 22:37:49 peterson.fenrir.org.uk systemd[1]: Starting exim.service - Exim Mail Transport Agent... Oct 19 22:37:50 peterson.fenrir.org.uk systemd[1]: Started exim.service - Exim Mail Transport Agent. This version appears to work correctly here, there are only 2 alternatives shown and exim remains enabled when selecting the current choice with <return>
Awesome. Does `dnf reinstall`ing exim leave it enabled too?
$ sudo dnf reinstall exim [sudo] password for xxxx: Updating and loading repositories: balena-etcher-source 100% | 3.2 KiB/s | 2.1 KiB | 00m01s balena-etcher 100% | 3.3 KiB/s | 2.1 KiB | 00m01s balena-etcher-noarch 100% | 3.2 KiB/s | 2.1 KiB | 00m01s Repositories loaded. Package Arch Version Repository Size Reinstalling: exim x86_64 4.98.2-4.fc43 fedora 4.9 MiB replacing exim x86_64 4.98.2-4.fc43 fedora 4.9 MiB Transaction Summary: Reinstalling: 1 package Replacing: 1 package Total size of inbound packages is 1 MiB. Need to download 1 MiB. After this operation, 0 B extra will be used (install 5 MiB, remove 5 MiB). Is this ok [y/N]: y [1/1] exim-0:4.98.2-4.fc43.x86_64 100% | 2.4 MiB/s | 1.5 MiB | 00m01s -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [1/1] Total 100% | 1.3 MiB/s | 1.5 MiB | 00m01s Running transaction [1/4] Verify package files 100% | 7.0 B/s | 1.0 B | 00m00s [2/4] Prepare transaction 100% | 0.0 B/s | 2.0 B | 00m05s >>> Running sysusers scriptlet: exim-0:4.98.2-4.fc43.x86_64 >>> Finished sysusers scriptlet: exim-0:4.98.2-4.fc43.x86_64 >>> Scriptlet output: >>> /usr/lib/sysusers.d/trousers.conf:1: Conflict with earlier configuration for user 'tss' in /usr/lib/sysusers.d/tpm2-tss.conf:2, ignoring line. >>> [3/4] Reinstalling exim-0:4.98.2-4.fc43.x86_64 100% | 2.6 MiB/s | 4.9 MiB | 00m02s [4/4] Removing exim-0:4.98.2-4.fc43.x86_64 100% | 7.0 B/s | 103.0 B | 00m13s >>> Running %triggerin scriptlet: systemd-0:258.1-1.fc43.x86_64 >>> Finished %triggerin scriptlet: systemd-0:258.1-1.fc43.x86_64 >>> Scriptlet output: >>> /usr/lib/sysusers.d/trousers.conf:1: Conflict with earlier configuration for user 'tss' in /usr/lib/sysusers.d/tpm2-tss.conf:2, ignoring line. >>> Complete! bdm@peterson:~$ systemctl status exim ● exim.service - Exim Mail Transport Agent Loaded: loaded (/usr/lib/systemd/system/exim.service; enabled; preset: disabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: active (running) since Sat 2025-10-25 17:17:25 BST; 14s ago Invocation: d764f7b809534ad4af7ecfa7d736b325 Main PID: 19618 (exim) Tasks: 1 (limit: 9004) Memory: 4.3M (peak: 8.2M) CPU: 141ms CGroup: /system.slice/exim.service └─19618 /usr/sbin/exim -bd -q1h Oct 25 17:17:25 peterson.fenrir.org.uk systemd[1]: Starting exim.service - Exim Mail Transport Agent... Oct 25 17:17:25 peterson.fenrir.org.uk systemd[1]: Started exim.service - Exim Mail Transport Agent. So yes, the exim service remains enabled.
Since this is a pretty bad bug for affected folks, and testing seems to indicate my fix is working, and there's been no upstream or downstream response to my PR yet, I'm sending official builds with the patch backported, using provenpackager privileges. I'll create updates but will set them not to auto-push stable for now, to make sure there's time to catch any problems or errors in my fix.
FEDORA-2025-bbaef37b87 (chkconfig-1.33-3.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-bbaef37b87
FEDORA-2025-aa88a0d00b (chkconfig-1.33-3.fc43) has been submitted as an update to Fedora 43. https://bodhi.fedoraproject.org/updates/FEDORA-2025-aa88a0d00b
FEDORA-2025-aa88a0d00b has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-aa88a0d00b` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-aa88a0d00b See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-bbaef37b87 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-bbaef37b87` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-bbaef37b87 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Fwiw, after upgrading from F42 to F43 I found that nftables.service, which had been enabled before, was now disabled.
(In reply to Thomas Köller from comment #89) > Fwiw, after upgrading from F42 to F43 I found that nftables.service, which > had been enabled before, was now disabled. Adam mentioned elsewhere that this is because nftables has been split up into multiple packages and as a result the upgrade to F43 treated it as a new package for which the default service setting is disabled. He explained it a little more elegantly than I have.
Yeah, that's https://bugzilla.redhat.com/show_bug.cgi?id=2401676 . separate issue.
I upgraded my NAS on 3rd of November to F43 and postfix.service was disabled again. I noticed it yesterday night, since my surveillance camera which relies on sending mails, did not record anymore. So the issue is still present.
Created attachment 2113201 [details] dnf5 system-upgrade F42>F43
Yes, my F43 upgrade also disabled postfix. I had a few days prior upgraded alternative/chkconfig as per #88, and at that point `alternatives --config mta` listed only /usr/bin/sendmail.postfix (just once). However after the dnf upgrade to F43, the postfix was disabled and alternatives lists /usr/bin/sendmail.postfix twice. Will upload logs, let me know if I can help debug.
Created attachment 2113202 [details] dnf system-upgrade
Created attachment 2113203 [details] dnf upgrade advisory
Created attachment 2113206 [details] upgrade journalctl
Can you post the contents of/var/lib/alternatives here? I finally got to this and started writing a patch, but I haven't tested it yet, and it would be nice to have real files to work with.
Created attachment 2113694 [details] /var/lib/alternatives/mta as requested, here is an alternatives mta file featuring 2 postfix sections
looks similar on my system: root@nas:~# cat /var/lib/alternatives/mta auto /usr/bin/sendmail mta-pam /etc/pam.d/smtp mta-mailq /usr/bin/mailq mta-newaliases /usr/bin/newaliases mta-rmail /usr/bin/rmail mta-sendmail /usr/lib/sendmail mta-mailqman /usr/share/man/man1/mailq.1.gz mta-newaliasesman /usr/share/man/man1/newaliases.1.gz mta-aliasesman /usr/share/man/man5/aliases.5.gz mta-sendmailman /usr/share/man/man8/sendmail.8.gz mta-smtpman /usr/share/man/man8/smtp.8.gz mta-smtpdman /usr/share/man/man8/smtpd.8.gz /usr/bin/sendmail.postfix 60 postfix /etc/pam.d/smtp.postfix /usr/bin/mailq.postfix /usr/bin/newaliases.postfix /usr/bin/rmail.postfix /usr/lib/sendmail.postfix /usr/share/man/man1/mailq.postfix.1.gz /usr/share/man/man1/newaliases.postfix.1.gz /usr/share/man/man5/aliases.postfix.5.gz /usr/share/man/man1/sendmail.postfix.1.gz /usr/share/man/man8/smtp.postfix.8.gz /usr/share/man/man8/smtpd.postfix.8.gz /usr/bin/sendmail.postfix 60 postfix /etc/pam.d/smtp.postfix /usr/bin/mailq.postfix /usr/bin/newaliases.postfix /usr/bin/rmail.postfix /usr/lib/sendmail.postfix /usr/share/man/man1/mailq.postfix.1.gz /usr/share/man/man1/newaliases.postfix.1.gz /usr/share/man/man5/aliases.postfix.5.gz /usr/share/man/man1/sendmail.postfix.1.gz /usr/share/man/man8/smtp.postfix.8.gz /usr/share/man/man8/smtpd.postfix.8.gz
chmmm weird, I was expecting /usr/bin/sendmail.postfix and /usr/sbin/sendmail.postfix and not two /bin/
Yes, mine the same as #100. No sbin.
The updates that fix this are not stable yet, so unless you have updates-testing enabled or you cherry-picked them, that is unfortunately expected :/ I was waiting for more input from upstream on this. But it's been a while, so I'll ask on the upstream issue whether the maintainer thinks it's OK to push the updates stable as-is, and do it soon unless they say no.
(In reply to Adam Williamson from comment #103) > The updates that fix this are not stable yet, so unless you have > updates-testing enabled or you cherry-picked them, that is unfortunately > expected :/ Should installing advisory=FEDORA-2025-bbaef37b87 prior to upgrading have been enough to expect correct behaviour? If so, see my comment #94.
(In reply to Mark Knoop from comment #104) > (In reply to Adam Williamson from comment #103) > > The updates that fix this are not stable yet, so unless you have > > updates-testing enabled or you cherry-picked them, that is unfortunately > > expected :/ > > Should installing advisory=FEDORA-2025-bbaef37b87 prior to upgrading have > been enough to expect correct behaviour? If so, see my comment #94. If you had not enabled updates-testing then the F43 chkconfig package will be the one from updates.
Adam, the latest update is that Lukáš was working on a patch last week, but it's not ready yet. So maybe wait with pushing the update a bit more. Sorry for the delay.
> Adam, the latest update is that Lukáš was working on a patch last week, but it's not ready yet. So maybe wait with pushing the update a bit more. Sorry for the delay. He already said in the upstream issue that it was fine to push the updates stable, so I have. > Should installing advisory=FEDORA-2025-bbaef37b87 prior to upgrading have been enough to expect correct behaviour? If so, see my comment #94. sort of, but the upgrade to Fedora 43 *probably* downgraded chkconfig to the unpatched current-stable F43 version. If you check what you have installed right now it's probably chkconfig-1.33-1.fc43 , I would guess? That's an unpatched version.
FEDORA-2025-aa88a0d00b (chkconfig-1.33-3.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-bbaef37b87 (chkconfig-1.33-3.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.