Description of problem: The systemd-249.11-1.fc35 causes anaconda to traceback when @core group is being installed. Version-Release number of selected component (if applicable): systemd-249.11-1.fc35 How reproducible: Deterministic. Steps to Reproduce: 1. Try to provision Fedora 35 with the default (just @core) package set. Actual results: Console output ends with An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details Anaconda log ends with 12:50:47,271 CRT exception: Traceback (most recent call last): File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run threading.Thread.run(self) File "/usr/lib64/python3.10/threading.py", line 946, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.10/site-packages/pyanaconda/installation.py", line 415, in run_installation queue.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 496, in start self.run_task() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 458, in run_task self._task(*self._task_args, **self._task_kwargs) File "/usr/lib64/python3.10/site-packages/pyanaconda/network.py", line 293, in write_configuration sync_run_task(task_proxy) File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/__init__.py", line 46, in sync_run_task task_proxy.Finish() File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 444, in _call_method return self._get_method_reply( File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 477, in _get_method_reply return self._handle_method_error(error) File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 497, in _handle_method_error raise exception from None dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf' 12:50:47,273 DBG exception: Gtk cannot be initialized 12:50:47,273 DBG exception: In the main thread, running exception handler 12:51:02,933 INF core.util: Running kickstart %%onerror script(s) 12:51:14,215 INF core.util: All kickstart %%onerror script(s) have been run 12:51:14,215 INF kickstart.script: Running kickstart %%traceback script(s) 12:51:14,215 INF kickstart.script: All kickstart %%traceback script(s) have been run 12:51:14,215 INF core.util: Reporting the IPMI event: 10 Expected results: No traceback, Fedora 35 installs. Additional info: This is a regression against systemd-249.9-1.fc35.
Please raise the priority, as it affects F35 netinstalls... the current Fedora *stable* release.
*** Bug 2074122 has been marked as a duplicate of this bug. ***
From the other bug: During a routine upgrade, systemd-resolved printed this : Running scriptlet: systemd-resolved-249.11-1.fc35.x86_64 69/69 '/etc/resolv.conf' -> '../run/systemd/resolve/stub-resolv.conf' but without creating /run/systemd/resolve/stub-resolv.conf -- probably because the directory /run/systemd/resolve does not exist. Result : loss of file /etc/resolv.conf. I don't need systemd-resolved. Perhaps systemd-resolved should interpret the absence of /run/systemd/resolve as a cue to abstain.
OK, I know what is happening. There were two patches applied to Anaconda (https://github.com/rhinstaller/anaconda/pull/3884), but apparently they didn't make it into F35, but only into F36. @rvykydal what should we do here? Also pull in the patches for Anaconda or revert that change in systemd? I guess the later is the correct action. I'll start a build with that chunk reverted.
I'd agree that systemd needs to change/revert. We cannot rely on any changes to anaconda -- Fedora 35 needs to be able to provision with the GA ISOs (and updates repository enabled).
FEDORA-2022-1bf06aa328 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bf06aa328
(In reply to Zbigniew Jędrzejewski-Szmek from comment #5) > OK, I know what is happening. There were two patches applied to Anaconda > (https://github.com/rhinstaller/anaconda/pull/3884), > but apparently they didn't make it into F35, but only into F36. > > @rvykydal what should we do here? Also pull in the patches for > Anaconda or revert that change > in systemd? I guess the later is the correct action. I'll start a build with > that chunk reverted. Unfortunately, yes we need the revert the change. Jan is right in comment #6.
To explain, we added the fix to F35 anaconda for eventual respins but the fix didn't make it to F35 GOLD installer (https://bugzilla.redhat.com/show_bug.cgi?id=2019579#c30)
Where can the progress of the revert be tracked so that we know when to remove work-arounds in kickstart installs?
Since this bugzilla is linked to the errata with the build with the fix (see comment 7), status of this bugzilla will get updated accordingly as the errata becomes available in testing and then in stable repo.
*** Bug 2074907 has been marked as a duplicate of this bug. ***
FEDORA-2022-1bf06aa328 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-1bf06aa328` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bf06aa328 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
So the fixed build never made it to stable, because it seems that it broke something else (also involving the problematic symlink). In the meantime I'm installing F35 without updates enabled, which is a bit suboptimal. Is there any better workaround? I could put the unpushed update in a local repository but it was unpushed for a reason....
systemd-249.11-2.fc35 new bug: https://bugzilla.redhat.com/show_bug.cgi?id=2075428
*** Bug 2075612 has been marked as a duplicate of this bug. ***
*** Bug 2077582 has been marked as a duplicate of this bug. ***
*** Bug 2077890 has been marked as a duplicate of this bug. ***
FEDORA-2022-1bf06aa328 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
I don't see this issue fixed with systemd-249.11-2.fc35.x86_64. The installation still fails with dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf'
*** Bug 2078950 has been marked as a duplicate of this bug. ***
Trying to run some tests on Fedora 35 and I am still seeing this issue on Fedora 35 network installs.
That is very strange. I updated the scriptlet to be a bit more robust. Let's see if this next version works better.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #25) > That is very strange. I updated the scriptlet to be a bit more robust. Let's > see if this next > version works better. Zbigniew, What version of systemd has the fix?
That'll most likely be systemd-249.12-1.fc35.
FEDORA-2022-01079468a3 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-01079468a3
The systemd update is failing openQA tests. It seems like trying to build a Workstation live image with this systemd included fails, due to scriptlet errors in systemd-resolved: 18:53:08,525 INF dnf.rpm: /var/tmp/rpm-tmp.jzmyWP: line 18: mkdir: command not found /var/tmp/rpm-tmp.jzmyWP: line 19: /var/lib/rpm-state/systemd/systemd-resolved-initial-installation: No such file or directory warning: %post(systemd-resolved-249.12-1.fc35.x86_64) scriptlet failed, exit status 1 18:53:08,528 ERR dnf.rpm: Error in POSTIN scriptlet in rpm package systemd-resolved 18:53:08,680 INF dnf.rpm: /var/tmp/rpm-tmp.xOgK8A: line 2: ln: command not found
I think that last line's a stray, sorry. The problem is that the attempt to fix things here added an `mkdir` call to systemd-resolved %post, but systemd-resolved does not Requires(post): coreutils . It probably needs to. Just hope that doesn't cause any circular dependencies.
Oh right. I'm doing a build without 'ls' or 'mkdir' now, pure 'sh'. If that doesn't work, the next step will be to switch to %lua. (I don't want to add a dependency on coreutils because even if that works here it might cause dependency loops in other scenarios, and it's hard to figure out what might break.)
FEDORA-2022-01079468a3 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-01079468a3` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-01079468a3 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
systemd 249.12-2.fc35 still fails during Fedora 35 installation. I have tried a kickstart installation for Fedora 35 using an additional local repository where I have downloaded the packages systemd 249.12-2.fc35 from koji build 1957690. After installation of the packages the installation stops. anaconda.log contains the following error messages: 16:00:12,348 INF threading: Thread Failed: AnaInstallThread (140677780940352) 16:00:12,348 DBG exception: running handleException 16:00:12,356 CRT exception: Traceback (most recent call last): File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run threading.Thread.run(self) File "/usr/lib64/python3.10/threading.py", line 946, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.10/site-packages/pyanaconda/installation.py", line 415, in run_installation queue.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 496, in start self.run_task() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 458, in run_task self._task(*self._task_args, **self._task_kwargs) File "/usr/lib64/python3.10/site-packages/pyanaconda/network.py", line 293, in write_configuration sync_run_task(task_proxy) File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/__init__.py", line 46, in sync_run_task task_proxy.Finish() File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 444, in _call_method return self._get_method_reply( File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 477, in _get_method_reply return self._handle_method_error(error) File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 497, in _handle_method_error raise exception from None dasbus.error.DBusError: [Errno 2] Datei oder Verzeichnis nicht gefunden: '/mnt/sysroot/etc/resolv.conf' 16:00:12,359 DBG exception: Gtk running, queuing exception handler to the main loop 16:00:12,359 INF threading: Thread Done: AnaInstallThread (140677780940352)
I assume that "No such file or directory" is reported by trying to open a dangling symlink. Let's summarize: With systemd-249.9-1.fc35, we had #2032085 — Some variants are missing /etc/resolv.conf symlink, which actually meant that a static file would be created by NetworkManager and copied by Anaconda into the installed image. So in systemd-249.11-1.fc35, the symlink started being created by systemd-resolved scriptlets during installation. But this confused Anaconda, and Anaconda got patches in F36 to not try to handle /etc/resolv.conf itself. Then there were bugs in the scriptlets, and issues with dependencies being underspecified for scriplets, and dependency loops, and rpm not installing dependencies even when they are specified correctly and could be installed, in short a big mess that took a while to pound into shape. With systemd-249.12-2.fc35, the symlink is correctly created in all circumstances, but Anaconda gets confused because the symlink in the chroot is dangling because it's missing the patches to not touch it, and we can't update Anaconda. We could create an *absolute* symlink instead: Anaconda would try to read the symlink and load the file from the "host", i.e. it's livecd environment. But this falls flat because systemd-resolved is not installed in the F35 live image :( I don't have any more nice ideas. I'll try to implement a brute-force approach: a one time service to remove static /etc/resolv.conf that is created by NetworkManager after reboot. If anyone else has better ideas, please speak up.
*** Bug 2080455 has been marked as a duplicate of this bug. ***
I will delete NetworkManager after first boot (if it's installed). Then use systemd-networkd and systemd-resolved. So I don't want Anaconda generate any network setting for me. Leave it here (a dangling symlink) is the best result for me.
I updated the update with another build. This time a link ../usr/lib/systemd/resolv.conf instead of ../run/systemd/resolve/resolv.conf is created, so Anaconda should be happy. Most users don't configure own resolvers, so they shouldn't care about the difference.
If people who tested the previous iterations could also test this, that'd be great. I'm going away until Tuesday (Labour Day and Constitution Day here in Poland), and I'm not taking my computer with me, so I won't work on this for a few days.
Just test again in my VM: You can create empty /mnt/sysroot/run/systemd/resolve/stub-resolv.conf file (need mkdir run/systemd/resolve). Because systemd-resolved is enabled by default, it will update this file with right content after reboot. NetworkManager is not required here.
(In reply to 沙包妖梦 from comment #39) > You can create empty /mnt/sysroot/run/systemd/resolve/stub-resolv.conf file > (need mkdir run/systemd/resolve). > Because systemd-resolved is enabled by default, it will update this file > with right content after reboot. NetworkManager is not required here. I guess that'd work too. I'm not sure if /run is mounted in the installroot. If it is, then that'd actually work pretty nicely, because the file would go away automatically. If it's not, and we created the file, we would get a warning after reboot about /run containing files. We could mount /run in the installroot and then create the directories and file, but that becomes quite complex. So let's see if the current approach works… If it does, maybe we can iterate on it with your idea.
Hi There, I am still having this issue as of this morning using both Netinstall and full install. Are there any workaround for this on a clean install? Should I just use F36 instead? What are the recommendations here, I need to install ASAP. Many thanks in advance P.S. If this is not the right place for this message please advise, I am new at this. Thank you!
I'm also having this problem when attempting to install in a VM at the OVHCloud provider. They don't have a kickstart install option, only "mount ISO", so how do I do a proper install? If the new anaconda package was released yesterday, why doesn't "install from local mirror" pick it up? Is there a way to fix this in an alternate terminal during install? Perhaps even download and install the updated anaconda package after it fails at the end of the install?
I solved this problem by doing the following: - Set up the install as normal (choose disk layout, software, network setup, etc) - Immediately after clicking "begin installation", switch over to tty3 (ctrl-alt F3) - Run "df -h" until you see that /mnt/sysroot is mounted - Run "cd /mnt/sysroot; mkdir -p run/systemd/resolve; touch run/systemd/resolve/stub-resolv.conf; cd /" You can then switch back to tty6 to watch the install finish successfully.
I have still the same error with systemd-249.12-3.fc35 .
My workaround was to install using fedora server dvd iso, I unplugged ethernet cable before install. Then installation completed successfully. Afterward connected ethernet, configured internet and applied the updates. Couldn't net netinst iso to work. Off topic: The patch mentioned in other thread, If I apply using inst.updates= boot flag, installer fails to start in bare metal hardware. Worked in VM though :/
(In reply to Sarim Khan from comment #45) > My workaround was to install using fedora server dvd iso, I unplugged > ethernet cable before install. Then installation completed successfully. > Afterward connected ethernet, configured internet and applied the updates. You don't need to unplug the ethernet cable. Just don't include updates repository in the kickstart configuration. For example, comment it: #repo --name=updates
(In reply to Alex Regan from comment #42) > > If the new anaconda package was released yesterday, why doesn't "install > from local mirror" pick it up? Because anaconda which executes is anaconda on that ISO, not anaconda on the mirror. On a typical installation, anaconda in the repositories does not play any role.
*** Bug 2081020 has been marked as a duplicate of this bug. ***
Provisioning Fedora 35 is still broken with updates-testing repo enabled and systemd-249.12-3.fc35.x86_64 installed from it.
FEDORA-2022-01079468a3 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
(In reply to Fedora Update System from comment #52) > FEDORA-2022-01079468a3 has been pushed to the Fedora 35 stable repository. > If problem still persists, please make note of it in this bug report. All, This issue persists. tested today with Fedora-35 Everything aarch64 ---%<-snip->%--- Verifying systemd-libs.aarch64 (361/379) Verifying systemd-networkd.aarch64 (362/379) Verifying systemd-oomd-defaults.noarch (363/379) Verifying systemd-pam.aarch64 (364/379) Verifying systemd-resolved.aarch64 (365/379) Verifying systemd-udev.aarch64 (366/379) Verifying tzdata.noarch (367/379) Verifying udisks2.aarch64 (368/379) Verifying util-linux.aarch64 (369/379) Verifying util-linux-core.aarch64 (370/379) Verifying vim-data.noarch (371/379) Verifying vim-minimal.aarch64 (372/379) Verifying whois-nls.noarch (373/379) Verifying xz.aarch64 (374/379) Verifying xz-libs.aarch64 (375/379) Verifying yum.noarch (376/379) Verifying zchunk-libs.aarch64 (377/379) Verifying zram-generator.aarch64 (378/379) Verifying zram-generator-defaults.noarch (379/379) . Performing post-installation setup tasks . Configuring installed system . . . . . . . . . . . . . Writing network configuration An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details ine 496, in start self.run_task() File "/usr/lib64/python3.10/site-packages/pyanaconda/installationaconda/installages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation.py", line 415, in run_installation queue.start() File "/usr/lib64/python3.10/threading.py", line 946, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run threading.Thread.run(self) dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf' What do you want to do now? 1) Report Bug 2) Debug 3) Run shell 4) Quit Please make your choice from above: ---%<-snip->%--- Can this issue have a higher prioritized, please. Why... Beaker automated testing with Fedora 35 is crippling systems. Its leaving the bootorder wrong and marking systems broken in the double digits. Alot of time is being spent by sysadmins scrambling to correct this issue :/ Best, pbunyan
I have no other info to give instead of agreeing that this needs to be fixed, even if the changes in the systemd package got reverted to the systemd-249.9-1.fc35 state (which is what comment 0 says was a working NVR).
(In reply to Jan Pazdziora from comment #54) > I have no other info to give instead of agreeing that this needs to be > fixed, even if the changes in the systemd package got reverted to the > systemd-249.9-1.fc35 state (which is what comment 0 says was a working NVR). Thank you for quick reply, Jan. fwiw... Fedora36 installs work fine in Beaker. [root@ampere-hr330a-06 ~]# uname -a Linux ampere-hr330a-06.khw4.lab.eng.bos.redhat.com 5.17.9-300.fc36.aarch64 #1 SMP Wed May 18 14:50:13 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux [root@ampere-hr330a-06 ~]# cat /etc/redhat-release Fedora release 36 (Thirty Six) [root@ampere-hr330a-06 ~]# yum list installed | grep systemd rpm-plugin-systemd-inhibit.aarch64 4.17.0-10.fc36 @beaker-Fedora-Everything systemd.aarch64 250.3-8.fc36 @beaker-Fedora-Everything systemd-libs.aarch64 250.3-8.fc36 @beaker-Fedora-Everything systemd-networkd.aarch64 250.3-8.fc36 @beaker-Fedora-Everything systemd-oomd-defaults.noarch 250.3-8.fc36 @beaker-Fedora-Everything systemd-pam.aarch64 250.3-8.fc36 @beaker-Fedora-Everything systemd-resolved.aarch64 250.3-8.fc36 @beaker-Fedora-Everything systemd-udev.aarch64 250.3-8.fc36 @beaker-Fedora-Everything
How do we address the fact that this is crippling aarch64 systems when beaker jobs are calling for Fedora 35? The kernel-hw queue had 40 or so broken systems that were started by CKI.
(In reply to PaulB from comment #55) > (In reply to Jan Pazdziora from comment #54) > > I have no other info to give instead of agreeing that this needs to be > > fixed, even if the changes in the systemd package got reverted to the > > systemd-249.9-1.fc35 state (which is what comment 0 says was a working NVR). > > Thank you for quick reply, Jan. > > fwiw... Fedora36 installs work fine in Beaker. > > [root@ampere-hr330a-06 ~]# uname -a > Linux ampere-hr330a-06.khw4.lab.eng.bos.redhat.com 5.17.9-300.fc36.aarch64 > #1 SMP Wed May 18 14:50:13 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux > [root@ampere-hr330a-06 ~]# cat /etc/redhat-release > Fedora release 36 (Thirty Six) > [root@ampere-hr330a-06 ~]# yum list installed | grep systemd > rpm-plugin-systemd-inhibit.aarch64 4.17.0-10.fc36 > @beaker-Fedora-Everything > systemd.aarch64 250.3-8.fc36 > @beaker-Fedora-Everything > systemd-libs.aarch64 250.3-8.fc36 > @beaker-Fedora-Everything > systemd-networkd.aarch64 250.3-8.fc36 > @beaker-Fedora-Everything > systemd-oomd-defaults.noarch 250.3-8.fc36 > @beaker-Fedora-Everything > systemd-pam.aarch64 250.3-8.fc36 > @beaker-Fedora-Everything > systemd-resolved.aarch64 250.3-8.fc36 > @beaker-Fedora-Everything > systemd-udev.aarch64 250.3-8.fc36 > @beaker-Fedora-Everything Rawhide works too: [root@ampere-hr330a-06 ~]# uname -a Linux ampere-hr330a-06.khw4.lab.eng.bos.redhat.com 5.18.0-60.fc37.aarch64 #1 SMP PREEMPT_DYNAMIC Mon May 23 13:52:56 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux [root@ampere-hr330a-06 ~]# date Wed May 25 12:35:10 PM EDT 2022 [root@ampere-hr330a-06 ~]# cat /etc/redhat-release Fedora release 37 (Rawhide) root@ampere-hr330a-06 ~]# yum list installed | grep systemd rpm-plugin-systemd-inhibit.aarch64 4.18.0-0.alpha2.1.fc37 @beaker-Fedora-Everything systemd.aarch64 251-1.fc37 @beaker-Fedora-Everything systemd-libs.aarch64 251-1.fc37 @beaker-Fedora-Everything systemd-networkd.aarch64 251-1.fc37 @beaker-Fedora-Everything systemd-oomd-defaults.noarch 251-1.fc37 @beaker-Fedora-Everything systemd-pam.aarch64 251-1.fc37 @beaker-Fedora-Everything systemd-resolved.aarch64 251-1.fc37 @beaker-Fedora-Everything systemd-udev.aarch64 251-1.fc37 @beaker-Fedora-Everything Issue seems to be isolated to aarch64 and F35 :/
(In reply to PaulB from comment #57) > > Issue seems to be isolated to aarch64 and F35 :/ It is isolated to Fedora 35 as mentioned in 5. However, it is not ARM specific, it affects all architectures.
(In reply to Arthur Benoit from comment #56) > How do we address the fact that this is crippling aarch64 systems when > beaker jobs are calling for Fedora 35? > The kernel-hw queue had 40 or so broken systems that were started by CKI. To set the record straight, CKI works around this problem by running rawhide. So just for once we were not to blame for these particular broken machines 🙈.
I don't quite understand the discussion here but am I correct that the bug still exists? Ie, the everything netinst iso is still broken. I see comments about new builds being made available for testing. I'd like to help test. Can someone provide some pointers on how to go about downloading the "fixed" isos for testing? Thanks.
People keep asking when this is going to be "fixed". The problem is we've tried to fix it four times and it hasn't worked. Nobody has come up with a way to fully "fix" it in all cases yet. The problem is that there are various cases: 1. You run an install with all release-day packages (old anaconda, old systemd) then update systemd: should systemd's scriptlets try and do something to /etc/resolv.conf ? 2. You run an install with release-day anaconda but updated systemd (e.g. netinst install): what happens? 3. You run an install with updated anaconda and updated systemd (e.g. a live-respin or a manually generated image or something from openQA or Beaker): what happens? Every approach we've tried so far, AIUI, has had problems in at least one of those cases, and nobody seems to have an idea for an approach that would solve them all. This is different in Fedora 36 because we got anaconda and systemd lined up the way we want before release for Fedora 36, so it doesn't have the same considerations as 35 does.
I think that the first goal should be that the current stable release Fedora 35 should install with the Fedora provided images and repos without aborting the installation. This are your cases 1 and 2. If users was able to install Fedora 35 in the last few months, they should be able to do it until EOL. These users may already have a solution how to handle /etc/resolv.conf in this case. In no case package installation should crash in it's scripts and make the system unusable or abort the (kickstart) installation. (In reply to Adam Williamson from comment #62) > 1. You run an install with all release-day packages (old anaconda, old > systemd) then update systemd: should systemd's scriptlets try and do > something to /etc/resolv.conf ? I think it should do nothing with /etc/resolv.conf, because the system is already running in some way, possibly with a change by the user / administrator. > 2. You run an install with release-day anaconda but updated systemd (e.g. > netinst install): what happens? "Same behavior" as before: It should leave the file as it would be with an old systemd. > 3. You run an install with updated anaconda and updated systemd (e.g. a > live-respin or a manually generated image or something from openQA or > Beaker): what happens? These people can use Fedora 36 where the problem is fixed, or use the old images? I would prioritize this lower, because these are not official Fedora images. In my case, I have added the following script to the end (important!) of the %post part of my kickstart files for Fedora 35 because I know that NetworkManager and systemd-resolved are installed and will be running after reboot. It should be the last command running before reboot because it breaks dns name resolution. systemd package scripts cannot to this because they are not the last action that run on the installed system. if [[ ! -h /etc/resolv.conf ]] ; then ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf fi
I ran into this issue as well doing a kickstart install that I knew had worked in the past. My workaround before I found this bug report was to disable the local repository for the updates & install from base instead.
(In reply to Adam Williamson from comment #62) > People keep asking when this is going to be "fixed". The problem is we've > tried to fix it four times and it hasn't worked. Nobody has come up with a > way to fully "fix" it in all cases yet. The problem is that there are > various cases: > > 1. You run an install with all release-day packages (old anaconda, old > systemd) then update systemd: should systemd's scriptlets try and do > something to /etc/resolv.conf ? > 2. You run an install with release-day anaconda but updated systemd (e.g. > netinst install): what happens? > 3. You run an install with updated anaconda and updated systemd (e.g. a > live-respin or a manually generated image or something from openQA or > Beaker): what happens? > > Every approach we've tried so far, AIUI, has had problems in at least one of > those cases, and nobody seems to have an idea for an approach that would > solve them all. > > This is different in Fedora 36 because we got anaconda and systemd lined up > the way we want before release for Fedora 36, so it doesn't have the same > considerations as 35 does. Adam, The kernel-hw team provides and services a lions share of shared systems to Beaker for all users / groups to test with. This issue leaves all the systems cripples with broken boot order when they are tested with Fedora35. This takes alot of man hours to return the numerous broken systems back to working order, only for them to be set broken once again when a Fedora35 install is run. Would it be acceptable for the kernel-hw team to disable Fedora35 on all their shared test hardware until this issue is resolved? Best, pbunyan
(In reply to Adam Williamson from comment #62) > People keep asking when this is going to be "fixed". The problem is we've > tried to fix it four times and it hasn't worked. With systemd-249.9-1.fc35, provisioning of Fedora 35 worked. What were the issues with that systemd version-release and were they more severe than not being able to provision Fedora? Can't we just revert systemd package to that state, and have any fixes for other issues land in Fedora 36?
Those are questions for the maintainers. I'm just the QA guy. I was trying to help people understand why this has not been resolved already.
*** Bug 2083411 has been marked as a duplicate of this bug. ***
*** Bug 2083866 has been marked as a duplicate of this bug. ***
Zbigniew, can you chime in here? It doesn't seem like the current situation is sustainable. Is there still a way to make all three cases viable? If we can only fix cases 1) and 2) in my list, but 3) can be made to work with an updates.img, I can live with that (I can have openQA always use the updates.img when building updated F35 images).
OK, I think I see the problem here. Zbigniew's latest change didn't do what he thought it would. He added a check that makes the scriptlet only link /etc/resolv.conf to the dynamic stub if `/run/systemd/system/` is present; if it's not present, but /etc/resolv.conf doesn't exist, it gets linked to /usr/lib/systemd/resolv.conf . I think he expected the check would fail in the installer case, and we'd get a link to /usr/lib/systemd/resolv.conf . That's not what happens, though. /run/systemd/system/ *does* exist in the installed system root in the installer environment. So the check passes, and we still create the dangling symlink whose target is in a non-existent directory, and anaconda still crashes. I see two possible ways to fix this. One is to extend the check from: if test -d /run/systemd/system/; then to: if test -d /run/systemd/system/ && test -d /run/systemd/resolve/; then this would, I think, give the result Zbigniew intended in the installer environment: we'd get a symlink to ../usr/lib/systemd/resolv.conf , which should exist, which should cause anaconda not to crash (and not to replace it). The other option is to extend the symlink creation from this: ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || : to this: mkdir -p /run/systemd/resolve && touch /run/systemd/resolve/stub-resolv.conf && ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || : i.e., create the symlink target if it is not there already. This would make the symlink's target an existent regular file, and make anaconda not attempt to replace it. Of course the target would be an empty file and name resolution wouldn't work inside the installed system root, but that shouldn't be a terrible problem. If we wanted to give it a shot at working, I guess, instead of touching the target, we could make the target itself a symlink to /usr/lib/systemd/resolv.conf , but that might be gilding the lily. On regular boot of the installed system, the whole of /run should be recreated, and the symlink should wind up pointing to a stub that's actually there, and everything should work. i hope. I'm going to test these two approaches now.
So, I see that 沙包妖梦 actually suggested the "create /run/systemd/resolve/stub-resolv.conf" approach back in April in #c39, so credit to them for that (I came to the same idea independently). I've tested this now. It does work. In response to Zbigniew's concerns from #c40, the installed system's `/run` *is* mounted during the install process (it is not just a regular directory). I ran a test install with the change I suggested above to create the directory and file when creating the symlink. anaconda does not crash, the symlink is successfully created, and anaconda leaves it alone. On booting the installed system, there are no warnings that I can see, and /etc/resolv.conf is correctly linked to ../run/systemd/resolve/stub-resolv.conf which exists and has correct contents. Name resolution works. I also tested an install without systemd-resolved (I made a kickstart that excludes it). That install also succeeds. /etc/resolv.conf is created as a regular file 'generated by NetworkManager', and name resolution works. I did find one very specific scenario where this proposal makes things worse: if you try to install systemd-resolved after installing without it. If you start from the install without systemd-resolved and then install the current stable systemd-resolved, it replaces /etc/resolv.conf with a symlink to ../run/systemd/resolve/stub-resolv.conf . At first this does not work, because systemd-resolved is *enabled* but not *started* on install, and so the target of the symlink does not exist. If you start systemd-resolved.service, though, the target is created and /etc/resolv.conf becomes 'correct'. If you start from the install without systemd-resolved and then install my patched systemd-resolved, it replaces /etc/resolv.conf with a symlink to ../run/systemd/resolve/stub-resolv.conf and *creates that as an empty file*, just like we told it to. If you then start systemd-resolved.service , it does not replace the empty stub file with a correct one, and so /etc/resolv.conf is still empty and doesn't work. You have to reboot before the correct stub file will exist. That seems like a pretty minor issue, though, and probably not worth worrying about too hard? I don't think we need to worry about upgrade scenarios, since we switched to resolved-by-default in F33. I'm gonna create an update for systemd with this change, but disable autopush. If Zbigniew has concerns, we can pull it.
FEDORA-2022-6d2c62d6d6 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-6d2c62d6d6
So in the update I went with the "create /run/systemd/resolve/stub-resolv.conf as a symlink to /usr/lib/systemd/resolv.conf if it doesn't exist yet" approach: https://src.fedoraproject.org/rpms/systemd/c/6b234deaf8576d78278cd5d8aa8d77011ee1f12b?branch=f35 please test it out and see how it goes. Thanks.
FEDORA-2022-6d2c62d6d6 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-6d2c62d6d6` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-6d2c62d6d6 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
I have tested systemd-249.12-4.fc35. With kickstart installation, it doesn't work as expected. DNS search domain only contains "." (as of file /usr/lib/systemd/resolv.conf) instead of our own dns search list which was provided by dhcp. I don't find a proof in /var/log/anaconda/*, but I think that Anaconda or NetworkManager is creating the system-connection files in /etc/NetworkManager/system-connections/*.connection after package systemd-resolved has been installed, and after the scripts of package systemd-resolved has been run, which set the dns search list to ".", overwriting the value that the system got by dhcp. It works as expected with the original Fedora 35 iso images without updates during kickstart installation. The problem is that after reboot unqualified (short) host names are no longer resolved. Update systemd* on a already working Fedora 35 installation works (because it doesn't change resolv.conf).
Hmm. OK, I'll look into that tomorrow. I can go back to a build which creates the symlink target as an empty file and see if it behaves any differently in that case...
Actually, I'm on vacation tomorrow and Tuesday, will look Wednesday if nobody else does. It should be quite easy to tweak the build to create the symlink target as an empty file if you want to try it that way.
I have build systemd with "%posttrans resolved" creating an empty file /run/systemd/resolve/stub-resolv.conf . This does not work either, the dns search list is empty after installation as of systemd-249.12-4.fc35. I have build systemd with "%posttrans resolved" just creating /run/systemd/resolve, but not creating files in it and not creating a symlink (and printing some debug info for me). This works for me, because it leaves /etc/resolv.conf in the state created by NetworkManager (containing a "search" and three "nameserver" lines). I use a kickstart "%post" script that removes /etc/resolv.conf and creates the symlink to /run/systemd/resolve/stub-resolv.conf as the very last action of the "%post" script. I think the general solution was done in Anaconda for Fedora 36. For Fedora 35 we need something that is run just before reboot, but I'm unsure if this is possible (beside of administrator kickstart %post script). So leaving /etc/resolv.conf as a plain file is not a perfect solution, but it is the same state as installation from Fedora 35 release images without repo updates and it doesn't abort installation.
The thing that was changed in anaconda for F36 is that it now doesn't replace /etc/resolv.conf if it's a dangling symlink . So systemd-resolved %post can create it as a dangling symlink and anaconda leaves it that way. This was changed in F35 anaconda updates too, but of course GA images still have the original behaviour. Your "This works for me, because it leaves /etc/resolv.conf in the state created by NetworkManager" would be just the same as reverting systemd to exactly the state it was in at F35 release, because that's what happens with that setup - you get /etc/resolv.conf controlled by NM. This is what we started out trying to *fix*, because it's not what was intended. There's no way to lever in something that happens "just before reboot" for the scenario we're trying to handle here (netinst install with GA images), because to do that we have to change anaconda's behaviour, and we can't change anaconda's behaviour on the GA images. That's fixed. We can only make changes in the *packages that the netinst install deploys*, and the latest they can do anything is in %posttrans, which is already where the systemd-resolved scriptlet we're poking runs.
Thanks for the explanation. I think another change between F35 and F36 is that in F36 systemd-resolved is running during installation (I run "ps -ef" during kickstart %post to see what is running), while on F35 it isn't running. Maybe a solution would be to start systemd-resolved.service (only during system installation!) and wait for finishing startup, and only then set up the symlink to already existing files /run/systemd/resolve/stub-resolv.conf and /run/systemd/resolve/resolv.conf? Then dns resolution would still work during installation, and (hopefully) NetworkManager can save the right values of "dns-search" in /etc/NetworkManager/system-connections/? I don't know if it can work, it's just a guess.
I will try another build which starts systemd-resolved. What do you think about this? %posttrans resolved test -e %{_localstatedir}/lib/rpm-state/systemd-resolved-initial-installation || exit 0 # Initial installation rm %{_localstatedir}/lib/rpm-state/systemd-resolved-initial-installation || : ### ... comments not listed here if systemctl -q is-enabled systemd-resolved.service &>/dev/null && ! systemd-analyze cat-config systemd/resolved.conf 2>/dev/null | grep -iqE '^DNSStubListener\s*=\s*(no?|false|0|off)\s*$' && ! mountpoint /etc/resolv.conf &>/dev/null; then if ! systemctl is-active systemd-resolved ; then systemctl start systemd-resolved || : fi if test -d /run/systemd/resolve/ && ! mountpoint /etc/resolv.conf &>/dev/null; then ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf &>/dev/null || : fi fi
I suspect it might not work because I don't think systemd will start units inside a chroot. Also note that "initial installation" (as the spec file checks for) is not strictly the same thing as "system installation". "Initial installation" can also be "first time systemd-resolved is installed on a system that has already been deployed". It would also be encountered during things like live image builds, which are run through anaconda.
Yes, you are right. systemd-resolved wasn't started. /var/log/anaconda/packages.log contains Running in chroot, ignoring command 'is-active' I guess the last attempt would be to create an updates.img, because systemd-resolved should run outside of the chroot?
Well, it'd be trivial (I think) to ship an updates.img that solves the problem - just backport the anaconda changes from F36. But that's not going to make everyone happy, is it? Folks inside RH are unhappy because they want their beaker stuff to work, and "fixing" it with an updates image requires everyone to update their config to use the updates image. Any random person just trying to do an F35 netinst is not going to know they need an updates image, they'll try it, it'll fail, then *hopefully* they'll find this bug and try again with the updates image. Which isn't a great experience. I'd much rather we ship an update that makes installs not crash. If we can't fix the problem you noted in my update, we may just have to go with the idea of reverting to the GA behavior, i.e., let the NM-controlled /etc/resolv.conf be copied over from the install environment. That means choosing not to fix #2032085 for future F35 network installs and living with whatever problems/inconsistencies that causes.
Edgar: so, in testing, I can't seem to reproduce your result. I'm now testing on a bare metal test box on my home network. I configured my router to set a search domain for DHCP purposes. I checked that if I do an F36 network install, the line 'search happyassassin.net' (the domain I set) appears in /etc/resolv.conf and /run/systemd/resolve/resolv.conf . I then did an F35 network install from the GA netinst image, with a repo containing my systemd package added as an additional repo. If I check /etc/resolv.conf in the installed system root *before rebooting*, it does not have the 'search happyassassin.net' line (as I'd expect, given how my attempted fix works). /run/systemd/resolve/resolv.conf does not exist in the installed system root at all yet, again as expected. But if I then reboot to the installed system and check again, all is good. /run/systemd/resolve/stub-resolv.conf is now a regular file not a symlink to /usr/lib/systemd/resolv.conf , and it has the 'search happyassassin.net' line. /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf and of course is the same. /run/systemd/resolve/resolv.conf exists and has the 'search happyassassin.net' line. So, AFAICS, all is working as it should once I reboot to the installed system. Could you provide any additional details on the test scenario which didn't work as expected for you?
I assume you are using dynamic network configuration after reboot? I need to mention that I configure kickstart to create a static network configuration for our workstations and servers, because these should not depend on a running DHCP server. The DHCP server is running for notebooks and for (all) system installation with PXE. network --bootproto=static --device=bootif --ip=XXX --netmask=255.255.255.0 --gateway=YYY --noipv6 --nameserver=ZZZ --hostname=NNN I havn't tested it, but all systems that uses (only) dynamic network configuration are not affected by the dns search domain problem, because they will receive actual data at each boot (more precisely: every time a network connection is started). So, our configuration may be a unusual case?
Yeah, that's actually closer to what I first thought you were saying - that you get a problem with a fairly specific scenario with kickstart network configuration. So your expectation is that you configure *static* networking via kickstart, but that the DNS configuration (or just the search domains?) are retrieved via DHCP all the same? I am not a networking expert so I confess I don't know if that's expected behaviour. You say this is how things work with F35 GA (including GA systemd) though, right? Does it also work as you expect with F36 GA? Either way, it does make me think it might be OK to just send my update stable, if it will at least solve the crashes and make things work as intended in more common cases.
As far as I know there is no kickstart option to configure the dns search domain. So I cannot configure it this way. For system installation I usually use PXE boot, which gets all network configuration from DHCP, including dns server, dns search domain, time servers, bootp server, etc. It boots into pxelinux (for legacy bios) or grub2 (for uefi systems) and presents a list of boot options (which I have described in the corresponding config files). This allows booting a linux kernel and pass to it kernel boot parameters (initrd, inst.repo, inst.ks, etc.). It searches for a kickstart file on our nfs server and starts the installation. The installation system get all network configuration from DHCP. The kickstart configuration causes that the configured values and other current parameters are stored on the installed system for use after reboot. Yes, dns search domain comes only from DHCP, and it works with Fedora 36, and it has worked the same way on every previous Fedora version ever (as far as I remember, it's so many years). Only this new systemd updates on Fedora 35 has broken this installation method. Yes, it is ok to send your update to stable, because it makes the current situation better, as I also wrote in bodhi. If others are lucky with this solution, then I can create a workaround for our purpose, and we can save the time to search for further (unsatisfactory) solutions, because it works again with Fedora 36. Many thanks for your time!
Adam, note that installation with systemd-249.12-4.fc35.x86_64 fails on beaker because in the Running post-installation scripts, the fetch / curl operations cannot resolve the lab controller's hostname.
For non-RHers who notice a couple of private comments above - they're about an internal RH system, but they point up a general issue with my update: kickstart scripts may run in the installed system root and expect name resolution to work. With F35 GA this worked, with the proposed fix it does not. I'll see if there's anything I can come up with to change this.
OK, I think I reproduced the problem with %post scripts that need to access the network, and fixed it without (hopefully) breaking anything else. I've put a -5 build in the update. Jan, can you please try that? Edgar, can you also see if it fixes your similar issue with %post and NFS? I do not expect it to fix your DHCP search domain issue, but if I'm understanding that correctly I do think it'd be reasonable to push this out and think about that later. Thanks!
Sorry, forgot to mention - what the -5 build actually does is change the target of the /run/systemd/resolve/stub-resolv.conf symlink we create during initial package install. Instead of pointing to /usr/lib/systemd/resolv.conf , which is a static stub file that always has the same contents and says to use 127.0.0.53 , it now points to resolv.conf in the same directory, which should point to the "real" nameservers. This is still just a temporary setup that only exists until reboot (when /run will be regenerated and /run/systemd/resolve/stub-resolv.conf will be the proper dynamic stub file, not a symlink to somewhere else).
Adam, this new update fixes my problems with %post and NFS. It works as usual now, and after reboot my own postinstallation scripts are automatically started, and they can also use NFS. DNS search domain isn't preserved, but it doesn't matter, dns resolution works now, and my scripts already used fully qualified domain names. Also, my scripts has host class specific dns search domain lists, and they set the dns search domain if it doesn't match the value from our preset list. It works for me, you don't need to search for solution about this problem because of me. Thanks for your work!
I also confirm that with systemd-249.12-5.fc35, things are sane again. Karma in bodhi given. Thank you!
Zbigniew, can you please check my fix? I don't want to push it stable without your signoff, but since it does at least prevent installs crashing, if I don't hear either way by next week, I'll push it.
FEDORA-2022-6d2c62d6d6 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
AdamW, thanks for working on this and apologies for not getting to this earlier. Since it seems to work, then I'm all for it :)
Thanks Zbigniew! Since nobody complained I went ahead and pushed it stable yesterday, so we should be good.
Thanks for this. I am now getting a similar error when doing an install Performing post-installation setup tasks Configuring installed system Writing network configuration An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details I press '3' on the console and then get: return self._get_method_reply( File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/__init__.py", line 46, in sync_run_task task_proxy.Finish() File "/usr/lib64/python3.10/site-packages/pyanaconda/network.py", line 293, in write_configuration sync_run_task(task_proxy) File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 458, in run_task self._task(*self._task_args, **self._task_kwargs) File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 496, in start self.run_task() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start item.start() File "/usr/lib64/python3.10/site-packages/pyanaconda/installation.py", line 415, in run_installation queue.start() File "/usr/lib64/python3.10/threading.py", line 946, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run threading.Thread.run(self) dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf' This is different in that before the error message about resolv.conf was displayed on the screen during the install. I didn't need to press '3'. I checked our mirror repository log and it downloaded systemd-249.12-3.fc35.aarch64.rpm
I guess I may be wrong that it is different. As I see the initial bug said they also had to check the log file. So I might be mis-remembering that it showed up in the console log.
"I checked our mirror repository log and it downloaded systemd-249.12-3.fc35.aarch64.rpm" The fix is in 249.12-5.fc35. It looks like your mirror isn't up to date yet.
I need some help on how to use this update together with the current netinst iso that is available from the mirrors. Do I need to rebuild the iso? Or, is there a magic command to swap in this update during the install? Thanks.
(In reply to Adam Williamson from comment #106) > "I checked our mirror repository log and it downloaded > systemd-249.12-3.fc35.aarch64.rpm" > > The fix is in 249.12-5.fc35. It looks like your mirror isn't up to date yet. Thanks! We mirror from `mirrors.kernel.org` and looks like they haven't yet synced. :( Digging deeper into it it doesn't seem like they have anything newer than 1-June-2022 in the `s` updates directory. https://da.mirrors.kernel.org/fedora/updates/35/Everything/aarch64/Packages/s/
Ken: you don't need to do anything special. As long as the install hits on a mirror that has got the update, things should work without you having to do anything special. John: hmm, that seems like an issue. Thanks for spotting it. I'll try and get someone to follow up (I see you're in IRC poking about this too).
This fix is working for us on aarch64 architecture. Thanks!