Bug 2032085 - Some variants are missing /etc/resolv.conf symlink (use systemd-resolved)
Summary: Some variants are missing /etc/resolv.conf symlink (use systemd-resolved)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 36
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Anaconda Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F36BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2021-12-14 02:02 UTC by Chris Murphy
Modified: 2022-03-10 07:19 UTC (History)
20 users (show)

Fixed In Version: systemd-250~rc1-4.fc36 anaconda-36.16.2-2.fc36
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 07:19:52 UTC
Type: Bug


Attachments (Terms of Use)
End of Kickstart Installation (53.70 KB, image/png)
2022-01-16 07:01 UTC, Christoph Karl
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github rhinstaller anaconda pull 3884 0 None open Port resolv conf fixes from master 2022-02-25 18:04:02 UTC

Internal Links: 2018913 2022816

Description Chris Murphy 2021-12-14 02:02:43 UTC
Description of problem:

Since Fedora 35, Fedora Server (and possibly others) do not have an '/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf' symlink following a new clean install.

https://fedoraproject.org/wiki/Changes/systemd-resolved


Version-Release number of selected component (if applicable):
Fedora 35 and Rawhide

How reproducible:
Always, for affected editions/spins/images


Steps to Reproduce:
1. Do an installation
2.
3.

Actual results:

Missing '/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf'

Expected results:

'resolv.conf -> /run/systemd/resolve/stub-resolv.conf' should exist

Additional info:

Initial report:
https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/EEZPJSCFY3D2MBWPMJFDFEV4QU3XY7NS/

Possible it'll resolve itself (no pun intended):
https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/QWB6WAJS6PMUEULM5CAAOCMLGZCIWPPB/

Comment 1 Fedora Blocker Bugs Application 2021-12-14 02:11:35 UTC
Proposed as a Blocker for 36-beta by Fedora user chrismurphy using the blocker tracking app because:

 As far as I'm aware, we don't have a policy to consider inadvertently reverted Fedora features as blockers. We could get FESCo to just grant this blocker status. Main idea though is this slipped by Fedora 35, and to not let it happen in Fedora 36.

Comment 2 Zbigniew Jędrzejewski-Szmek 2021-12-14 10:43:31 UTC
> Missing '/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf'
To clarify: there is no file there, right?

systemd-resolve.rpm has (in rawhide only, it was moved recently) /usr/lib/tmpfiles.d/systemd-resolve.conf,
which would create the symlink. So I really expect the symlink to be there after a reboot.

Comment 3 Chris Murphy 2021-12-18 02:25:13 UTC
>To clarify: there is no file there, right?

There is a file, it's created by NetworkManager.

Comment 4 Zbigniew Jędrzejewski-Szmek 2021-12-18 09:56:14 UTC
systemd-resolved package has a scriplet to symlink /etc/resolv.conf.
This scriptlet went through a bunch of changes, and then was moved to systemd-resolved package
when that was created. Though if the the packages are installed into a chroot, the script
doesn't do anything: it only does anything if systemd is running. This is done with the assumption
that the symlink will be created by tmpfiles after a reboot.

The changes mentioned in
https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/QWB6WAJS6PMUEULM5CAAOCMLGZCIWPPB/
are not relevant: they only touch nsswitch.conf.

If there is a file created by NetworkManager, it's mean that either something creates
the symlink during installation, or that NetworkManager gets started before systemd-tmpfiles-setup.service
after a reboot.

It would be good to figure out who and when exactly creates the symlink.

One option would be to change the scriptlet in systemd-resolved.rpm to create the symlink
in a chroot. It'll remove some flexibility, but it should prevent issues like this bug.

Comment 5 Zbigniew Jędrzejewski-Szmek 2021-12-18 11:13:57 UTC
I pulled Fedora-Server-netinst-x86_64-35-1.2.iso and did an all-defaults installation.
*Before rebooting* I see the following:

/etc/resolv.conf is a normal file with the comment "Generated by NetworkManager" and
some specific addresses listed.

An identical file appears as /mnt/sysroot/etc/resolv.conf.
Anaconda copies it there. I assume it's the function called copy_resolv_conf_to_root().

Interestingly, the installation image is not using systemd-resolved, it doesn't even
have systemd-resolved.rpm installed. In the installed system, systemd-resolved.rpm is
present.

I see the following options:
1. remove copy_resolv_conf_to_root() from Anaconda. In general the approach of copying
   the file is broken. It contains dynamic configuration that can be useful after a reboot
   only if we are lucky.
2. make systemd-resolved.rpm create the symlink when installed into a chroot.
3. pull systemd-resolved.rpm into the installation image
4. some combination of the above, even possibly 1+2+3

For systemd that have a r/w root, not having the symlink at all (and creating it
after a reboot) is a good choice. But for systems with r/o root, we'd want to create the
symlink. How does this work on ostree, does it expect the symlink to be there?

Comment 6 Zbigniew Jędrzejewski-Szmek 2021-12-18 15:52:01 UTC
I'm implementing 2 in systemd now. But I think the other steps should be considered too.

Comment 7 Chris Murphy 2021-12-24 00:24:41 UTC
Also affects Silverblue.

Comment 8 Tomas Popela 2022-01-03 08:11:14 UTC
(In reply to Chris Murphy from comment #7)
> Also affects Silverblue.

Yes, it has been reported in https://github.com/fedora-silverblue/issue-tracker/issues/192.

Comment 9 Fedora Update System 2022-01-11 21:42:54 UTC
FEDORA-2022-f38f479b8f has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f38f479b8f

Comment 10 Fedora Update System 2022-01-12 01:49:34 UTC
FEDORA-2022-f38f479b8f has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-f38f479b8f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-f38f479b8f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 11 Fedora Update System 2022-01-12 22:17:42 UTC
FEDORA-2022-f38f479b8f has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f38f479b8f

Comment 12 Fedora Update System 2022-01-13 01:10:43 UTC
FEDORA-2022-f38f479b8f has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-f38f479b8f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-f38f479b8f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2022-01-15 01:21:19 UTC
FEDORA-2022-f38f479b8f has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Christoph Karl 2022-01-16 06:59:51 UTC
I just did a fresh kickstart installation of fc35 and I still get: See attached.

Comment 15 Christoph Karl 2022-01-16 07:01:25 UTC
Created attachment 1851083 [details]
End of Kickstart Installation

Comment 16 Christoph Karl 2022-01-16 07:08:14 UTC
bash-5.1# /mnt/sysroot/usr/bin/systemd-run --version
systemd 249 (v249.4-2.fc35)

Shouldn't this be:
>/usr/bin/systemd-run --version
systemd 249 (v249.9-1.fc35)

Comment 17 Christoph Karl 2022-01-16 07:09:31 UTC
Very same kickstart installation works for f34 and rawhide.

Comment 18 Alexander Sosedkin 2022-01-18 19:14:24 UTC
I also observe this installation regression on systemd-resolved-249.9-1.fc35:

[  176.607753] anaconda[1964]: Writing network configuration
[  178.611817] anaconda[1964]: The installation was stopped due to an error which occurred while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation.
[  178.611938] anaconda[1964]: The exact error message is:
[  178.612259] anaconda[1964]: Non interactive installation failed: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf'.
[  178.612595] anaconda[1964]: The installer will now terminate.

https://github.com/rhinstaller/anaconda/blob/a1ff62e575294b33d7870072d9befe98278224e2/pyanaconda/modules/network/installation.py#L228 seems to be a relevant place inside anaconda.

Comment 19 Alexander Sosedkin 2022-01-18 19:28:13 UTC
Oh, it's filed as bz2019579

Comment 20 Chris Murphy 2022-01-24 20:44:28 UTC
This is still happening with a clean install of Fedora-Server-netinst-x86_64-Rawhide-20220124.n.0.iso systemd-250.3-1.fc36.x86_64

/etc/resolv.conf is a file, not a symlink.

-rw-r--r--. 1 root root        55 Jan 24 13:40 resolv.conf

$ cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.122.1
$ resolvectl
Global
         Protocols: LLMNR=resolve -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
  resolv.conf mode: foreign
Current DNS Server: 192.168.122.1
       DNS Servers: 192.168.122.1

Link 2 (enp1s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
Current DNS Server: 192.168.122.1
       DNS Servers: 192.168.122.1
$

Comment 21 Radek Vykydal 2022-02-03 11:57:04 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #5)

For the record:

> I see the following options:
> 1. remove copy_resolv_conf_to_root() from Anaconda. In general the approach
> of copying
>    the file is broken. It contains dynamic configuration that can be useful
> after a reboot
>    only if we are lucky.

PRs under review:
https://github.com/rhinstaller/anaconda/pull/3814
https://github.com/rhinstaller/anaconda/pull/3818

> 2. make systemd-resolved.rpm create the symlink when installed into a chroot.

This seems to be still not working? (bug #2018913)

> 3. pull systemd-resolved.rpm into the installation image

Done in https://github.com/rhinstaller/anaconda/pull/3783

So currently - Fedora-Rawhide-20220202.n.1 - where 1. and 2. is not working there is an /etc/resolv.conf containing content of target of the symlink from installer environment (/run/systemd/resolve/stub-resolv.conf)

When we merge 1. there will be no /etc/resolv.conf after installation (given 2. is still broken)

> 4. some combination of the above, even possibly 1+2+3

Comment 22 Edgar Hoch 2022-02-07 10:03:48 UTC
Because blocker ticket #581 should be used for blocker discussion, I copy my tests about this problem here so the developers can see it.


For Fedora 35 I have added the following script to kickstart %post section because otherwise DNS doesn't work (hostname resolution failed, hostname not found).

    if [[ ! -h /etc/resolv.conf ]] ; then
        # Link setzen
        ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
    fi

Now I have run a test kickstart installation with Fedora 36 Rawhide 20220205.n.1 nightly compose. I have not run the script above.

After the reboot file /etc/resolve.conf is a plain text file created (or updated?) by NetworkManager (# Generated by NetworkManager).
NetworkManager.service and systemd-resolved.service are both running.

So some commands and libaries and system calls use /etc/resolve.conf and some use systemd-resolved to do DNS name and ip resolution. This may lead to different results / answers depending on the configuration and the selected dns servers. It is not a consistent configuration.

But it is true: host and dig for example works resolving dns names, but both use the first dns server listed in /etc/resolve.conf .

If I remove file /etc/resolve.conf and reboot the system, then /etc/resolve.conf is a symbolic link to ../run/systemd/resolve/stub-resolv.conf .

But this symbolic link is created both with systemd-resolved.service and systemd-resolved.service not running (disabled). In the last case the symbolic link refers to a nonexistent file! Something (*) creates the symbolic link if the file doesn't exist even if systemd-resolved will not be started!

(*) Something is tmpfiles:

  /usr/lib/tmpfiles.d/systemd-resolve.conf

But if systemd-resolved is not running, the symbolic link should not exist, at least it should not refer to a nonexistent file. In this case NetworkManager should create /etc/resolve.conf as in earlier Fedora versions.

It seams that if the file /etc/resolve.conf exists either as plain file or symbolic link, it isn't changed. If the file doesn't exist, a symbolic link to the stub file is created.

But this is not a consistent and useful behavior. It may be right to not block the beta release, but I think this problem should not go to the final Fedora 36 release.

Also note that I reported bug 2022816 last November about this problem in Fedora 35. But it seems it doesn't solve it for Fedora 36.
And a similar problem was described in bug 1921057 in Janary 2021.

Comment 23 Ben Cotton 2022-02-07 14:47:24 UTC
Setting status to POST since upstream patches are pending.

Comment 24 Geoffrey Marr 2022-02-07 20:52:44 UTC
Discussed during the 2022-02-07 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug was made as we don't think this qualifies as a blocker under the release criteria currently, but there seems to be a strong case for FESCo to declare it a blocker as the current state is not at all how we want things to be, and this is an important area. At this time, instead of rejecting, we will file a ticket for FESCo to consider declaring it a blocker (as they are allowed to do).

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-02-07/f36-blocker-review.2022-02-07-17.00.txt

Comment 25 Ben Cotton 2022-02-08 20:07:38 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Comment 26 Chris Murphy 2022-02-10 05:00:32 UTC
Fedora-Silverblue-ostree-x86_64-36-20220209.n.0.iso has
lrwxrwxrwx. 1 root root        39 Feb  9 21:33 resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

Comment 27 Chris Murphy 2022-02-10 05:01:43 UTC
Clarification: following a default clean installation, the destination has the expected symlink.

Comment 28 Adam Williamson 2022-02-14 05:25:22 UTC
Filed https://pagure.io/fesco/issue/2760 to ask FESCo to consider this bug.

Comment 29 Geoffrey Marr 2022-02-14 19:24:29 UTC
Discussed during the 2022-02-14 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug was made as per last week's meeting. We don't think this is a blocker under the criteria but believe FESCo should consider it, a ticket has now been filed for FESCo: https://pagure.io/fesco/issue/2760, so we will wait for them to consider it.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-02-14/f36-blocker-review.2022-02-14-17.01.txt

Comment 30 Miro Hrončok 2022-02-21 23:25:59 UTC
This has been accepted as a FESCo blocker in https://pagure.io/fesco/issue/2760

Comment 32 Ben Cotton 2022-02-25 18:04:02 UTC
Setting component to anaconda for merge of upstream PR 3884

Comment 33 Fedora Update System 2022-03-03 03:23:56 UTC
FEDORA-2022-cf175d0e19 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-cf175d0e19

Comment 34 Fedora Update System 2022-03-03 23:52:58 UTC
FEDORA-2022-cf175d0e19 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-cf175d0e19`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-cf175d0e19

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 35 Adam Williamson 2022-03-04 20:20:16 UTC
I'm not clear on exactly what's still considered to be broken here, but I'd say we can push the anaconda update stable at least. The same code is in Rawhide already and doesn't seem to be causing problems.

Does anyone know what's still broken here and if anything beyond the anaconda update is needed to fix it?

Comment 36 Adam Williamson 2022-03-04 20:42:58 UTC
Well, actually. I just tested a case where we did a static network config during install, including a pair of DNS servers. On boot into the installed system, the DNS servers set during installation are not configured. /etc/resolv.conf looks like this:

===

# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .

===

`resolvectl status` shows no name servers, and pinging www.google.com doesn't work.

This seems bad? Wasn't the point of copying resolv.conf to the installed system supposed to be so that the networking configuration set during install would be preserved on the installed system?

Comment 37 Adam Williamson 2022-03-04 20:45:14 UTC
I also wonder - what would happen in a case where someone runs an install using a kickstart that leaves out systemd-resolved? We made systemd-resolved *default*, but we didn't make it *mandatory*. We still leave open the possibility that you could choose not to use it.

Comment 38 Radek Vykydal 2022-03-07 09:09:13 UTC
(In reply to Adam Williamson from comment #36)
 
> This seems bad? Wasn't the point of copying resolv.conf to the installed
> system supposed to be so that the networking configuration set during
> install would be preserved on the installed system?

I'd think the static dns configuration is preserved in NM ifcfg/keyfiles and the systemd-resolved should be configured based on it (via NM?).
Also see point 1. from comment #5.

Comment 39 Radek Vykydal 2022-03-07 09:14:11 UTC
(In reply to Adam Williamson from comment #37)
> I also wonder - what would happen in a case where someone runs an install
> using a kickstart that leaves out systemd-resolved? We made systemd-resolved
> *default*, but we didn't make it *mandatory*. We still leave open the
> possibility that you could choose not to use it.

I'd expect in case systemd-resolved is not present (does not handle /etc/resolv.conf) NM is getting opportunity to generate/control the /etc/resolv.conf. IIRC there is some synchronization between systemd-resolved and NM around ownership of /etc/resolv.conf.

Comment 40 Radek Vykydal 2022-03-07 09:19:09 UTC
(In reply to Adam Williamson from comment #36)
 
> `resolvectl status` shows no name servers, and pinging www.google.com
> doesn't work.

I did a quick check on *rawhide* - I've added an additional static configuration of a device in GUI and resolvectl status was showing the manually configured dns server on the installed system.

Comment 41 Adam Williamson 2022-03-07 16:20:56 UTC
Hmm, wonder what the difference is. I'll test again manually here in that case. Thanks for trying it.

Comment 42 Adam Williamson 2022-03-07 16:58:55 UTC
OK, I tried it here manually and indeed it worked OK, also looks like it would work without systemd-resolved as the DNS config is written to /etc/NetworkManager/system-connections . So I agree this is probably good.

Still don't know if there's anything else that needs fixing here.

Comment 43 Fedora Update System 2022-03-10 07:19:52 UTC
FEDORA-2022-cf175d0e19 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.