Bug 2032085

Proposed as a Blocker for 36-beta by Fedora user chrismurphy using the blocker tracking app because:

 As far as I'm aware, we don't have a policy to consider inadvertently reverted Fedora features as blockers. We could get FESCo to just grant this blocker status. Main idea though is this slipped by Fedora 35, and to not let it happen in Fedora 36.

> Missing '/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf'
To clarify: there is no file there, right?

systemd-resolve.rpm has (in rawhide only, it was moved recently) /usr/lib/tmpfiles.d/systemd-resolve.conf,
which would create the symlink. So I really expect the symlink to be there after a reboot.

>To clarify: there is no file there, right?

There is a file, it's created by NetworkManager.

systemd-resolved package has a scriplet to symlink /etc/resolv.conf.
This scriptlet went through a bunch of changes, and then was moved to systemd-resolved package
when that was created. Though if the the packages are installed into a chroot, the script
doesn't do anything: it only does anything if systemd is running. This is done with the assumption
that the symlink will be created by tmpfiles after a reboot.

The changes mentioned in
https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/QWB6WAJS6PMUEULM5CAAOCMLGZCIWPPB/
are not relevant: they only touch nsswitch.conf.

If there is a file created by NetworkManager, it's mean that either something creates
the symlink during installation, or that NetworkManager gets started before systemd-tmpfiles-setup.service
after a reboot.

It would be good to figure out who and when exactly creates the symlink.

One option would be to change the scriptlet in systemd-resolved.rpm to create the symlink
in a chroot. It'll remove some flexibility, but it should prevent issues like this bug.

I pulled Fedora-Server-netinst-x86_64-35-1.2.iso and did an all-defaults installation.
*Before rebooting* I see the following:

/etc/resolv.conf is a normal file with the comment "Generated by NetworkManager" and
some specific addresses listed.

An identical file appears as /mnt/sysroot/etc/resolv.conf.
Anaconda copies it there. I assume it's the function called copy_resolv_conf_to_root().

Interestingly, the installation image is not using systemd-resolved, it doesn't even
have systemd-resolved.rpm installed. In the installed system, systemd-resolved.rpm is
present.

I see the following options:
1. remove copy_resolv_conf_to_root() from Anaconda. In general the approach of copying
   the file is broken. It contains dynamic configuration that can be useful after a reboot
   only if we are lucky.
2. make systemd-resolved.rpm create the symlink when installed into a chroot.
3. pull systemd-resolved.rpm into the installation image
4. some combination of the above, even possibly 1+2+3

For systemd that have a r/w root, not having the symlink at all (and creating it
after a reboot) is a good choice. But for systems with r/o root, we'd want to create the
symlink. How does this work on ostree, does it expect the symlink to be there?

I'm implementing 2 in systemd now. But I think the other steps should be considered too.

Also affects Silverblue.

(In reply to Chris Murphy from comment #7)
> Also affects Silverblue.

Yes, it has been reported in https://github.com/fedora-silverblue/issue-tracker/issues/192.

FEDORA-2022-f38f479b8f has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f38f479b8f

FEDORA-2022-f38f479b8f has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-f38f479b8f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-f38f479b8f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-f38f479b8f has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f38f479b8f

FEDORA-2022-f38f479b8f has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-f38f479b8f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-f38f479b8f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-f38f479b8f has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

I just did a fresh kickstart installation of fc35 and I still get: See attached.

Created attachment 1851083 [details]
End of Kickstart Installation

bash-5.1# /mnt/sysroot/usr/bin/systemd-run --version
systemd 249 (v249.4-2.fc35)

Shouldn't this be:
>/usr/bin/systemd-run --version
systemd 249 (v249.9-1.fc35)

Very same kickstart installation works for f34 and rawhide.

I also observe this installation regression on systemd-resolved-249.9-1.fc35:

[  176.607753] anaconda[1964]: Writing network configuration
[  178.611817] anaconda[1964]: The installation was stopped due to an error which occurred while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation.
[  178.611938] anaconda[1964]: The exact error message is:
[  178.612259] anaconda[1964]: Non interactive installation failed: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf'.
[  178.612595] anaconda[1964]: The installer will now terminate.

https://github.com/rhinstaller/anaconda/blob/a1ff62e575294b33d7870072d9befe98278224e2/pyanaconda/modules/network/installation.py#L228 seems to be a relevant place inside anaconda.

Oh, it's filed as bz2019579

This is still happening with a clean install of Fedora-Server-netinst-x86_64-Rawhide-20220124.n.0.iso systemd-250.3-1.fc36.x86_64

/etc/resolv.conf is a file, not a symlink.

-rw-r--r--. 1 root root        55 Jan 24 13:40 resolv.conf

$ cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.122.1
$ resolvectl
Global
         Protocols: LLMNR=resolve -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
  resolv.conf mode: foreign
Current DNS Server: 192.168.122.1
       DNS Servers: 192.168.122.1

Link 2 (enp1s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
Current DNS Server: 192.168.122.1
       DNS Servers: 192.168.122.1
$

(In reply to Zbigniew Jędrzejewski-Szmek from comment #5)

For the record:

> I see the following options:
> 1. remove copy_resolv_conf_to_root() from Anaconda. In general the approach
> of copying
>    the file is broken. It contains dynamic configuration that can be useful
> after a reboot
>    only if we are lucky.

PRs under review:
https://github.com/rhinstaller/anaconda/pull/3814
https://github.com/rhinstaller/anaconda/pull/3818

> 2. make systemd-resolved.rpm create the symlink when installed into a chroot.

This seems to be still not working? (bug #2018913)

> 3. pull systemd-resolved.rpm into the installation image

Done in https://github.com/rhinstaller/anaconda/pull/3783

So currently - Fedora-Rawhide-20220202.n.1 - where 1. and 2. is not working there is an /etc/resolv.conf containing content of target of the symlink from installer environment (/run/systemd/resolve/stub-resolv.conf)

When we merge 1. there will be no /etc/resolv.conf after installation (given 2. is still broken)

> 4. some combination of the above, even possibly 1+2+3

Because blocker ticket #581 should be used for blocker discussion, I copy my tests about this problem here so the developers can see it.


For Fedora 35 I have added the following script to kickstart %post section because otherwise DNS doesn't work (hostname resolution failed, hostname not found).

    if [[ ! -h /etc/resolv.conf ]] ; then
        # Link setzen
        ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
    fi

Now I have run a test kickstart installation with Fedora 36 Rawhide 20220205.n.1 nightly compose. I have not run the script above.

After the reboot file /etc/resolve.conf is a plain text file created (or updated?) by NetworkManager (# Generated by NetworkManager).
NetworkManager.service and systemd-resolved.service are both running.

So some commands and libaries and system calls use /etc/resolve.conf and some use systemd-resolved to do DNS name and ip resolution. This may lead to different results / answers depending on the configuration and the selected dns servers. It is not a consistent configuration.

But it is true: host and dig for example works resolving dns names, but both use the first dns server listed in /etc/resolve.conf .

If I remove file /etc/resolve.conf and reboot the system, then /etc/resolve.conf is a symbolic link to ../run/systemd/resolve/stub-resolv.conf .

But this symbolic link is created both with systemd-resolved.service and systemd-resolved.service not running (disabled). In the last case the symbolic link refers to a nonexistent file! Something (*) creates the symbolic link if the file doesn't exist even if systemd-resolved will not be started!

(*) Something is tmpfiles:

  /usr/lib/tmpfiles.d/systemd-resolve.conf

But if systemd-resolved is not running, the symbolic link should not exist, at least it should not refer to a nonexistent file. In this case NetworkManager should create /etc/resolve.conf as in earlier Fedora versions.

It seams that if the file /etc/resolve.conf exists either as plain file or symbolic link, it isn't changed. If the file doesn't exist, a symbolic link to the stub file is created.

But this is not a consistent and useful behavior. It may be right to not block the beta release, but I think this problem should not go to the final Fedora 36 release.

Also note that I reported bug 2022816 last November about this problem in Fedora 35. But it seems it doesn't solve it for Fedora 36.
And a similar problem was described in bug 1921057 in Janary 2021.

Setting status to POST since upstream patches are pending.

Discussed during the 2022-02-07 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug was made as we don't think this qualifies as a blocker under the release criteria currently, but there seems to be a strong case for FESCo to declare it a blocker as the current state is not at all how we want things to be, and this is an important area. At this time, instead of rejecting, we will file a ticket for FESCo to consider declaring it a blocker (as they are allowed to do).

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-02-07/f36-blocker-review.2022-02-07-17.00.txt

This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Fedora-Silverblue-ostree-x86_64-36-20220209.n.0.iso has
lrwxrwxrwx. 1 root root        39 Feb  9 21:33 resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

Clarification: following a default clean installation, the destination has the expected symlink.

Filed https://pagure.io/fesco/issue/2760 to ask FESCo to consider this bug.

Discussed during the 2022-02-14 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug was made as per last week's meeting. We don't think this is a blocker under the criteria but believe FESCo should consider it, a ticket has now been filed for FESCo: https://pagure.io/fesco/issue/2760, so we will wait for them to consider it.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-02-14/f36-blocker-review.2022-02-14-17.01.txt

This has been accepted as a FESCo blocker in https://pagure.io/fesco/issue/2760

Setting component to anaconda for merge of upstream PR 3884

FEDORA-2022-cf175d0e19 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-cf175d0e19

FEDORA-2022-cf175d0e19 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-cf175d0e19`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-cf175d0e19

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

I'm not clear on exactly what's still considered to be broken here, but I'd say we can push the anaconda update stable at least. The same code is in Rawhide already and doesn't seem to be causing problems.

Does anyone know what's still broken here and if anything beyond the anaconda update is needed to fix it?

Well, actually. I just tested a case where we did a static network config during install, including a pair of DNS servers. On boot into the installed system, the DNS servers set during installation are not configured. /etc/resolv.conf looks like this:

===

# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .

===

`resolvectl status` shows no name servers, and pinging www.google.com doesn't work.

This seems bad? Wasn't the point of copying resolv.conf to the installed system supposed to be so that the networking configuration set during install would be preserved on the installed system?

I also wonder - what would happen in a case where someone runs an install using a kickstart that leaves out systemd-resolved? We made systemd-resolved *default*, but we didn't make it *mandatory*. We still leave open the possibility that you could choose not to use it.

(In reply to Adam Williamson from comment #36)
 
> This seems bad? Wasn't the point of copying resolv.conf to the installed
> system supposed to be so that the networking configuration set during
> install would be preserved on the installed system?

I'd think the static dns configuration is preserved in NM ifcfg/keyfiles and the systemd-resolved should be configured based on it (via NM?).
Also see point 1. from comment #5.

(In reply to Adam Williamson from comment #37)
> I also wonder - what would happen in a case where someone runs an install
> using a kickstart that leaves out systemd-resolved? We made systemd-resolved
> *default*, but we didn't make it *mandatory*. We still leave open the
> possibility that you could choose not to use it.

I'd expect in case systemd-resolved is not present (does not handle /etc/resolv.conf) NM is getting opportunity to generate/control the /etc/resolv.conf. IIRC there is some synchronization between systemd-resolved and NM around ownership of /etc/resolv.conf.

(In reply to Adam Williamson from comment #36)
 
> `resolvectl status` shows no name servers, and pinging www.google.com
> doesn't work.

I did a quick check on *rawhide* - I've added an additional static configuration of a device in GUI and resolvectl status was showing the manually configured dns server on the installed system.

Hmm, wonder what the difference is. I'll test again manually here in that case. Thanks for trying it.

OK, I tried it here manually and indeed it worked OK, also looks like it would work without systemd-resolved as the DNS config is written to /etc/NetworkManager/system-connections . So I agree this is probably good.

Still don't know if there's anything else that needs fixing here.

FEDORA-2022-cf175d0e19 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

*** Bug 2022816 has been marked as a duplicate of this bug. ***

Searches for 'fedora missing etc/reslov.conf' brought me here. I don't know if it's related but my Fedora 37 system was upgraded from 35-> 36-> 37 and was missing this file this morning. I hadn't done anything different other than a hard shutdown yesterday, but not sure why that would make a symlink vanish. 

# cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory

And as expected nslookup fails:

# nslookup google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused
;; no servers could be reached

Creating the link resolves the issue:

# ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
# nslookup google.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	google.com
Address: 142.250.190.46
Name:	google.com
Address: 2607:f8b0:4009:803::200e

I'm brought here by the same search as in comment 45. I have merely installed fedora 37 and updated packages.
The symlink disappears on each reboot.