Bug 1873856 - resolv.conf misconfigured on fresh install
Summary: resolv.conf misconfigured on fresh install
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 33
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: BetaBlocker, F33BetaBlocker 1834846
TreeView+ depends on / blocked
 
Reported: 2020-08-30 15:36 UTC by Michael Catanzaro
Modified: 2020-09-18 18:24 UTC (History)
24 users (show)

Fixed In Version: systemd-246.4-1.fc33 systemd-246.4-2.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-18 18:24:52 UTC
Type: Bug


Attachments (Terms of Use)

Description Michael Catanzaro 2020-08-30 15:36:19 UTC
I just did a test install of F33 Workstation. From the live environment, post-installation, I see:

$ pwd
/mnt/sysroot/etc
$ ls -l | grep resolv
-rw-r--r--. 1 root root 729 Aug 30 11:19 resolv.conf

The file says it is generated by systemd-resolved, but it is ***not a symlink***. That tells NetworkManager that the file is to be managed by NetworkManager, not by systemd.

When the system is booted for the first time, we wind up with:

$ ls -l /etc | grep resolv
-rw-r--r--. 1 root root        55 Aug 30 10:22 resolv.conf
$ cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.122.1

So systemd-resolved is not working as intended. It should be a symlink to /run/systemd/resolve/stub-resolv.conf.

I'm not sure why the file is not a symlink. We need to fix it ASAP or we'll have to defer systemd-resolved change to F34. :/

Comment 1 Vitaly Zaitsev 2020-08-31 13:33:54 UTC
Network Manager can be configured to use an external resolv.conf generator for example by adding a new config file /etc/NetworkManager/conf.d/99-resolved.conf:

[main]
dns=systemd-resolved

After adding this option you don't need to make /etc/resolv.conf a symbolic link to /run/NetworkManager/resolv.conf.

Comment 2 Geoffrey Marr 2020-08-31 17:28:05 UTC
Discussed during the 2020-08-31 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedFreezeException" was made as it is a noticeable issue that cannot be fixed with an update.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-08-31/f33-blocker-review.2020-08-31-16.00.txt

Comment 3 Zbigniew Jędrzejewski-Szmek 2020-09-02 08:27:31 UTC
I'll add a scriptlet to create the symlink in %post (if [ $1 -eq 1 ]).

I like Vitaly's idea, but I think this dropin would belong in the NM package. Also,
I'm not sure what would happen if the file is already there and was managed by NM, and the
drop-in is created: would NM "abandon" the old file letting it go stale?

Comment 4 Fedora Update System 2020-09-02 11:07:28 UTC
FEDORA-2020-2255b438a2 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-2255b438a2

Comment 5 Fedora Update System 2020-09-02 16:20:26 UTC
FEDORA-2020-2255b438a2 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-2255b438a2`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-2255b438a2

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 6 Fedora Update System 2020-09-08 17:04:30 UTC
FEDORA-2020-2255b438a2 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 7 Michael Catanzaro 2020-09-08 17:16:52 UTC
This needs to be tested and verified for both (a) fresh installs, once live media is available with this systemd update, and (b) upgrades from F32, once the update is available on mirrors.

Comment 8 Michael Catanzaro 2020-09-08 20:10:43 UTC
(In reply to Michael Catanzaro from comment #7)
> This needs to be tested and verified for both (a) fresh installs, once live
> media is available with this systemd update, and (b) upgrades from F32, once
> the update is available on mirrors.

(I'll try tomorrow and see if it's working.)

Comment 9 Michael Catanzaro 2020-09-09 15:06:53 UTC
*** Bug 1863041 has been marked as a duplicate of this bug. ***

Comment 10 Michael Catanzaro 2020-09-09 18:46:51 UTC
(In reply to Michael Catanzaro from comment #7)
> This needs to be tested and verified for both (a) fresh installs, once live
> media is available with this systemd update, and (b) upgrades from F32, once
> the update is available on mirrors.

Verified fixed, both (a) and (b) scenarios are working properly.

I'll send a mail to devel@ to advise that manual intervention is required for anyone who had previously installed or upgraded.

Comment 11 Adam Williamson 2020-09-09 19:23:14 UTC
The dupe was accepted as a Beta blocker, so marking this with the same status for safety. If we need to re-open it for any reason.

Comment 12 Villy Kruse 2020-09-10 17:07:24 UTC
I tested installing Fedora33 from the netinstall media.  The symbolic link /etc/resolv.conf was not created as somehow it existed as a file created by NetworkManager.

Extract from "dnf history info 1"

Scriptlet output:
   1 ln: failed to create symbolic link '/etc/resolv.conf': File exists
   2 Created symlink /etc/systemd/system/sockets.target.wants/dbus.socket → /usr/lib/systemd/system/dbus.socket.
   3 Created symlink /etc/systemd/user/sockets.target.wants/dbus.socket → /usr/lib/systemd/user/dbus.socket.
   4 Created symlink /etc/systemd/system/dbus.service → /usr/lib/systemd/system/dbus-broker.service.
   5 Created symlink /etc/systemd/user/dbus.service → /usr/lib/systemd/user/dbus-broker.service.
   6 Running in chroot, ignoring request.
   7 Running in chroot, ignoring request.

Comment 13 Michael Catanzaro 2020-09-10 17:45:31 UTC
Reopening. I only tested Workstation Live install, which is now fixed. Sounds like netinstall is still broken.

Comment 14 Michael Catanzaro 2020-09-10 18:00:11 UTC
OK, I think the creation of the symlink in systemd.spec's %triggerun is safe, but that's not run here anyway. The problem must be in %post, where we have no code to delete preexisting /etc/resolv.conf when creating our symlink. I don't think we need the safety checks from %triggerun here, though, because we don't need to create this symlink on upgrades. Looking at [1], I think it would be safe to remove /etc/resolv.conf and create the symlink only if $1 == 1 (i.e. if this is not a package upgrade).

[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/

Comment 15 Villy Kruse 2020-09-10 19:08:21 UTC
Also consider that some users may have deliberately chosen to create /etc/resolv.conf as a regular file.  Thus, don't touch during upgrade.

Comment 16 mattias.eriksson 2020-09-13 07:50:17 UTC
Another possible issue might be that the last time I tested systemd-resolved, wq-quick didn't support it, so it failed to update the resolve.conf properly. This was this summer on F32. So, it might be good to test this to see if this is still the case, and F33 will end up breaking wireguard.

Comment 17 Fedora Update System 2020-09-14 18:29:24 UTC
FEDORA-2020-8c3c5881b2 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-8c3c5881b2

Comment 18 Michael Catanzaro 2020-09-14 18:38:49 UTC
Thanks Zbigniew.

Once systemd-246.4-2.fc33 reaches stable, we just need to test a netinstall (or other non-live) installation and ensure /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf. Then we can close this bug. (It should be really fixed this time, but better make sure.)

Comment 19 Adam Williamson 2020-09-14 23:31:26 UTC
openQA suggests the change causes problems for mock:
https://bugzilla.redhat.com/show_bug.cgi?id=1878924

Comment 20 Villy Kruse 2020-09-15 05:42:39 UTC
(In reply to Adam Williamson from comment #19)
> openQA suggests the change causes problems for mock:
> https://bugzilla.redhat.com/show_bug.cgi?id=1878924

Seems like an understatement.


Because we have the entry "resolve" in /etc/nsswitch, the file /etc/resolv.conf is not really used at all, so it is not a big problem if we leave it to be managed by NetworkManager -- at least for now.

It is more important that the mock environment keeps on working.

Comment 21 Neal Gompa 2020-09-15 13:44:34 UTC
Wouldn't this fix the issue in Mock? https://github.com/rpm-software-management/mock/commit/c44571821570e8cb0e65ef5a445313b793992706

Comment 22 Michael Catanzaro 2020-09-15 14:05:59 UTC
Indeed, looks like the problem with mock is under control in bug #1878924. Thanks everyone. :)

Comment 23 Michael Catanzaro 2020-09-15 14:09:32 UTC
(In reply to Neal Gompa from comment #21)
> Wouldn't this fix the issue in Mock?
> https://github.com/rpm-software-management/mock/commit/
> c44571821570e8cb0e65ef5a445313b793992706

Well there's actually https://github.com/rpm-software-management/mock/pull/631 for this issue. Adam, can we land that as a freeze exception given that this issue depends on it?

Comment 24 Pavel Raiskup 2020-09-15 14:11:29 UTC
The commit Neal mentioned is related to the new systemd-nspawn option
--resolv-conf=, which also brought a different _default_ behavior into
nspawn.  IOW, that fix is solving a different problem, not
bug #1878924.  But anyways, both fixes should be in the next Mock 2.6
release.

Comment 25 Michael Catanzaro 2020-09-15 14:13:51 UTC
(In reply to Pavel Raiskup from comment #24)
> But anyways, both fixes should be in the next Mock 2.6
> release.

OK, but this is a F32 beta blocker, so we cannot really wait until Mock 2.6. :)

Anyway, thanks so much for your quick response.

Comment 26 Pavel Raiskup 2020-09-15 14:35:50 UTC
> OK, but this is a F32 beta blocker,

Typo probably: F33.

> so we cannot really wait until Mock 2.6. :)

I am just not sure what you mean by "wait until Mock release".  We could do
that pretty promptly, if that helped.

Comment 27 Fedora Update System 2020-09-15 14:53:27 UTC
FEDORA-2020-8c3c5881b2 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-8c3c5881b2`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-8c3c5881b2

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 28 Michael Catanzaro 2020-09-15 14:58:48 UTC
(In reply to Pavel Raiskup from comment #26)
> > OK, but this is a F32 beta blocker,
> 
> Typo probably: F33.

Of course. :)

> I am just not sure what you mean by "wait until Mock release".  We could do
> that pretty promptly, if that helped.

It would certainly help a lot, because F33 beta is going to slip if we don't get an update today or tomorrow. If you could prepare a new build, then let's attach it to FEDORA-2020-8c3c5881b2 instead of submitting it as a separate bodhi update, to ensure it inherits the blocker status so it can go to stable, and to ensure FEDORA-2020-8c3c5881b2 doesn't break the compose.

Comment 29 Pavel Raiskup 2020-09-15 15:26:12 UTC
Running the tests, for the sake of completeness (they should pass, so in
an hour or so I can have a new mock build).

Though I'm not sure.  Is it really truth that mock should go to that
update?  I don't think that the machine which is working on making F33
composes is actually running Mock _on F33_.  I suppose we'd have to have
some infra help to get the new mock on (some) production systems..?

Also, there's already:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-8b8b3ade7b
That update is unfortunately flipping to stable for quite some time.
Is it because of _this bugreport_?  I'm asking because the update
where the new Mock 2.6 will be needs to properly obsolete the
FEDORA-2020-8b8b3ade7b (which means that we need to re-ship the
mock-core-configs, too).

Comment 30 Michael Catanzaro 2020-09-15 15:44:32 UTC
(In reply to Pavel Raiskup from comment #29)
> Though I'm not sure.  Is it really truth that mock should go to that
> update?  I don't think that the machine which is working on making F33
> composes is actually running Mock _on F33_.  I suppose we'd have to have
> some infra help to get the new mock on (some) production systems..?

Uh-oh, you're probably right....

Comment 31 Pavel Raiskup 2020-09-15 15:50:28 UTC
In bug 1878924, comment #7, Adam wrote:

> So, I guess we need a new mock build for Rawhide at least. If we are still
> planning to have the change in F33, we need a new mock build for F33 as well
> and it should be added to the systemd update, I think.

Adam, can you please double-check it makes sense to touch the systemd
updates, per the previous two comments above?

Comment 32 Pavel Raiskup 2020-09-15 16:45:52 UTC
Buidls are here, feel free to attach mock build(s) to any update(s):
https://koji.fedoraproject.org/koji/packageinfo?packageID=2544

I have to go offline for today, but I'd create a new (separate)
updates for those builds tomorrow if the builds aren't taken elsewhere.

Comment 33 Zbigniew Jędrzejewski-Szmek 2020-09-15 17:11:05 UTC
IIUC, we need to push updates for mock in F32 and F31. We also need to update F33 at some point,
but it doesn't need to be tied to the system update. Accordingly, I edited the mock+mock-core-configs
update for F33 with the new build and created new mock updates for F31 and F32.

Comment 34 Michael Catanzaro 2020-09-15 17:44:32 UTC
Thanks everyone....

Comment 35 Kevin Fenzi 2020-09-15 19:57:59 UTC
ok, so we need to update builders to a mock with the "Fix _copy_config() for broken symlinks in dst=" patch?

Looks like there's a bunch of other misc stuff in 2.6... do we want to pull all that in? Perhaps it would be safer for me to just apply that one patch to 2.4? Or are other changes needed?

Comment 36 Adam Williamson 2020-09-15 20:02:46 UTC
AFAIK that is all the change that's needed to fix the problem.

Comment 37 Adam Williamson 2020-09-16 00:27:20 UTC
So I backported the fix (hopefully correctly) to mock 2.4 and Kevin deployed it on the builders. Hopefully that should be enough to fix the compose. We'll see how the next Rawhide goes (the 0915.n.1 compose was done by untagging the systemd build).

Comment 38 Mohan Boddu 2020-09-16 13:05:41 UTC
This seems to be fixed in rawhide with the backport to mock 2.4

Comment 39 Adam Williamson 2020-09-16 17:26:49 UTC
Yes, the Rawhide compose succeeded, so I guess the systemd update for F33 should be safe now.

Comment 40 Fedora Update System 2020-09-17 00:18:02 UTC
FEDORA-2020-8c3c5881b2 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 41 Adam Williamson 2020-09-17 01:37:43 UTC
Setting this back to ON_QA so we can have manual verification that the bug is fixed and all else looks well in the next F33 compose.

Comment 42 Adam Williamson 2020-09-18 00:10:14 UTC
Michael (and others), can you please test with today's Branched compose and see if this all looks OK? https://kojipkgs.fedoraproject.org/compose/branched/Fedora-33-20200917.n.0/

Comment 43 Villy Kruse 2020-09-18 09:24:07 UTC
/etc/resolv.conf is now created as a symlink as expected when installing from 
Fedora-Everything-netinst-x86_64-33-20200917.n.0.iso

Comment 44 Michael Catanzaro 2020-09-18 18:24:52 UTC
Thanks Villy.

I tested Fedora-Server-dvd-x86_64-33-20200917.n.0.iso and it's fixed there too. Closing.


Note You need to log in before you can comment on or make changes to this bug.