Hide Forgot
I just did a test install of F33 Workstation. From the live environment, post-installation, I see: $ pwd /mnt/sysroot/etc $ ls -l | grep resolv -rw-r--r--. 1 root root 729 Aug 30 11:19 resolv.conf The file says it is generated by systemd-resolved, but it is ***not a symlink***. That tells NetworkManager that the file is to be managed by NetworkManager, not by systemd. When the system is booted for the first time, we wind up with: $ ls -l /etc | grep resolv -rw-r--r--. 1 root root 55 Aug 30 10:22 resolv.conf $ cat /etc/resolv.conf # Generated by NetworkManager nameserver 192.168.122.1 So systemd-resolved is not working as intended. It should be a symlink to /run/systemd/resolve/stub-resolv.conf. I'm not sure why the file is not a symlink. We need to fix it ASAP or we'll have to defer systemd-resolved change to F34. :/
Network Manager can be configured to use an external resolv.conf generator for example by adding a new config file /etc/NetworkManager/conf.d/99-resolved.conf: [main] dns=systemd-resolved After adding this option you don't need to make /etc/resolv.conf a symbolic link to /run/NetworkManager/resolv.conf.
Discussed during the 2020-08-31 blocker review meeting: [0] The decision to classify this bug as an "AcceptedFreezeException" was made as it is a noticeable issue that cannot be fixed with an update. [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-08-31/f33-blocker-review.2020-08-31-16.00.txt
I'll add a scriptlet to create the symlink in %post (if [ $1 -eq 1 ]). I like Vitaly's idea, but I think this dropin would belong in the NM package. Also, I'm not sure what would happen if the file is already there and was managed by NM, and the drop-in is created: would NM "abandon" the old file letting it go stale?
FEDORA-2020-2255b438a2 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-2255b438a2
FEDORA-2020-2255b438a2 has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-2255b438a2` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-2255b438a2 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-2255b438a2 has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.
This needs to be tested and verified for both (a) fresh installs, once live media is available with this systemd update, and (b) upgrades from F32, once the update is available on mirrors.
(In reply to Michael Catanzaro from comment #7) > This needs to be tested and verified for both (a) fresh installs, once live > media is available with this systemd update, and (b) upgrades from F32, once > the update is available on mirrors. (I'll try tomorrow and see if it's working.)
*** Bug 1863041 has been marked as a duplicate of this bug. ***
(In reply to Michael Catanzaro from comment #7) > This needs to be tested and verified for both (a) fresh installs, once live > media is available with this systemd update, and (b) upgrades from F32, once > the update is available on mirrors. Verified fixed, both (a) and (b) scenarios are working properly. I'll send a mail to devel@ to advise that manual intervention is required for anyone who had previously installed or upgraded.
The dupe was accepted as a Beta blocker, so marking this with the same status for safety. If we need to re-open it for any reason.
I tested installing Fedora33 from the netinstall media. The symbolic link /etc/resolv.conf was not created as somehow it existed as a file created by NetworkManager. Extract from "dnf history info 1" Scriptlet output: 1 ln: failed to create symbolic link '/etc/resolv.conf': File exists 2 Created symlink /etc/systemd/system/sockets.target.wants/dbus.socket → /usr/lib/systemd/system/dbus.socket. 3 Created symlink /etc/systemd/user/sockets.target.wants/dbus.socket → /usr/lib/systemd/user/dbus.socket. 4 Created symlink /etc/systemd/system/dbus.service → /usr/lib/systemd/system/dbus-broker.service. 5 Created symlink /etc/systemd/user/dbus.service → /usr/lib/systemd/user/dbus-broker.service. 6 Running in chroot, ignoring request. 7 Running in chroot, ignoring request.
Reopening. I only tested Workstation Live install, which is now fixed. Sounds like netinstall is still broken.
OK, I think the creation of the symlink in systemd.spec's %triggerun is safe, but that's not run here anyway. The problem must be in %post, where we have no code to delete preexisting /etc/resolv.conf when creating our symlink. I don't think we need the safety checks from %triggerun here, though, because we don't need to create this symlink on upgrades. Looking at [1], I think it would be safe to remove /etc/resolv.conf and create the symlink only if $1 == 1 (i.e. if this is not a package upgrade). [1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/
Also consider that some users may have deliberately chosen to create /etc/resolv.conf as a regular file. Thus, don't touch during upgrade.
Another possible issue might be that the last time I tested systemd-resolved, wq-quick didn't support it, so it failed to update the resolve.conf properly. This was this summer on F32. So, it might be good to test this to see if this is still the case, and F33 will end up breaking wireguard.
FEDORA-2020-8c3c5881b2 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-8c3c5881b2
Thanks Zbigniew. Once systemd-246.4-2.fc33 reaches stable, we just need to test a netinstall (or other non-live) installation and ensure /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf. Then we can close this bug. (It should be really fixed this time, but better make sure.)
openQA suggests the change causes problems for mock: https://bugzilla.redhat.com/show_bug.cgi?id=1878924
(In reply to Adam Williamson from comment #19) > openQA suggests the change causes problems for mock: > https://bugzilla.redhat.com/show_bug.cgi?id=1878924 Seems like an understatement. Because we have the entry "resolve" in /etc/nsswitch, the file /etc/resolv.conf is not really used at all, so it is not a big problem if we leave it to be managed by NetworkManager -- at least for now. It is more important that the mock environment keeps on working.
Wouldn't this fix the issue in Mock? https://github.com/rpm-software-management/mock/commit/c44571821570e8cb0e65ef5a445313b793992706
Indeed, looks like the problem with mock is under control in bug #1878924. Thanks everyone. :)
(In reply to Neal Gompa from comment #21) > Wouldn't this fix the issue in Mock? > https://github.com/rpm-software-management/mock/commit/ > c44571821570e8cb0e65ef5a445313b793992706 Well there's actually https://github.com/rpm-software-management/mock/pull/631 for this issue. Adam, can we land that as a freeze exception given that this issue depends on it?
The commit Neal mentioned is related to the new systemd-nspawn option --resolv-conf=, which also brought a different _default_ behavior into nspawn. IOW, that fix is solving a different problem, not bug #1878924. But anyways, both fixes should be in the next Mock 2.6 release.
(In reply to Pavel Raiskup from comment #24) > But anyways, both fixes should be in the next Mock 2.6 > release. OK, but this is a F32 beta blocker, so we cannot really wait until Mock 2.6. :) Anyway, thanks so much for your quick response.
> OK, but this is a F32 beta blocker, Typo probably: F33. > so we cannot really wait until Mock 2.6. :) I am just not sure what you mean by "wait until Mock release". We could do that pretty promptly, if that helped.
FEDORA-2020-8c3c5881b2 has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-8c3c5881b2` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-8c3c5881b2 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
(In reply to Pavel Raiskup from comment #26) > > OK, but this is a F32 beta blocker, > > Typo probably: F33. Of course. :) > I am just not sure what you mean by "wait until Mock release". We could do > that pretty promptly, if that helped. It would certainly help a lot, because F33 beta is going to slip if we don't get an update today or tomorrow. If you could prepare a new build, then let's attach it to FEDORA-2020-8c3c5881b2 instead of submitting it as a separate bodhi update, to ensure it inherits the blocker status so it can go to stable, and to ensure FEDORA-2020-8c3c5881b2 doesn't break the compose.
Running the tests, for the sake of completeness (they should pass, so in an hour or so I can have a new mock build). Though I'm not sure. Is it really truth that mock should go to that update? I don't think that the machine which is working on making F33 composes is actually running Mock _on F33_. I suppose we'd have to have some infra help to get the new mock on (some) production systems..? Also, there's already: https://bodhi.fedoraproject.org/updates/FEDORA-2020-8b8b3ade7b That update is unfortunately flipping to stable for quite some time. Is it because of _this bugreport_? I'm asking because the update where the new Mock 2.6 will be needs to properly obsolete the FEDORA-2020-8b8b3ade7b (which means that we need to re-ship the mock-core-configs, too).
(In reply to Pavel Raiskup from comment #29) > Though I'm not sure. Is it really truth that mock should go to that > update? I don't think that the machine which is working on making F33 > composes is actually running Mock _on F33_. I suppose we'd have to have > some infra help to get the new mock on (some) production systems..? Uh-oh, you're probably right....
In bug 1878924, comment #7, Adam wrote: > So, I guess we need a new mock build for Rawhide at least. If we are still > planning to have the change in F33, we need a new mock build for F33 as well > and it should be added to the systemd update, I think. Adam, can you please double-check it makes sense to touch the systemd updates, per the previous two comments above?
Buidls are here, feel free to attach mock build(s) to any update(s): https://koji.fedoraproject.org/koji/packageinfo?packageID=2544 I have to go offline for today, but I'd create a new (separate) updates for those builds tomorrow if the builds aren't taken elsewhere.
IIUC, we need to push updates for mock in F32 and F31. We also need to update F33 at some point, but it doesn't need to be tied to the system update. Accordingly, I edited the mock+mock-core-configs update for F33 with the new build and created new mock updates for F31 and F32.
Thanks everyone....
ok, so we need to update builders to a mock with the "Fix _copy_config() for broken symlinks in dst=" patch? Looks like there's a bunch of other misc stuff in 2.6... do we want to pull all that in? Perhaps it would be safer for me to just apply that one patch to 2.4? Or are other changes needed?
AFAIK that is all the change that's needed to fix the problem.
So I backported the fix (hopefully correctly) to mock 2.4 and Kevin deployed it on the builders. Hopefully that should be enough to fix the compose. We'll see how the next Rawhide goes (the 0915.n.1 compose was done by untagging the systemd build).
This seems to be fixed in rawhide with the backport to mock 2.4
Yes, the Rawhide compose succeeded, so I guess the systemd update for F33 should be safe now.
FEDORA-2020-8c3c5881b2 has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.
Setting this back to ON_QA so we can have manual verification that the bug is fixed and all else looks well in the next F33 compose.
Michael (and others), can you please test with today's Branched compose and see if this all looks OK? https://kojipkgs.fedoraproject.org/compose/branched/Fedora-33-20200917.n.0/
/etc/resolv.conf is now created as a symlink as expected when installing from Fedora-Everything-netinst-x86_64-33-20200917.n.0.iso
Thanks Villy. I tested Fedora-Server-dvd-x86_64-33-20200917.n.0.iso and it's fixed there too. Closing.