Bug 1313085

Summary: Name resolution fails (resolv.conf is a broken systemd symlink) on livemedia-creator live image
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 24CC: admiller, anaconda-maint-list, bcl, bruno, bugzilla, dcbw, dustymabe, jdulaney, jfrieben, johannbg, juliux.pigface, kevin, kparal, lkundrak, lnykryn, msekleta, muadda, pbrobinson, projects.rg, psimerda, robatino, satellitgo, s, systemd-maint, vanmeeuwen+fedora, vpavlin, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: AcceptedBlocker
Fixed In Version: systemd-229-6.fc24 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-13 16:33:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1230431, 2020852    

Description Adam Williamson 2016-02-29 22:18:27 UTC
I booted the 2016-02-25 Rawhide Workstation live (which was before branching, so this is for F24) on a real laptop. It doesn't have an ethernet connection, so on boot there was no network.

I connected to my wifi network in the usual way (from the Shell network menu, entering the passphrase when asked) and that appeared to work fine, but name resolution does not work. Basic connectivity is there - I can ping 8.8.8.8.

There is no /etc/resolv.conf . /var/run/NetworkManager/resolv.conf does exist and looks correct. In the journal I see:

<warn> dns-mgr: could not commit DNS changes: Could not stat /etc/resolv.conf: No such file or directory

this is with NetworkManager-1.2.0-0.6.beta1.fc24 .

Proposing as an Alpha blocker, as a conditional violation of all network-related criteria (web browser 'working', for e.g.) at least with wireless networking on the Workstation live.

Comment 1 Adam Williamson 2016-02-29 22:22:43 UTC
Just noticed this is the old systemd symlink issue again; /etc/resolv.conf is a dangling symlink to ..//run/systemd/resolv/resolv.conf .

So this is like https://bugzilla.redhat.com/show_bug.cgi?id=1268974 and https://bugzilla.redhat.com/show_bug.cgi?id=1264364 , only affecting a live environment, not anaconda.

Comment 2 Adam Williamson 2016-02-29 22:30:10 UTC
This does indeed also affect a boot in a VM, so the wireless stuff has nothing to do with it, it's just systemd nerfing /etc/resolv.conf.

This was not broken in the 2016-02-15 Rawhide nightly, networking worked there. Neither systemd nor NetworkManager has changed significantly in the mean time, so my suspect here is livemedia-creator; the 2016-02-25 nightly is a livemedia-creator product, 2016-02-15 was produced by livecd-creator.

Comment 3 David Shea 2016-02-29 22:37:45 UTC
lorax does not define what's in the live environment.

Comment 4 Adam Williamson 2016-02-29 22:40:26 UTC
I thought lorax was the appropriate component for livemedia-creator issues, since livemedia-creator is a part of lorax.

This is not really a 'what's in the live image' issue, unless you're suggesting we remove either systemd or NetworkManager? Neither of which seems entirely viable. We can't really hack up systemd-tmpfiles in the kickstart, I don't think.

Comment 5 Adam Williamson 2016-02-29 22:55:55 UTC
So FWIW it seems like this works with livecd-creator lives because those appear to have a zero-length /etc/resolv.conf in them. That would prevent systemd-tmpfiles from creating the dangling symlink, and I guess NM is OK with replacing a regular file /etc/resolv.conf with its own symlink whereas it won't replace a dangling symlink.

By contrast there is no /etc/resolv.conf in a livemedia-creator produced live image at all; that means systemd-tmpfiles feels free to create its stupid dangling symlink, which NM then refuses to overwrite.

I don't know whether to say this bug is systemd's fault or NM's or livemedia-creator's, but it's definitely not the fault of the kickstarts.

Comment 6 Adam Williamson 2016-02-29 23:01:19 UTC
Possibly relevant, https://bugzilla.redhat.com/show_bug.cgi?id=1116651 - there is a note in livecd-tools `imgcreate/kickstart.py` in the `write_resolv` function which links out to that bug, it seems like we made livecd-creator try harder to create resolv.conf when this whole systemd crap originally showed up. https://github.com/rhinstaller/livecd-tools/pull/1 was the pull request.

Comment 7 Kamil Páral 2016-03-07 17:29:34 UTC
Discussed at today's blocker review meeting [1]. Accepted as an Alpha blocker -  it's a clear violation of "It must be possible to run the default web browser and a terminal application from all release-blocking desktop environments." and any other network-related criterion for both KDE and Workstation lives.

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2016-03-07/

Comment 8 Mairi Dulaney 2016-03-08 22:25:26 UTC
Seems to me that if systemd-resolvd is not used, systemd ought to be smart enough to not try to symlink things around ...

Comment 9 Zbigniew Jędrzejewski-Szmek 2016-03-09 04:43:07 UTC
So... I think it makes sense to not create the symlink in Fedora (upstream is a different issue).

There seem to be two possible scenarios for systemd-resolved: if it becomes widely accepted, nss-resolve will be used, which does not require /etc/resolv.conf. If it does not gain acceptance, something else will have to provide /etc/resolv.conf. In neither scenario is the link needed. So while I think that NM could behave smarter wrt. to a dangling symlink, it's just a waste of time to discuss a symlink that is going to go away anyway.

There is a small, I think, group which falls outside of those two possibilities: people who are using systemd-networkd and systemd-resolved with nss-dns. To cater to those users, the symlink could be created when resolved is started.

I'll try to make a patch to this effect.

Comment 10 Raphael Groner 2016-03-10 14:23:02 UTC
Does not work:
# rm -f /etc/resolv.conf ; echo 192.168.1.1 >/etc/resolv.conf
# cat /etc/resolv.conf
192.168.1.1
# ping fedoraproject.org
ping: fedoraproject.org: Temporary failure in name resolution

192.168.1.1 is the local DNS that's also assigned to the host machine via DHCP. Setting it via NetworkManager GUI does not help either.

Comment 11 Zbigniew Jędrzejewski-Szmek 2016-03-10 14:36:59 UTC
> # cat /etc/resolv.conf
> 192.168.1.1
That's invalid syntax.
I'm not sure what you're trying to say here.

Comment 12 Raphael Groner 2016-03-10 14:51:17 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #11)
> > # cat /etc/resolv.conf
> > 192.168.1.1
> That's invalid syntax.
> I'm not sure what you're trying to say here.

Oh, I tried to find a workaround and confused the syntax:
# cat /etc/resolv.conf
nameserver 192.168.1.1
# ping -c1 fedoraproject.org >/dev/null && echo works!
works!

Sorry for the noise.

Comment 13 Adam Williamson 2016-03-10 16:15:03 UTC
The workaround is just `rm -f /etc/resolv.conf; ln -s /var/run/NetworkManager/resolv.conf /etc`

Comment 14 Adam Williamson 2016-03-10 20:03:28 UTC
*** Bug 1312778 has been marked as a duplicate of this bug. ***

Comment 15 Joachim Frieben 2016-03-11 01:07:23 UTC
(In reply to Adam Williamson from comment #13)
As pointed out in https://bugzilla.redhat.com/show_bug.cgi?id=1312778#c6 it is even sufficient to remove the stub link by executing 'rm -f /etc/resolv.conf' and then deactivate and activate the network device again in the GNOME panel. Then the correct symlink is created automatically.

Comment 16 Fedora Update System 2016-03-12 03:59:36 UTC
systemd-229-6.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-64146d940c

Comment 17 Fedora Update System 2016-03-12 07:49:39 UTC
systemd-229-6.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-64146d940c

Comment 18 Fedora Update System 2016-03-13 16:32:55 UTC
systemd-229-6.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.