Can you attach the contents of openstack/latest/network_data.json from the config drive?

# cat openstack/latest/network_data.json
{"services": [{"type": "dns", "address": "10.11.142.1"}, {"type": "dns", "address": "10.11.142.2"}], "networks": [{"network_id": "3d065b27-8f5f-4fe0-abad-0a05a0fdba75", "type": "ipv4", "netmask": "255.255.252.0", "link": "tapf80c0351-5d", "routes": [{"netmask": "0.0.0.0", "network": "0.0.0.0", "gateway": "10.8.231.254"}], "ip_address": "10.8.229.8", "id": "network0"}], "links": [{"ethernet_mac_address": "fa:16:3e:2b:c5:5e", "mtu": 1500, "type": "ovs", "id": "tapf80c0351-5d", "vif_id": "f80c0351-5d4a-450b-9c4d-35800dadcecc"}]}

I'm testing this using cloud-init-0.7.9-7, which includes some patches that impact network config generation.

Using your network_data.json, I see that as you reported, /etc/resolv.conf is configured correctly, although the DNS servers are not added to ifcfg-eth0.  The resolver configuration in /etc/resolv.conf appears to survive both an ifdown/ifup and a reboot.  That is, it does not get clobbered by NetworkManager.

NetworkManager will never update /etc/resolv.conf directly; it only updates /run/NetworkManager/resolv.conf, and if /etc/resolv.conf is simply a file rather than a symlink, that information will never be removed by NetworkManager.

I agree that the DNS server information *ought* to be added to ifcfg-eth0, but is the current behavior actively causing a problem?

Actually, I lied about how NM treats /etc/resolv.conf, but the rest of the #4 stands; everything seems to persist as expected and I was not able to generate a problem.  For the record, I discussed this with the NetworkManager folks on #nm:

09:27 <larsks> Will NM update /etc/resolv.conf if it's a file? Or only if it is a symlink to e.g. /run/NetworkManager/resolv.conf?
09:30 <thaller> larsks, depends on rc-manager setting. But both for "file" and "symlink" it would
09:30 <thaller> described in `man NetworkManager.conf`
09:30 <larsks> I was just reading through that.  To make sure I've reading it correctly...when rc-manager is "file", it will just update /etc/resolv.conf in place if it's a file, right?
09:32 <thaller> with rc-manager=file:    yes, a file will be updated. And a symlink will be followed.   
09:32 <larsks> ..and if /etc/resolv.conf is pre-configured with nameserver entries, and ifcfg-eth* does not contain any DNSn=x.x.x.x entries, the pre-existing nameserver configuration should persist?
09:32 <larsks> (assuming static interface configs and not dhcP)
09:33 <thaller> no. NM would not pick up configuration from /etc/resolv.conf. It would update resolve.conf with the information it has (no name servers)...
09:34 <larsks> Huh. So now I am confused.
09:34 <larsks> THis question stems from how cloud-init is setting up resolv.conf.
09:34 <larsks> It is editing nameserver entries to /etc/resolv.conf, but does not include that information in the generate interface config files (ifcfg-eth*).
09:34 <larsks> Howerver, running 'ifdown eth0/ifup eth0' does not clobber /etc/resolv.conf.
09:35 <larsks> This is on a CentOS 7 image, running NetworkManager 1.8.0.
09:37 <thaller> larsks, I think we changed, that NM will not write out resolv.conf it has no nameservers... in that case, writing out an empty resolve.conf is deemed pointless, and it would leave the existing one in place (bengal ??).    
09:39 <bengal> thaller: correct
09:39 <larsks> bengal: thaller: awesome, thanks.
09:39 <thaller> (however, that does not mean that NM could merge DNS configuration from outside sources. Meaning: (a) you either manage all your DNS with NM exclusively; or (b) you tell NM to not manage it (rc-manager=unmanged, dns=none);     or (c) you have an external service that can manage nameserver information from multiple sources (sd-resolved, resolveconf, netconf, your own script)
09:40 <larsks> Right. THe question was simply whether the current behavior of cloud-init was going to be problematic.
09:40 <larsks> I think the answer is "could be".
09:41 <thaller> yes "could be". To be sure, drop a configuration snippet with "dns=none" to /etc/NetworkManager/conf.d
09:41 <larsks> RIght. Or correctly configure the interface config files.
09:41 <thaller> or (rc-manager=unmanaged)


Thinking about this for a bit, the problem would seem to exist at a level higher than cloud-init.  Take a look at your network_data.json; you'll note that the DNS nameserver information is *not* interface scoped.  That means that is simply not possible for cloud-init to correctly add the nameserver information to the ifcfg-eth* files, because there is no way to associate the nameservers with a particular interface.

I think the correct short-term solution is to add the dns=none configuration setting to NetworkManager.  I've opened upstream bug https://bugs.launchpad.net/cloud-init/+bug/1693251 with that request.

The longer-term solution would be to fix Nova to provide interface-scoped nameserver information.

See also:

https://bugs.launchpad.net/nova/+bug/1693265

Hi Lars,

So what you are saying does make sense, however I can provide a bit more context for our specific use case.

We are attempting to install Openshift on Openstack using provider networks. This means that the network has no dhcp agent, so relies entirely on config drive to configure the networking correctly.

Initially I was working around this issue by manually writing the dns entries to /etc/resolv.conf, but the team installing Openshift came back to me and said that this is not a satisfactory solution, as the Openshift installer restarts and reconfigures the networking (to use OVS for Openshifts internal networking), which then causes either NetworkManager or something else to remove those entries from /etc/resolv.conf. In fact, according to them the only correct way is to have the dns entries written into /etc/sysconfig/network-scripts/ifcfg* as DNS1 and DNS2 so that they correctly survive and apply regardless of what Openshift does with the networking.

This makes sense to me, but your point about dns information being not scoped to an interface also makes sense, so unfortunately we might be in between a rock and a hard place. That being said, I'm sure you can appreciate the motivation for us to have Openshift on Openstack be a first class citizen, so we unfortunately might need to figure something out.

Fix accepted by upstream in launchpad commit 67bab5bb804e2346673430868935f6bbcdb88f13

Can we resolve this? Looks like it did in fact make it into 7.4

Verified on: cloud-init-0.7.9-23.el7


fedora@fedora-5084 ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0 
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=static
DEFROUTE=yes
DEVICE=eth0
GATEWAY=192.168.100.1
HWADDR=fa:16:3e:4b:45:ea
IPADDR=192.168.100.10
MTU=1450
NETMASK=255.255.255.0
ONBOOT=yes
TYPE=Ethernet
USERCTL=no


[fedora@fedora-5084 ~]$ cat /etc/resolv.conf 
; Created by cloud-init on instance boot automatically, do not edit.
;
nameserver 10.35.28.28
search localdomain

[fedora@fedora-5084 ~]$ cat /etc/NetworkManager/
conf.d/       dispatcher.d/ 
[fedora@fedora-5084 ~]$ cat /etc/NetworkManager/conf.d/99-cloud-init.conf 
# Created by cloud-init on instance boot automatically, do not edit.
#
[main]
dns = none

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0806