Bug 1536397

Vladimir, please provide Engine log, cloud init logs (and cloudinit version!). In particular, if you think the problem is a bad network_data.json, please attach it here.

Created attachment 1384447 [details]
cloud-init.log , engine.log, network_data.json

Hi Ryan,
I am trying to fix the dns nameserver issue in our network_data.json (key should be 'dns' - according to what Vladimir mentions in the first comment):

When I apply both a network configuration and a dns nameserver configuration:

{
  "links" : [ {
    "name" : "eth0",
    "id" : "eth0",
    "type" : "vif"
  } ],
  "services" : [ {
    "address" : "8.8.8.8",
    "type" : "dns"
  }, {
    "address" : "111.111.111.101",
    "type" : "dns"
  } ],
  "networks" : [ {
    "netmask" : "255.255.255.0",
    "link" : "eth0",
    "id" : "eth0",
    "ip_address" : "192.168.122.111",
    "type" : "ipv4",
    "gateway" : "192.168.122.1"
  } ]
}

I get the expected result in /etc/resolv.conf:

# cat /etc/resolv.conf
; Created by cloud-init on instance boot automatically, do not edit.
;
; generated by /usr/sbin/dhclient-script
nameserver 192.168.122.1
nameserver 8.8.8.8
nameserver 111.111.111.101

------------------------------------------------

But when i specify only the nameservers in my configuration:

{
  "services" : [ {
    "address" : "8.8.8.8",
    "type" : "dns"
  }, {
    "address" : "111.111.111.101",
    "type" : "dns"
  } ]
}

they do not get written to /etc/resolv.conf:

; generated by /usr/sbin/dhclient-script
nameserver 192.168.122.1

----------------------------------------------------
i am using the latest rpm you gave me:

# rpm -q cloud-init
cloud-init-0.7.9-20test1.el7.x86_6

Any idea?
Thanks

update for above:

I ran some more tests and found that for network_data.json:

{
  "services" : [ {
    "address" : "8.8.8.8",
    "type" : "dns"
  }, {
    "address" : "111.111.111.101",
    "type" : "dns"
  } ]
}

- if there is any content in /etc/resolv.conf - the nameservers from the network_data.json of the latest run are NOT added to /etc/resolv.conf.
- if /etc/resolv.conf is completely erased, the nameservers from the network_data.json of the latest run are added to it. 
- so my previous suspicion that networks need to be present in network_data.json is not true - the problem is that there was content in /etc/resolv.conf

---------------------------------------------------------------
here is /etc/resolv.conf for the above json:

# cat /etc/resolv.conf
; Created by cloud-init on instance boot automatically, do not edit.
;
nameserver 8.8.8.8
nameserver 111.111.111.101

And after several consequetive runs without erasing:

; Created by cloud-init on instance boot automatically, do not edit.
;
; Created by cloud-init on instance boot automatically, do not edit.
;
; Created by cloud-init on instance boot automatically, do not edit.
;
nameserver 8.8.8.8
nameserver 111.111.111.101


---------------------------------------------------------
BTW, AFAIU the initial content generated by the dhclient script is because the initial centos image i downloaded had cloud-init-0.7.9.9 on it, and i had to start the vm without any network configuration in order to install cloud-init-0.7.9-20test1.el7.x86_6.

After some more tests, I think I can narrow down the problem:

If the current network_data.json contains a static ip network, or if there already is an ifcfg with a static ip configuration - then the dhclient is not invoked and the dns's from the services clause are applied by cloud-init to /etc/resolv.conf
Otherwise, dhclient is called and over-writes the content in /etc/resolv.conf resulting in no dns's from the network_data.json there.

looking at cloud-init code at sysconfig.py i see that indeed dns's specified for an interface is only written if the configuration of the interface is static. is this also true for the dns's specified under the services clause?

(In reply to eraviv from comment #8)
> looking at cloud-init code at sysconfig.py i see that indeed dns's specified
> for an interface is only written if the configuration of the interface is
> static. is this also true for the dns's specified under the services clause?

If you specify nameservers in the services stanza, cloud-init will drop a file in /etc/NetworkManager/conf.d telling it to not manage dns so that nameserver configuration from dhcp won't clobber what's been written to /etc/resolv.conf by cloud-init. The entries you provided should be there.

If there have been previous runs, cloud-init will load all the nameservers that exist in /etc/resolv.conf and write them back out, but it enforces a 3 nameserver limit per resolv.conf(5), so that may potentially cause problems, causing output to be truncated. Output loaded from resolv.conf will be written before new configuration.

so what is the correct method of passing nameservers to cloud-init?

- only on subnets if subnets are configured
- only on the services stanza
- on both
- on subnets if they are static and on services stanza if they are dhcp
- other?

is this documented anywhere?

thanks

WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops

Verified on 4.3.0-0.4.master.20181207184726.git7928cae.el7.
The following test scenarios were tested:
1. no networks, only dns nameservers
> The dns nameservers were written to /etc/resolv.conf and to network_data.json as it should.
2. no networks, only search domains
> Nothing was written to /etc/resolv.conf and and the file network_data.json wasn't created as it should.
3. no networks, dns nameservers + search domains.
> The dns nameservers were written to /etc/resolv.conf and to network_data.json as it should.
4. network with static\dhcp ipv4, dns nameservers + search domains
> The dns nameservers were written to /etc/resolv.conf and all the configurations were written to network_data.json as it should.
5. network with static\dhcp ipv6, dns nameservers + search domains
> The dns nameservers were written to /etc/resolv.conf and all the configurations were written to network_data.json as it should.
6. network with static\dhcp ipv4 + static\dhcp ivp6, dns nameservers + search domains
> The dns nameservers were written to /etc/resolv.conf and all the configurations were written to network_data.json as it should.

WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1085