Bug 1299689

Summary: overcloud nodes loosing their nameserver configuration in /etc/resolv.conf minutes after deployment starts
Product: Red Hat OpenStack Reporter: wes hayutin <whayutin>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.0 (Kilo)CC: apevec, lhh, mburns, rhel-osp-director-maint, srevivo, yeylon
Target Milestone: ---Keywords: Automation, AutomationBlocker
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-19 16:09:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description wes hayutin 2016-01-19 01:50:14 UTC 
Description of problem:

overcloud nodes loosing their dns configuration in /etc/resolv.conf minutes after deployment starts

Recreate:
1. Set up the undercloud 
2. Ensure the the overcloud is configured to receive dns configuration via dhcp
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html-single/Director_Installation_and_Usage/index.html#sect-Setting_a_Nameserver_for_the_Overcloud
3. Deploy the cloud
4. wait for the nodes to spawn/boot
5. immediately ssh to a overcloud node
6. watch the /etc/resolv.conf

Note that the /etc/resolv.conf is initially configured correctly and name resolution works

wait a few minutes as the node continues to be configured.

Results:
The overcloud node looses it's nameserver config after a few minutes:


[stack@instack ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 00d7a348-16e4-40e6-abaf-3f696b5423b3 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| feba8f3b-9dfc-4b8a-bf4b-f05ba42a9fcd | overcloud-compute-0     | BUILD  | spawning   | NOSTATE     | ctlplane=192.0.2.9  |
| 362baa85-1784-4333-a3e6-9ebb2f895ba3 | overcloud-controller-0  | BUILD  | spawning   | NOSTATE     | ctlplane=192.0.2.10 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ ssh heat-admin.2.8
ssh: connect to host 192.0.2.8 port 22: Connection refused
[stack@instack ~]$ ssh heat-admin.2.8
ssh: connect to host 192.0.2.8 port 22: Connection refused
[stack@instack ~]$ ssh heat-admin.2.8
The authenticity of host '192.0.2.8 (192.0.2.8)' can't be established.
ECDSA key fingerprint is a6:d9:20:00:9f:49:e9:21:95:0a:5b:79:b4:ee:0a:ba.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.8' (ECDSA) to the list of known hosts.
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain
nameserver 192.168.122.1
[heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search localdomain


# No nameservers found; try putting DNS servers into your
# ifcfg files in /etc/sysconfig/network-scripts like so:
#
# DNS1=xxx.xxx.xxx.xxx
# DNS2=xxx.xxx.xxx.xxx
# DOMAIN=lab.foo.com bar.foo.com
Comment 1 wes hayutin 2016-01-19 01:51:38 UTC 
Note:
RHSM enabled, cdn repos in play here:

Setting up Package Sacks
pkgsack time: 0.091
Repo-id      : rhel-7-server-aus-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux 7 Server - AUS (RPMs)
Repo-revision: 1452806283
Repo-updated : Thu Jan 14 16:20:09 2016
Repo-pkgs    : 10,110
Repo-size    : 11 G
Repo-baseurl : https://cdn.redhat.com/content/aus/rhel/server/7/7Server/x86_64/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:19 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-eus-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux 7 Server - Extended Update Support (RPMs)
Repo-revision: 1452806256
Repo-updated : Thu Jan 14 16:19:30 2016
Repo-pkgs    : 10,110
Repo-size    : 11 G
Repo-baseurl : https://cdn.redhat.com/content/eus/rhel/server/7/7Server/x86_64/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:20 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-extras-rpms/x86_64
Repo-name    : Red Hat Enterprise Linux 7 Server - Extras (RPMs)
Repo-revision: 1449807908
Repo-updated : Thu Dec 10 23:25:09 2015
Repo-pkgs    : 143
Repo-size    : 290 M
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/extras/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:20 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-htb-rpms/x86_64
Repo-name    : Red Hat Enterprise Linux 7 Server HTB (RPMs)
Repo-revision: 1447273278
Repo-updated : Wed Nov 11 15:21:18 2015
Repo-pkgs    : 6,770
Repo-size    : 5.5 G
Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:20 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-openstack-7.0-director-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux OpenStack Platform 7.0 director for RHEL 7 (RPMs)
Repo-revision: 1450716322
Repo-updated : Mon Dec 21 11:45:22 2015
Repo-pkgs    : 70
Repo-size    : 11 M
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/openstack-director/7.0/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:21 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-openstack-7.0-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux OpenStack Platform 7.0 for RHEL 7 (RPMs)
Repo-revision: 1452467561
Repo-updated : Sun Jan 10 18:12:41 2016
Repo-pkgs    : 923
Repo-size    : 387 M
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/openstack/7.0/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:21 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-optional-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux 7 Server - Optional (RPMs)
Repo-revision: 1452806854
Repo-updated : Thu Jan 14 16:28:48 2016
Repo-pkgs    : 8,181
Repo-size    : 9.3 G
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/optional/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:21 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux 7 Server (RPMs)
Repo-revision: 1452806059
Repo-updated : Thu Jan 14 16:16:31 2016
Repo-pkgs    : 10,110
Repo-size    : 11 G
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:22 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-rt-beta-rpms/x86_64
Repo-name    : Red Hat Enterprise Linux for Real Time Beta (RHEL 7 Server) (RPMs)
Repo-revision: 1441144237
Repo-updated : Tue Sep  1 17:50:37 2015
Repo-pkgs    : 15
Repo-size    : 199 M
Repo-baseurl : https://cdn.redhat.com/content/beta/rhel/server/7/x86_64/rt/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:22 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-rt-htb-rpms/x86_64
Repo-name    : Red Hat Enterprise Linux for Real Time HTB (RHEL 7 Server) (RPMs)
Repo-revision: 1447153130
Repo-updated : Tue Nov 10 05:59:03 2015
Repo-pkgs    : 53
Repo-size    : 835 M
Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/rt/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:22 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-7-server-rt-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux for Real Time (RHEL 7 Server) (RPMs)
Repo-revision: 1449084995
Repo-updated : Wed Dec  2 14:36:35 2015
Repo-pkgs    : 70
Repo-size    : 1.1 G
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rt/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:23 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-ha-for-rhel-7-server-eus-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux High Availability (for RHEL 7 Server) - Extended Update Support (RPMs)
Repo-revision: 1449686227
Repo-updated : Wed Dec  9 13:37:10 2015
Repo-pkgs    : 190
Repo-size    : 214 M
Repo-baseurl : https://cdn.redhat.com/content/eus/rhel/server/7/7Server/x86_64/highavailability/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:23 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-ha-for-rhel-7-server-htb-rpms/x86_64
Repo-name    : Red Hat Enterprise Linux High Availability (for RHEL 7 Server) HTB (RPMs)
Repo-revision: 1446656084
Repo-updated : Wed Nov  4 11:54:46 2015
Repo-pkgs    : 76
Repo-size    : 48 M
Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/highavailability/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:23 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-ha-for-rhel-7-server-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs)
Repo-revision: 1449686149
Repo-updated : Wed Dec  9 13:35:54 2015
Repo-pkgs    : 190
Repo-size    : 214 M
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/highavailability/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:24 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-lb-for-rhel-7-server-htb-rpms/x86_64
Repo-name    : Red Hat Enterprise Linux Load Balancer (for RHEL 7 Server) HTB (RPMs)
Repo-revision: 1403809660
Repo-updated : Thu Jun 26 15:07:41 2014
Repo-pkgs    : 0
Repo-size    : 0
Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/loadbalancer/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:24 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-rs-for-rhel-7-server-eus-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Server) - Extended Update Support (RPMs)
Repo-revision: 1452215589
Repo-updated : Thu Jan  7 20:13:11 2016
Repo-pkgs    : 221
Repo-size    : 221 M
Repo-baseurl : https://cdn.redhat.com/content/eus/rhel/server/7/7Server/x86_64/resilientstorage/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:25 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-rs-for-rhel-7-server-htb-rpms/x86_64
Repo-name    : Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Server) HTB (RPMs)
Repo-revision: 1446656388
Repo-updated : Wed Nov  4 11:59:50 2015
Repo-pkgs    : 91
Repo-size    : 53 M
Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/resilientstorage/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:25 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-rs-for-rhel-7-server-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Server) (RPMs)
Repo-revision: 1452215214
Repo-updated : Thu Jan  7 20:07:05 2016
Repo-pkgs    : 221
Repo-size    : 221 M
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/resilientstorage/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:25 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-sap-for-rhel-7-server-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux for SAP (RHEL 7 Server) (RPMs)
Repo-revision: 1449593355
Repo-updated : Tue Dec  8 11:49:18 2015
Repo-pkgs    : 31
Repo-size    : 1.3 M
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sap/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:26 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-sap-hana-for-rhel-7-server-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux for SAP HANA (RHEL 7 Server) (RPMs)
Repo-revision: 1449576525
Repo-updated : Tue Dec  8 07:08:45 2015
Repo-pkgs    : 4
Repo-size    : 149 k
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sap-hana/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:26 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : rhel-sjis-for-rhel-7-server-rpms/7Server/x86_64
Repo-name    : Red Hat Enterprise Linux for S-JIS (RHEL 7 Server) (RPMs)
Repo-revision: 1450799461
Repo-updated : Tue Dec 22 10:51:01 2015
Repo-pkgs    : 0
Repo-size    : 0
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sjis/os
Repo-expire  : 86,400 second(s) (last: Sun Jan 17 11:39:26 2016)
Repo-filename: /etc/yum.repos.d/redhat.repo
Comment 2 wes hayutin 2016-01-19 16:09:38 UTC 
solution found.. include the dnsserver as a paramdefault

parameter_defaults:
  InternalApiNetCidr: 172.16.20.0/24
  StorageNetCidr: 172.16.21.0/24
  TenantNetCidr: 172.16.22.0/24
  ExternalNetCidr: 172.16.23.0/24
  StorageMgmtNetCidr: 172.16.19.0/24
  InternalApiAllocationPools: [{'start': '172.16.20.10', 'end': '172.16.20.100'}]
  StorageAllocationPools: [{'start': '172.16.21.10', 'end': '172.16.21.100'}]
  StorageMgmtAllocationPools: [{'start': '172.16.19.10', 'end': '172.16.19.100'}]
  TenantAllocationPools: [{'start': '172.16.22.10', 'end': '172.16.22.100'}]
  ExternalAllocationPools: [{'start': '172.16.23.110', 'end': '172.16.23.150'}]
  ExternalInterfaceDefaultRoute: 172.16.23.251
  NeutronExternalNetworkBridge: "''"
  ControlPlaneSubnetCidr: "24"
  ControlPlaneDefaultRoute: 192.0.2.1
  EC2MetadataIp: 192.0.2.1
  DnsServers: ["192.168.122.1", "10.11.5.19"]