Hide Forgot
Description of problem: overcloud nodes loosing their dns configuration in /etc/resolv.conf minutes after deployment starts Recreate: 1. Set up the undercloud 2. Ensure the the overcloud is configured to receive dns configuration via dhcp https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html-single/Director_Installation_and_Usage/index.html#sect-Setting_a_Nameserver_for_the_Overcloud 3. Deploy the cloud 4. wait for the nodes to spawn/boot 5. immediately ssh to a overcloud node 6. watch the /etc/resolv.conf Note that the /etc/resolv.conf is initially configured correctly and name resolution works wait a few minutes as the node continues to be configured. Results: The overcloud node looses it's nameserver config after a few minutes: [stack@instack ~]$ nova list +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+ | 00d7a348-16e4-40e6-abaf-3f696b5423b3 | overcloud-cephstorage-0 | ACTIVE | - | Running | ctlplane=192.0.2.8 | | feba8f3b-9dfc-4b8a-bf4b-f05ba42a9fcd | overcloud-compute-0 | BUILD | spawning | NOSTATE | ctlplane=192.0.2.9 | | 362baa85-1784-4333-a3e6-9ebb2f895ba3 | overcloud-controller-0 | BUILD | spawning | NOSTATE | ctlplane=192.0.2.10 | +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+ [stack@instack ~]$ ssh heat-admin.2.8 ssh: connect to host 192.0.2.8 port 22: Connection refused [stack@instack ~]$ ssh heat-admin.2.8 ssh: connect to host 192.0.2.8 port 22: Connection refused [stack@instack ~]$ ssh heat-admin.2.8 The authenticity of host '192.0.2.8 (192.0.2.8)' can't be established. ECDSA key fingerprint is a6:d9:20:00:9f:49:e9:21:95:0a:5b:79:b4:ee:0a:ba. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.0.2.8' (ECDSA) to the list of known hosts. [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain nameserver 192.168.122.1 [heat-admin@overcloud-cephstorage-0 ~]$ cat /etc/resolv.conf # Generated by NetworkManager search localdomain # No nameservers found; try putting DNS servers into your # ifcfg files in /etc/sysconfig/network-scripts like so: # # DNS1=xxx.xxx.xxx.xxx # DNS2=xxx.xxx.xxx.xxx # DOMAIN=lab.foo.com bar.foo.com
Note: RHSM enabled, cdn repos in play here: Setting up Package Sacks pkgsack time: 0.091 Repo-id : rhel-7-server-aus-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux 7 Server - AUS (RPMs) Repo-revision: 1452806283 Repo-updated : Thu Jan 14 16:20:09 2016 Repo-pkgs : 10,110 Repo-size : 11 G Repo-baseurl : https://cdn.redhat.com/content/aus/rhel/server/7/7Server/x86_64/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:19 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-eus-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux 7 Server - Extended Update Support (RPMs) Repo-revision: 1452806256 Repo-updated : Thu Jan 14 16:19:30 2016 Repo-pkgs : 10,110 Repo-size : 11 G Repo-baseurl : https://cdn.redhat.com/content/eus/rhel/server/7/7Server/x86_64/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:20 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-extras-rpms/x86_64 Repo-name : Red Hat Enterprise Linux 7 Server - Extras (RPMs) Repo-revision: 1449807908 Repo-updated : Thu Dec 10 23:25:09 2015 Repo-pkgs : 143 Repo-size : 290 M Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/extras/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:20 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-htb-rpms/x86_64 Repo-name : Red Hat Enterprise Linux 7 Server HTB (RPMs) Repo-revision: 1447273278 Repo-updated : Wed Nov 11 15:21:18 2015 Repo-pkgs : 6,770 Repo-size : 5.5 G Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:20 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-openstack-7.0-director-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux OpenStack Platform 7.0 director for RHEL 7 (RPMs) Repo-revision: 1450716322 Repo-updated : Mon Dec 21 11:45:22 2015 Repo-pkgs : 70 Repo-size : 11 M Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/openstack-director/7.0/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:21 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-openstack-7.0-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux OpenStack Platform 7.0 for RHEL 7 (RPMs) Repo-revision: 1452467561 Repo-updated : Sun Jan 10 18:12:41 2016 Repo-pkgs : 923 Repo-size : 387 M Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/openstack/7.0/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:21 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-optional-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux 7 Server - Optional (RPMs) Repo-revision: 1452806854 Repo-updated : Thu Jan 14 16:28:48 2016 Repo-pkgs : 8,181 Repo-size : 9.3 G Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/optional/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:21 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux 7 Server (RPMs) Repo-revision: 1452806059 Repo-updated : Thu Jan 14 16:16:31 2016 Repo-pkgs : 10,110 Repo-size : 11 G Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:22 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-rt-beta-rpms/x86_64 Repo-name : Red Hat Enterprise Linux for Real Time Beta (RHEL 7 Server) (RPMs) Repo-revision: 1441144237 Repo-updated : Tue Sep 1 17:50:37 2015 Repo-pkgs : 15 Repo-size : 199 M Repo-baseurl : https://cdn.redhat.com/content/beta/rhel/server/7/x86_64/rt/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:22 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-rt-htb-rpms/x86_64 Repo-name : Red Hat Enterprise Linux for Real Time HTB (RHEL 7 Server) (RPMs) Repo-revision: 1447153130 Repo-updated : Tue Nov 10 05:59:03 2015 Repo-pkgs : 53 Repo-size : 835 M Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/rt/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:22 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-7-server-rt-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux for Real Time (RHEL 7 Server) (RPMs) Repo-revision: 1449084995 Repo-updated : Wed Dec 2 14:36:35 2015 Repo-pkgs : 70 Repo-size : 1.1 G Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rt/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:23 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-ha-for-rhel-7-server-eus-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux High Availability (for RHEL 7 Server) - Extended Update Support (RPMs) Repo-revision: 1449686227 Repo-updated : Wed Dec 9 13:37:10 2015 Repo-pkgs : 190 Repo-size : 214 M Repo-baseurl : https://cdn.redhat.com/content/eus/rhel/server/7/7Server/x86_64/highavailability/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:23 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-ha-for-rhel-7-server-htb-rpms/x86_64 Repo-name : Red Hat Enterprise Linux High Availability (for RHEL 7 Server) HTB (RPMs) Repo-revision: 1446656084 Repo-updated : Wed Nov 4 11:54:46 2015 Repo-pkgs : 76 Repo-size : 48 M Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/highavailability/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:23 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-ha-for-rhel-7-server-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs) Repo-revision: 1449686149 Repo-updated : Wed Dec 9 13:35:54 2015 Repo-pkgs : 190 Repo-size : 214 M Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/highavailability/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:24 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-lb-for-rhel-7-server-htb-rpms/x86_64 Repo-name : Red Hat Enterprise Linux Load Balancer (for RHEL 7 Server) HTB (RPMs) Repo-revision: 1403809660 Repo-updated : Thu Jun 26 15:07:41 2014 Repo-pkgs : 0 Repo-size : 0 Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/loadbalancer/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:24 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-rs-for-rhel-7-server-eus-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Server) - Extended Update Support (RPMs) Repo-revision: 1452215589 Repo-updated : Thu Jan 7 20:13:11 2016 Repo-pkgs : 221 Repo-size : 221 M Repo-baseurl : https://cdn.redhat.com/content/eus/rhel/server/7/7Server/x86_64/resilientstorage/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:25 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-rs-for-rhel-7-server-htb-rpms/x86_64 Repo-name : Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Server) HTB (RPMs) Repo-revision: 1446656388 Repo-updated : Wed Nov 4 11:59:50 2015 Repo-pkgs : 91 Repo-size : 53 M Repo-baseurl : https://cdn.redhat.com/content/htb/rhel/server/7/x86_64/resilientstorage/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:25 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-rs-for-rhel-7-server-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Server) (RPMs) Repo-revision: 1452215214 Repo-updated : Thu Jan 7 20:07:05 2016 Repo-pkgs : 221 Repo-size : 221 M Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/resilientstorage/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:25 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-sap-for-rhel-7-server-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux for SAP (RHEL 7 Server) (RPMs) Repo-revision: 1449593355 Repo-updated : Tue Dec 8 11:49:18 2015 Repo-pkgs : 31 Repo-size : 1.3 M Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sap/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:26 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-sap-hana-for-rhel-7-server-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux for SAP HANA (RHEL 7 Server) (RPMs) Repo-revision: 1449576525 Repo-updated : Tue Dec 8 07:08:45 2015 Repo-pkgs : 4 Repo-size : 149 k Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sap-hana/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:26 2016) Repo-filename: /etc/yum.repos.d/redhat.repo Repo-id : rhel-sjis-for-rhel-7-server-rpms/7Server/x86_64 Repo-name : Red Hat Enterprise Linux for S-JIS (RHEL 7 Server) (RPMs) Repo-revision: 1450799461 Repo-updated : Tue Dec 22 10:51:01 2015 Repo-pkgs : 0 Repo-size : 0 Repo-baseurl : https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sjis/os Repo-expire : 86,400 second(s) (last: Sun Jan 17 11:39:26 2016) Repo-filename: /etc/yum.repos.d/redhat.repo
solution found.. include the dnsserver as a paramdefault parameter_defaults: InternalApiNetCidr: 172.16.20.0/24 StorageNetCidr: 172.16.21.0/24 TenantNetCidr: 172.16.22.0/24 ExternalNetCidr: 172.16.23.0/24 StorageMgmtNetCidr: 172.16.19.0/24 InternalApiAllocationPools: [{'start': '172.16.20.10', 'end': '172.16.20.100'}] StorageAllocationPools: [{'start': '172.16.21.10', 'end': '172.16.21.100'}] StorageMgmtAllocationPools: [{'start': '172.16.19.10', 'end': '172.16.19.100'}] TenantAllocationPools: [{'start': '172.16.22.10', 'end': '172.16.22.100'}] ExternalAllocationPools: [{'start': '172.16.23.110', 'end': '172.16.23.150'}] ExternalInterfaceDefaultRoute: 172.16.23.251 NeutronExternalNetworkBridge: "''" ControlPlaneSubnetCidr: "24" ControlPlaneDefaultRoute: 192.0.2.1 EC2MetadataIp: 192.0.2.1 DnsServers: ["192.168.122.1", "10.11.5.19"]