Bug 1954873 - dns server may not be specified for clusters with more than 2 dns servers specified by openstack.
Summary: dns server may not be specified for clusters with more than 2 dns servers spe...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.8.0
Assignee: Adolfo Duarte
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-28 23:56 UTC by Adolfo Duarte
Modified: 2021-07-27 23:05 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:04:50 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4939 0 None open Bug 1954873: CoreOS only supports three dns namservers 2021-05-17 20:50:19 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:05:04 UTC

Description Adolfo Duarte 2021-04-28 23:56:20 UTC
On an openstack deployment that is providing two or more dns servers per subnet, it is possible that a dns server specified by the 
"externalDNS" setting in install-config.yaml may not work correctly. 
For example, on a openstack infrastructure that is configured to provide three dns servers per subnet (.10, .11, and .12 for example), the following install-config.yaml entry: 

platform:
  openstack:
    cloud: openstack
    computeFlavor: aci_rhel_medium
    externalDNS: ["172.28.184.18"]

Will produce the following /etc/resolv.conf on the cluster nodes: 

# Generated by NetworkManager
search openstacklocal
nameserver 127.0.0.1
nameserver 15.11.0.12
nameserver 15.11.0.11
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
nameserver 15.11.0.10
nameserver 172.28.184.18

#See the troubleshooting documentation (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md) for ideas about what information to collect.

#For example, 

# If the installer fails to create resources (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#installer-fails-to-create-resources), attach the relevant portions of your `.openshift_install.log.`
# If the installer fails to bootstrap the cluster (https://github.com/openshift/installer/blob/master/docs/user/troubleshootingbootstrap.md), attach the bootstrap log bundle.
# If the installer fails to complete installation after bootstrapping completes (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#installer-fails-to-initialize-the-cluster), attach the must-gather log bundle using `oc adm must-gather`

# Always at least include the `.openshift_install.log`

What did you expect to happen?

#Enter text here.

How to reproduce it (as minimally and precisely as possible)?

$ your-commands-here

Anything else we need to know?

#Enter text here.

Comment 1 Adolfo Duarte 2021-04-29 00:04:36 UTC
There is a flag in neutron which configures the number of dns servers to provide per network. 


dhcp_agents_per_network=1

needs to be added to the [DEFAULT] section in our neutron.conf file

Comment 2 Adolfo Duarte 2021-04-30 06:53:13 UTC
Tested and reproduce this on 4.5

To reproduce create an install-config.yaml with three or more dns servers: 
platform:
  openstack:
    apiVIP: 10.0.0.5
    cloud: openshift
    computeFlavor: m1.xlarge
    externalDNS: 
     - 10.10.160.2
     - 10.5.30.160
     - 10.11.5.19
    externalNetwork: external

The resulting boostrap node will have the following on its /etc/resolv.conf

cat /etc/resolv.conf

# Generated by NetworkManager
search shiftstack
nameserver 127.0.0.1
nameserver 10.10.160.2
nameserver 10.5.30.160
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
nameserver 10.11.5.19

Comment 3 Adolfo Duarte 2021-04-30 06:55:05 UTC
Tested on 4.8 with the following: 

platform:
  openstack:
    apiFloatingIP: 192.168.25.35
    apiVIP: 10.0.0.5
    cloud: openshift
    defaultMachinePlatform:
      type: m1.xlarge
    externalDNS: 
     - 10.10.160.2
     - 10.5.30.160
     - 10.11.5.19
    externalNetwork: external

The resulting nodes did not have a warning: 

[core@oc48-gw68h-bootstrap ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 127.0.0.1
search shiftstack
nameserver 10.10.160.2
nameserver 10.5.30.160
nameserver 10.11.5.19


which suggest the problem does not exists in 4.8 machines.

Comment 4 Adolfo Duarte 2021-04-30 07:06:08 UTC
The root cause of the problem is that the version of libc for 4.5 does not support more than three dns nameservers in /etc/resolv.conf 
So if namesevers need to be specified in the install-config.yaml, care must be taken to not provide more than three dns namesevers to the cluster nodes attached to the openstack tenant network. 

There seems to be a couple of workarounds for this problem: 
One: set up openstack so only one dns server is provided by OSP on its subnets: commet 1 above

Two: precreate the subnet and set the "--no-dns-namesever" flag on it:  "openstack subnet set --no-dns-nameserver ...." 

     Then use the machinesSubnet configuration parameter to deploy, as described here [1] and here[2]




[1] https://docs.openshift.com/container-platform/4.5/installing/installing_openstack/installing-openstack-installer-custom.html#installation-configuration-parameters_installing-openstack-installer-custom

[2] https://github.com/openshift/installer/blob/master/docs/user/openstack/customization.md

Comment 9 Jon Uriarte 2021-06-14 08:47:00 UTC
The doc PR has been merged in the upstream installer repository.

Comment 12 errata-xmlrpc 2021-07-27 23:04:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.