Description of problem:

Undercloud deployment failed [1] after customer defined net_config_override in undercloud.conf according to our official recommendations [2] with single IP address specified for dns_servers variable. We isolated this problem to /etc/resolv.conf and /etc/sysconfig/network-scripts/ifcfg-br-ctlplane:

[root@director ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search REMOVED
nameserver 1 # Original value is provided, it wasn't tuned to hide customer's details
nameserver 0 # Same as above

/etc/sysconfig/network-scripts/ifcfg-br-ctlplane
# This file is autogenerated by os-net-config
DEVICE=br-ctlplane
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=REMOVED
NETMASK=255.255.255.0
IPADDR1=REMOVED
NETMASK1=255.255.255.0
IPADDR2=REMOVED
NETMASK2=255.255.255.0
OVS_EXTRA="set bridge br-ctlplane other-config:mac-table-size=50000 -- br-set-external-id br-ctlplane bridge-id br-ctlplane -- set bridge br-ctlplane fail_mode=standalone -- del-controller br-ctlplane -- set bridge br-ctlplane fail_mode=standalone -- del-controller br-ctlplane"
DNS1=1
DNS2=0
DOMAIN=REMOVED

When os-net-config was executed manually, the following output was returned (I have replaced IP address for dns_servers, but preserved formatting):

2023-06-27 11:42:08.801 INFO os_net_config.main Using config file at: /etc/os-net-config/config.yaml
2023-06-27 11:42:08.801 INFO os_net_config.impl_ifcfg.__init__ Ifcfg net config provider created.
2023-06-27 11:42:08.802 INFO os_net_config.main Not using any mapping file.
2023-06-27 11:42:09.279 WARNING os_net_config.main Config file failed schema validation at network_config/0:
    {'addresses': [{'ip_netmask': 'REMOVED'}, {'ip_netmask': 'REMOVED'}, {'ip_netmask': 'REMOVED'}], 'dns_servers': '192.168.1.1', 'domain': 'REMOVED', 'members': [{'name': 'ens224', 'type': 'interface'}], 'name': 'br-ctlplane', 'ovs_extra': ['br-set-external-id br-ctlplane bridge-id br-ctlplane'], 'type': 'ovs_bridge', 'use_dhcp': False, 'nic_mapping': None, 'persist_mapping': False} is not valid under any of the given schemas
  Sub-schemas tested and not matching:
  - items/oneOf/ovs_bridge/dns_servers/oneOf: '192.168.1.1' is not valid under any of the given schemas
  -- items/oneOf/ovs_bridge/dns_servers/oneOf/0/type: '192.168.1.1' is not of type 'array'
  -- items/oneOf/ovs_bridge/dns_servers/oneOf/param/oneOf: '192.168.1.1' is not valid under any of the given schemas
  --- items/oneOf/ovs_bridge/dns_servers/oneOf/param/oneOf/0/type: '192.168.1.1' is not of type 'object'
  --- items/oneOf/ovs_bridge/dns_servers/oneOf/param/oneOf/1/type: '192.168.1.1' is not of type 'object'
2023-06-27 11:42:09.280 INFO os_net_config.utils._ordered_nics Finding active nics
2023-06-27 11:42:09.281 INFO os_net_config.utils._ordered_nics lo is not an active nic
2023-06-27 11:42:09.282 INFO os_net_config.utils._ordered_nics ens224 is an active nic
2023-06-27 11:42:09.282 INFO os_net_config.utils._ordered_nics ovs-system is not an active nic
2023-06-27 11:42:09.283 INFO os_net_config.utils._ordered_nics ens192 is an active nic
2023-06-27 11:42:09.284 INFO os_net_config.utils._ordered_nics br-ctlplane is not an active nic
2023-06-27 11:42:09.284 INFO os_net_config.utils._ordered_nics No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)
2023-06-27 11:42:09.284 INFO os_net_config.utils._ordered_nics Active nics are ['ens192', 'ens224']
2023-06-27 11:42:09.285 INFO os_net_config.objects.mapped_nics nic2 mapped to: ens224
2023-06-27 11:42:09.285 INFO os_net_config.objects.mapped_nics nic1 mapped to: ens192
2023-06-27 11:42:09.286 INFO os_net_config.impl_ifcfg.add_bridge adding bridge: br-ctlplane
2023-06-27 11:42:09.287 WARNING os_net_config.impl_ifcfg._add_common ifcfg format supports max 2 resolvers.
2023-06-27 11:42:09.288 INFO os_net_config.impl_ifcfg.add_interface adding interface: ens224
2023-06-27 11:42:09.288 INFO os_net_config.impl_ifcfg.apply applying network configs...
2023-06-27 11:42:09.289 INFO os_net_config.impl_ifcfg.apply No changes required for interface: ens224
2023-06-27 11:42:09.290 INFO os_net_config.execute running ifdown on interface: ens224
2023-06-27 11:42:09.867 INFO os_net_config.execute running ifdown on bridge: br-ctlplane
2023-06-27 11:42:10.456 INFO os_net_config.write_config Writing config /etc/sysconfig/network-scripts/ifcfg-br-ctlplane
2023-06-27 11:42:10.458 INFO os_net_config.execute running ifup on bridge: br-ctlplane
2023-06-27 11:42:20.470 INFO os_net_config.execute running ifup on interface: ens224


Workaround: problem was solved after switching to list value for dns_servers:
  dns_servers: [192.168.1.1]


[1]
2023-06-27 11:27:00.142365 | 00505684-f5f6-0ad3-efe2-0000000008d8 |       TASK | Ensure firewall is installed
2023-06-27 11:28:43.733569 | 00505684-f5f6-0ad3-efe2-0000000008d8 |      FATAL | Ensure firewall is installed | director17 | error={"changed": false, "msg": "Failed to download metadata for repo 'rhel-9-for-x86_64-highavailability-eus-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []}

[2]
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.0/html-single/director_installation_and_usage/index#proc_configuring-undercloud-network-interfaces_installing-director-on-the-undercloud
Version-Release number of selected component (if applicable):


How reproducible:
Define net_config_override according to https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.0/html-single/director_installation_and_usage/index#proc_configuring-undercloud-network-interfaces_installing-director-on-the-undercloud
and run "openstack undercloud install" command


Actual results:
/etc/resolv.conf is broken because os-net-config failed to validate DNS server's IP address

Expected results:
os-net-config is able to provision configured IP address