Bug 2212275

Created attachment 1968972 [details]
Raw test output captured in the file paste-2137e5113543821e.txt, attached herewith.

Created attachment 1968972 [details]
Raw test output captured in the file paste-2137e5113543821e.txt, attached herewith.

Description of problem:

In Neutron config (`/var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf`) if we effectively set the `dns_domain` parameter to any other value than `openstacklocal` in the standard domain name convention (for example, openstackgate.test), and create servers (VM instances) in the isolated tenant networks (with tenant router attached), we can see the hostname to IP address records in the OVN Northbound (NB) Database (DB) serving forward DNS resolution requests. These records help the servers within that isolated tenant network ping each other using their names. In fact, the DNS querying commands like `nslookup` and `host` are able to leverage the records in the OVN NB DB to return the resulting IP address of the queried hostname. But these records do not serve as DNS PTR records that can be leveraged to serve DNS reverse resolution requests. For example, if the DNS querying commands like `nslookup` and `host` tries to get an answer for the respective hostname by providing IP address in the DNS query command, it fails.

This reverse DNS resolution feature works for an environment with ML2/OVS setup. This feature also works with ML2/OVN shipped with RHOSP 17, but does not work with RHOSP 16.2.

So, this RFE request is to backport the feature to RHOSP 16.2 (ML2/OVN).


Version-Release number of selected component (if applicable):
==========================================
From RHOSP 16.2.4 (ML2/OVN+DVR) controller node,
~~~
[root@control01 /]# rpm -qa | grep ovn
rhosp-ovn-host-2021-4.el8ost.1.noarch
ovn-2021-21.12.0-116.el8fdp.x86_64
rhosp-ovn-2021-4.el8ost.1.noarch
ovn-2021-host-21.12.0-116.el8fdp.x86_64
~~~

~~~
[root@control01 /]# rpm -qa | grep openvswitch
rhosp-network-scripts-openvswitch-2.15-4.el8ost.1.noarch
openvswitch-selinux-extra-policy-1.0-30.el8fdp.noarch
rhosp-openvswitch-2.15-4.el8ost.1.noarch
python3-rhosp-openvswitch-2.15-4.el8ost.1.noarch
network-scripts-openvswitch2.15-2.15.0-136.el8fdp.x86_64
openvswitch2.15-2.15.0-136.el8fdp.x86_64
python3-openvswitch2.15-2.15.0-136.el8fdp.x86_64
~~~

~~~
[neutron@control01 /]$ rpm -qa | grep neutron
python3-neutronclient-6.14.1-2.20220111010835.a09e824.el8ost.noarch
python3-neutron-15.3.5-2.20230216175502.c3c7146.el8ost.noarch
openstack-neutron-ml2-15.3.5-2.20230216175502.c3c7146.el8ost.noarch
openstack-neutron-common-15.3.5-2.20230216175502.c3c7146.el8ost.noarch
openstack-neutron-15.3.5-2.20230216175502.c3c7146.el8ost.noarch
python3-neutron-lib-1.29.1-2.20221201145504.4ef4b71.el8ost.noarch
python3-neutron-dynamic-routing-15.0.1-2.20220829135458.56de1c4.el8ost.noarch
~~~

From RHOSP 16.2.4 compute (ML2/OVN+DVR) node,
~~~
[root@compute00 /]# rpm -qa | grep ovn
rhosp-ovn-host-2021-4.el8ost.1.noarch
ovn-2021-21.12.0-116.el8fdp.x86_64
rhosp-ovn-2021-4.el8ost.1.noarch
ovn-2021-host-21.12.0-116.el8fdp.x86_64
~~~

~~~
[root@compute00 /]# rpm -qa | grep openvswitch
rhosp-network-scripts-openvswitch-2.15-4.el8ost.1.noarch
openvswitch-selinux-extra-policy-1.0-30.el8fdp.noarch
rhosp-openvswitch-2.15-4.el8ost.1.noarch
python3-rhosp-openvswitch-2.15-4.el8ost.1.noarch
network-scripts-openvswitch2.15-2.15.0-136.el8fdp.x86_64
openvswitch2.15-2.15.0-136.el8fdp.x86_64
python3-openvswitch2.15-2.15.0-136.el8fdp.x86_64
~~~
==========================================

How reproducible:
Always

Steps to Reproduce:
1. Create an environment file for the tripleo heat templates with the `NeutronDnsDomain` parameter set as intended and include this environment file in the overcloud deployment command/script. For example,
```
(overcloud) [stack@director ~]$ cat templates/scripts/overcloud-deploy.sh 
#!/usr/bin/bash
time openstack overcloud deploy --templates --ntp-server 192.168.100.250 --stack overcloud --timeout 120 --libvirt-type kvm \
-r /home/stack/templates/roles_data.yaml \
-n /home/stack/templates/network_data.yaml \
-e /home/stack/templates/00-node-info.yaml \
-e /home/stack/templates/01-containers-prepare-parameter.yaml \
-e /home/stack/templates/02-environments/02-01-network-isolation.yaml \
-e /home/stack/templates/02-environments/02-02-network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/templates/03-ceph-config.yaml \
-e /home/stack/templates/04-NeutronDNSDomain.yaml ### Appending this line

(overcloud) [stack@director ~]$ cat templates/04-NeutronDNSDomain.yaml 
parameter_defaults:
  NeutronDnsDomain: openstackgate.test
(overcloud) [stack@director ~]$ 
```

2. Invoke the overcloud deployment command/script for the stack update to occur so that the intended DNS domain is configured appropriately across the overcloud environment.

If you wish to avoid running the overcloud deployment command/script you may manually edit the Neutron configuration file and restart the tripleo_neutron_api.service systemd unit in both the controller and the compute nodes. Please be aware that the manually edited Neutron configuration, as shown, below will be overridden whenever the overcloud deployment command/script runs next time. This is also not the recommended method. The best way to set this configuration option is to use the tripleo heat templates and rerunning the overcloud deployment command/script as stated above.

a. Edit the Neutron config to set the value of `dns_domain` as intended in all the controller and compute nodes.
~~~
[root@overcloud-controller-0 ~]# grep dns /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf 
dns_domain=openstackgate.test
[root@overcloud-novacompute-0 ~]# grep dns /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf
dns_domain=openstackgate.test
~~~

b. Restart the Neutron server in all the controller and compute nodes.
~~~
[root@overcloud-controller-0 ~]# systemctl restart tripleo_neutron_api.service
[root@overcloud-novacompute-0 ~]# systemctl restart tripleo_neutron_api.service
~~~


3. Create a Neutron network and a subnet. 
~~~
(overcloud) [stack@director ~]$ openstack network create test-admin-net1 && openstack subnet create --subnet-range 192.168.11.0/24 --dhcp --network test-admin-net1 test-admin-subnet1
~~~

4. Create a router and add the subnet to the router.
~~~
(overcloud) [stack@director ~]$ openstack router create admin-r1
(overcloud) [stack@director ~]$ openstack router add subnet admin-r1 test-admin-subnet1 
~~~

5. Create two servers with the NICs attached from the above network.
~~~
(overcloud) [stack@director ~]$ openstack server create --image rhel-8-4 --flavor Disk_10_VCPUS_1_RAM_1024 --network test-admin-net1 --user-data ~/workload/scripts/userdata.sh admin-server12 --wait -f json
(overcloud) [stack@director ~]$ openstack server create --image rhel-8-4 --flavor Disk_10_VCPUS_1_RAM_1024 --network test-admin-net1 --user-data ~/workload/scripts/userdata.sh admin-server13 --wait -f json 
~~~

6. Log in to one of the server and try pinging the other server with its name. For example, from `admin-server12`, try running `ping admin-server13`. This command should work.

7. From the same server (`admin-server12`), run the commands like `nslookup <ip_address>` or `host <ip_address>`. These commands fail to return the respective hostname of the ip address. Replace <ip_address> with the IP address of the other server.


Actual results:
The DNS forward resolution is working but the DNS reverse resolution is failing from the servers (VM instances).

Expected results:
The DNS reverse resolution should work along with DNS forward resolution from the servers.

Additional info:
Raw test output captured in the file paste-2137e5113543821e.txt, attached herewith.