Bug 1956801

Summary: 16.2 tempest test: neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest failed with MismatchError: 'openstackgate.local' not in 'nameserver 10.5.30.160
Product: Red Hat OpenStack Reporter: Sandeep Yadav <sandyada>
Component: openstack-neutronAssignee: Candido Campos <ccamposr>
Status: CLOSED NOTABUG QA Contact: Eran Kuris <ekuris>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16.2 (Train)CC: amuller, chrisw, ralonsoh, scohen
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-17 14:00:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sandeep Yadav 2021-05-04 12:42:13 UTC 
16.2 Integration pipeline, tempest test: neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest is failing in CI.

Job name:  periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2


Logs:
https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2/96d184b/logs/undercloud/var/log/tempest/stestr_results.html

~~~
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 108, in wrapper
    return func(*func_args, **func_kwargs)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_internal_dns.py", line 76, in test_dns_domain_and_name
    self.assertIn('openstackgate.local', resolv_conf)
  File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 417, in assertIn
    self.assertThat(haystack, Contains(needle), message)
  File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'openstackgate.local' not in 'nameserver 10.5.30.160\n'
~~~

Following error observed in neutronserver logs:-

https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2/96d184b/logs/undercloud/var/log/containers/neutron/server.log.1
~~~
2021-05-03 01:47:23.285 18 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to tcp:192.168.24.1:6641 to retrieve schema: Connection refused
.
2021-05-03 01:47:23.364 16 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to tcp:192.168.24.1:6641 to retrieve schema: Connection refused
.
.
2021-05-03 01:53:56.212 15 ERROR ovsdbapp.backend.ovs_idl.transaction [req-e8bb48e2-6490-4680-b551-d235ad5196e4 a75c96127fe9433390969eeefceebd71 b0d6aefc397d4a1e801c23b6df857468 - default default] Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/commands.py", line 937, in run_idl
    resource_id=self.name, resource_type=self.resource_type)
networking_ovn.common.exceptions.RevisionConflict: OVN revision number for a1d1fb0c-4044-41b8-bd00-142f811d9c9a (type: ports) is equal or higher than the given resource. Skipping update

~~~

Another example:
https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2/06615e7/logs/undercloud/var/log/tempest/stestr_results.html


Version-Release number of selected component (if applicable):
16.2

How reproducible:
Everytime in Integration pipeline

https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2
Comment 1 Sandeep Yadav 2021-05-05 10:04:09 UTC 
The failed test was skipped earlier, failed test is also skipped in upstream for all the branches. 

https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L71-L97

Ex: neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest was not running earlier

Now:
https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2/96d184b/logs/undercloud/var/log/tempest/stestr_results.html

Earlier:
https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2/dda768e/logs/undercloud/var/log/tempest/stestr_results.html


This is caused by tripleo-quickstart-extras patch caused some tests which were earlier skipped to run, we are reverting that patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/789450 

In skiplist https://bugs.launchpad.net/tripleo/+bug/1832166 is mentioned as reason to add these test in skiplist, but it seems this bug was never fully solved, Would be great if we can investigate the failure so that we can unskip these tests in downstream and upstream if they add value.