Bug 1552541

Summary: ovn ci failure: testtools.matchers._impl.MismatchError: 'd1' != u''
Product: Red Hat OpenStack Reporter: Eran Kuris <ekuris>
Component: python-neutron-tests-tempestAssignee: Lucas Alvares Gomes <lmartins>
Status: CLOSED ERRATA QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: abregman, apevec, bcafarel, dalvarez, ekuris, lhh, majopela, nusiddiq, nyechiel, sclewis, tfreger
Target Milestone: rcKeywords: AutomationBlocker, Reopened, Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-neutron-tests-tempest-0.0.1-0.20180416200225.c515df4.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:35:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eran Kuris 2018-03-07 10:11:45 UTC
Description of problem:
1. neutron_tempest_plugin.api.test_ports.PortsTestJSON.test_create_update_port_with_dns_name

2. neutron_tempest_plugin.api.admin.test_quotas.QuotasTest.test_detail_quotas

The above tests failed with error: 
        Body: {"port": {"network_id": "2e684d6a-6eed-4289-ace1-1d5ff02ddea3", "dns_name": "d1"}}
    Response - Headers: {'status': '201', u'content-length': '849', 'content-location': 'http://10.0.0.108:9696/v2.0/ports', u'date': 'Mon, 05 Mar 2018 23:11:17 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-9aa6c2fa-e2c0-4b85-93d5-8a019877fc98'}
        Body: {"port":{"allowed_address_pairs":[],"extra_dhcp_opts":[],"updated_at":"2018-03-05T23:11:17Z","device_owner":"","revision_number":6,"port_security_enabled":true,"fixed_ips":[{"subnet_id":"01af2531-79d8-49c8-b9ff-94d95884f5ac","ip_address":"10.100.0.10"}],"id":"042f9c2f-a1b3-4417-bb02-d98b150ed521","security_groups":["354bd5d9-9a5e-436d-9360-50dfecfbdbc2"],"mac_address":"fa:16:3e:53:16:0d","device_id":"","status":"DOWN","description":"","tags":[],"dns_assignment":[{"hostname":"host-10-100-0-10","ip_address":"10.100.0.10","fqdn":"host-10-100-0-10.openstacklocal."}],"name":"","project_id":"0f5eafc70ca44ec9a495d02f3291c007","qos_policy_id":null,"admin_state_up":true,"network_id":"2e684d6a-6eed-4289-ace1-1d5ff02ddea3","dns_name":"","created_at":"2018-03-05T23:11:16Z","binding:vnic_type":"normal","tenant_id":"0f5eafc70ca44ec9a495d02f3291c007"}}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 107, in wrapper
    return func(*func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/api/test_ports.py", line 69, in test_create_update_port_with_dns_name
    self.assertEqual('d1', body['dns_name'])
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'd1' != u''


Version-Release number of selected component (if applicable):
OSP13 - puddle  2018-03-02.2 

How reproducible:
always

Steps to Reproduce:
1. run CI job 
2.
3.

Actual results:
failed

Expected results:
pass

Additional info:
logs: 
https://drive.google.com/open?id=1oFVLWVojdnunb-P-Qg3nIl4PASJyNiPr

Comment 2 Numan Siddique 2018-04-02 15:04:52 UTC
The issue is seen because the CI job is not setting the t-h-t param - "NeutronDnsDomain". Since it is not setting, the param "dns_domain" in neutron.conf is not set and the default value "openstacklocal" is used by neutron-server and because of this wheb dns_name is passed while creating the port, the "dns_name" attribute is not set by neutron-server.

In the test output we can confirm that "openstacklocal" is used.

****************************
2018-03-31 11:33:19,632 29935 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"port": {"network_id": "dbb31073-109a-4cc6-b3a4-dc8d3b8294b2", "dns_name": "d1"}}
    Response - Headers: {'status': '201', u'content-length': '845', 'content-location': 'http://10.0.0.109:9696/v2.0/ports', u'date': 'Sat, 31 Mar 2018 15:33:19 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-8c4107e4-a3eb-46ae-aa24-a5d8150c3a0d'}
        Body: {"port":{"allowed_address_pairs":[],..."dns_assignment":[{"hostname":"host-10-100-0-6","ip_address":"10.100.0.6","fqdn":"host-10-100-0-6.openstacklocal."}],..."network_id":"dbb31073-109a-4cc6-b3a4-dc8d3b8294b2","dns_name":"","created_at":"2018-03-31T15:33:17Z","binding:vnic_type":"normal","tenant_id":"c83a602d6980400b82028e4126443ccf"}}
}}}
**************

The solution is when deploying OVN, the CI job should set the t-h-t param "NeutronDnsDomain" to some value (other than "openstacklocal").

Looks like the sos report doesn't copy the folder "/var/lib/config-data/puppet-generated/neutron/" because of which I couldn't check the value of "dns_domain" in neutron.conf and verify myself.

However, on my devstack setup, I set "dns_domain=ovn.test" in neutron.conf and created a port

$openstack port create --network priv tmp1 --dns-name d1
+-----------------------+--------------------------------------------------------------------------+
| Field                 | Value                                                                    |
+-----------------------+--------------------------------------------------------------------------+
| admin_state_up        | UP                                                                       |
| allowed_address_pairs |                                                                          |
| binding_host_id       |                                                                          |
| binding_profile       |                                                                          |
| binding_vif_details   |                                                                          |
| binding_vif_type      | unbound                                                                  |
| binding_vnic_type     | normal                                                                   |
| created_at            | 2018-04-02T14:43:08Z                                                     |
| data_plane_status     | None                                                                     |
| description           |                                                                          |
| device_id             |                                                                          |
| device_owner          |                                                                          |
| dns_assignment        | fqdn='d1.ovn.test.', hostname='d1', ip_address='30.0.0.13'               |
| dns_domain            | None                                                                     |
| dns_name              | d1                                                                       |
| extra_dhcp_opts       |                                                                          |
| fixed_ips             | ip_address='30.0.0.13', subnet_id='2464a745-19b3-460b-9073-2a4dc43e2c21' |
| id                    | f7f733c9-5b22-4be0-8f45-6915743da3eb                                     |
| ip_address            | None                                                                     |
| mac_address           | fa:16:3e:e8:fb:26                                                        |
| name                  | tmp1                                                                     |
| network_id            | b88a4839-1e46-4436-a7a7-edbc0e23d1a4                                     |
| option_name           | None                                                                     |
| option_value          | None                                                                     |
| port_security_enabled | True                                                                     |
| project_id            | b306153543c0457480bd0139340baeec                                         |
| qos_policy_id         | None                                                                     |
| revision_number       | 7                                                                        |
| security_group_ids    | 94ddef58-4032-411f-b91e-5ba87cce478f                                     |
| status                | DOWN                                                                     |
| subnet_id             | None                                                                     |
| tags                  |                                                                          |
| trunk_details         | None                                                                     |
| updated_at            | 2018-04-02T14:43:08Z                                                     |
+-----------------------+--------------------------------------------------------------------------+


Below is the output when I created a port with "dns_domain" NOT set in mysetup.

$openstack port create --network priv tmp1 --dns-name d1
+-----------------------+---------------------------------------------------------------------------------------+
| Field                 | Value                                                                                 |
+-----------------------+---------------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                                    |
| allowed_address_pairs |                                                                                       |
| binding_host_id       |                                                                                       |
| binding_profile       |                                                                                       |
| binding_vif_details   |                                                                                       |
| binding_vif_type      | unbound                                                                               |
| binding_vnic_type     | normal                                                                                |
| created_at            | 2018-04-02T14:45:55Z                                                                  |
| data_plane_status     | None                                                                                  |
| description           |                                                                                       |
| device_id             |                                                                                       |
| device_owner          |                                                                                       |
| dns_assignment        | fqdn='host-30-0-0-5.openstacklocal.', hostname='host-30-0-0-5', ip_address='30.0.0.5' |
| dns_domain            | None                                                                                  |
| dns_name              |                                                                                       |
| extra_dhcp_opts       |                                                                                       |
| fixed_ips             | ip_address='30.0.0.5', subnet_id='2464a745-19b3-460b-9073-2a4dc43e2c21'               |
| id                    | 0d2ab298-2387-4ca5-ac72-e06bdc56b066                                                  |
| ip_address            | None                                                                                  |
| mac_address           | fa:16:3e:76:96:2d                                                                     |
| name                  | tmp1                                                                                  |
| network_id            | b88a4839-1e46-4436-a7a7-edbc0e23d1a4                                                  |
| option_name           | None                                                                                  |
| option_value          | None                                                                                  |
| port_security_enabled | True                                                                                  |
| project_id            | b306153543c0457480bd0139340baeec                                                      |
| qos_policy_id         | None                                                                                  |
| revision_number       | 6                                                                                     |
| security_group_ids    | 94ddef58-4032-411f-b91e-5ba87cce478f                                                  |
| status                | DOWN                                                                                  |
| subnet_id             | None                                                                                  |
| tags                  |                                                                                       |
| trunk_details         | None                                                                                  |
| updated_at            | 2018-04-02T14:45:55Z                                                                  |
+-----------------------+---------------------------------------------------------------------------------------+

Comment 19 Toni Freger 2018-04-12 05:59:56 UTC
I am reopening this bug, since we still need to decide how to resolve it.

As a first step we must skip it, as we do in neutron  - 
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-network-neutron-12_director-rhel-virthost-3cont_2comp_1net-ipv4-vxlan-composable/176/testReport/neutron.tests.tempest.api.test_ports/PortsTestJSON/

Second step we need to enable it and to add DNS server in order to test it since I agree with Numan, we will support this feature eventually.

Thanks,
Toni

Comment 24 Lucas Alvares Gomes 2018-04-19 12:42:59 UTC
I was able to reproduce the problem. As other people pointed out, configuring the dns_domain to something other than "openstacklocal" (and making sure the driver_extensions ml2 config also contains "dns") would make the tests pass.

The quota problem that was pointed out later in this BZ should already be fixed by: https://code.engineering.redhat.com/gerrit/#/c/135778.

@Eran, can you try it again with those configs/fixes in place ?

...

As to networking-ovn upstream, the DNS tests were being skipped cause DNS wasn't properly configured. I have pushed a patch upstream to enable it now: https://review.openstack.org/#/c/562527/

Comment 25 Eran Kuris 2018-04-22 07:18:22 UTC
(In reply to Lucas Alvares Gomes from comment #24)
> I was able to reproduce the problem. As other people pointed out,
> configuring the dns_domain to something other than "openstacklocal" (and
> making sure the driver_extensions ml2 config also contains "dns") would make
> the tests pass.
> 
> The quota problem that was pointed out later in this BZ should already be
> fixed by: https://code.engineering.redhat.com/gerrit/#/c/135778.
> 
> @Eran, can you try it again with those configs/fixes in place ?
> 
> ...
> 
> As to networking-ovn upstream, the DNS tests were being skipped cause DNS
> wasn't properly configured. I have pushed a patch upstream to enable it now:
> https://review.openstack.org/#/c/562527/

I will do my best to try that.
I have a backlog of tests that I need to cover.

Comment 28 errata-xmlrpc 2018-06-27 13:35:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086