Description of problem:

When deploying OSP13 with TLS everywhere, OVN metadata fails to serve metadata to instances.

Version-Release number of selected component (if applicable):

13

How reproducible:

Always

Steps to Reproduce:
1. Deploy OSP13 with TLS Everywhere and OVN-DVR
2. Spawn instances
3. Instances get network config but can't reach metadata service.

Actual results:

Inside the guest (cloud-init.log)
2018-09-24 12:02:44,916 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/2009-04-04/meta-data/instance-id' with {'url': 'http://169.254.169.254/2009-04-04/meta-data/instance-id', 'headers': {'User-Agent': 'Cloud-Init/0.7.9'}, 'a
llow_redirects': True, 'method': 'GET', 'timeout': 50.0} configuration
2018-09-24 12:02:44,930 - url_helper.py[DEBUG]: Read from http://169.254.169.254/2009-04-04/meta-data/instance-id (500, 207b) after 1 attempts
2018-09-24 12:02:44,931 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: bad status code [500]
2018-09-24 12:02:44,931 - url_helper.py[DEBUG]: Please wait 1 seconds while we wait to try again


ovn-metadata-agent.log (on the compute)
2018-09-24 12:02:45.498 24 INFO eventlet.wsgi.server [-] 192.168.1.22,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500  len: 362 time: 0.0078270
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server [-] Unexpected error.: BadStatusLine: ''
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server Traceback (most recent call last):
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/server.py", line 68, in __call__
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server     return self._proxy_request(instance_id, project_id, req)
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/server.py", line 119, in _proxy_request
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server     body=req.body)
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1621, in request
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1363, in _request
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server     (response, content) = self._conn_request(conn, request_uri, method, body, headers)
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1319, in _conn_request
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server     response = conn.getresponse()
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server   File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server     response.begin()
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server   File "/usr/lib64/python2.7/httplib.py", line 444, in begin
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server     version, status, reason = self._read_status()
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server   File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server     raise BadStatusLine(line)
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server BadStatusLine: ''
2018-09-24 12:02:46.511 24 ERROR networking_ovn.agent.metadata.server

Expected results:

Instances successfully reach the metadata service and configure itself (hostname, SSH key, etc)

Additional info:

OVN metadata agent configuration does not include TLS parameters therefore it cannot reach nova internal API VIP.

The following fixes the issue:

Open /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini

Replace
ova_metadata_ip=172.17.1.150 # internal API VIP
By 
nova_metadata_host=overcloud.internalapi.redhat.local # Internal API VIP FQDN

As well as adding the following:
nova_metadata_protocol = https
auth_ca_cert=/etc/pki/tls/cert.pem

Finally restart service: 
docker restart ovn_metadata_agent

We can see these options are not included in the core THT:
https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/puppet/services/ovn-metadata.yaml#L84

As a side note, this case is covered for ML2-OVS (not tested):
https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/puppet/services/neutron-metadata.yaml#L116