Bug 1560741

Summary: [Deployment][TLS] Neutron fails to read certificates in container due to UID mistmatch between host and container
Product: Red Hat OpenStack Reporter: Tim Rozet <trozet>
Component: openstack-tripleo-heat-templatesAssignee: Tim Rozet <trozet>
Status: CLOSED ERRATA QA Contact: Itzik Brown <itbrown>
Severity: high Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: aadam, itbrown, jschluet, mburns, mkolesni, rhel-osp-director-maint, sclewis
Target Milestone: betaKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: odl_deployment, odl_tls
Fixed In Version: puppet-tripleo-8.3.2-0.20180327181746, openstack-tripleo-heat-templates-8.0.2-8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
N/A
Last Closed: 2018-06-27 13:48:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1488826    

Description Tim Rozet 2018-03-26 21:51:02 UTC
Description of problem:
When deploying with TLS and OpenDaylight, neutron dhcp agent is configured with TLS certificate/key in order to be able to communicate with OVSDB (listening in passive ssl).  However, neutron dhcp agent fails to add the dhcp tap port to OVSDB because it cannot read the key/certificate.  The reason for this bug is because the key and certificate are generated on the host with the uid of neutron on the host.  They are then mounted into the container.  However, the UID of neutron in the container is not the same as the UID of the host.  The neutron packaging distgit spec does not specify a unique UID.


Version-Release number of selected component (if applicable):


How reproducible:
Everytime

Steps to Reproduce:
1.  Deploy with  TLS + ODL
2.  After deployment, check neutron dhcp agent log:
2018-03-26 12:10:36.150 77314 ERROR neutron.agent.linux.dhcp [req-4f78a1e5-d50e-4d93-a0de-e62897b3d8e5 - - - - -] Unable to plug DHCP port for network aa811fc4-5602-4a69-bf1e-abf754b1c251. Releasing port.: Error: [('system library', 'fopen', 'Permission denied'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'system lib')]


Actual results:
No tap port is created in OVS on the control nodes, nova instances fail to spawn in dhcp enabled tenant networks.

Expected results:
Tap interface should be created in ip netns and added into OVS.  Nova instances should spawn correctly.

Additional info:

Comment 1 Tim Rozet 2018-03-26 21:56:17 UTC
Nova distgit does specify a UID:

https://review.rdoproject.org/r/gitweb?p=openstack/nova-distgit.git;a=blob;f=openstack-nova.spec;h=319f92e99e534f41a6a0a609894ffab0c3e4ceaf;hb=refs/heads/queens-rdo#l632

Neutron does not:
https://review.rdoproject.org/r/gitweb?p=openstack/neutron-distgit.git;a=blob;f=openstack-neutron.spec;h=94b009ef6ca5caad050154216ba2525c7e5463ca;hb=refs/heads/queens-rdo#l488

The obvious solution here would be to make Neutron UID static.  However after talking to with some folks it looks like this path is more complicated.  Ideally we should have a common approach of either specifying or not specifying the UID for every service in RDO.  However, an easier fix for this is to implement a step to mount the cert/key as RW and then chmod the files with the neutron UID of the container.

Comment 7 Tim Rozet 2018-04-26 18:48:19 UTC
*** Bug 1572236 has been marked as a duplicate of this bug. ***

Comment 8 Tim Rozet 2018-04-26 18:50:35 UTC
The fix did not work:

https://review.openstack.org/#/c/558667/1/docker/services/neutron-dhcp.yaml

ERROR:__main__:Failed to change ownership of /etc/pki/tls/certs/neutron.crt to 42435:42435
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 345, in set_perms
    os.chown(path, uid, gid)
OSError: [Errno 30] Read-only file system: '/etc/pki/tls/certs/neutron.crt'
INFO:__main__:Setting permission for /etc/pki/tls/private/neutron.key
ERROR:__main__:Failed to change ownership of /etc/pki/tls/private/neutron.key to 42435:42435
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 345, in set_perms
    os.chown(path, uid, gid)
OSError: [Errno 30] Read-only file system: '/etc/pki/tls/private/neutron.key'

Comment 11 Itzik Brown 2018-05-03 08:19:37 UTC
Checked with:
puppet-tripleo-8.3.2-4.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-9.el7ost.noarch

Comment 13 errata-xmlrpc 2018-06-27 13:48:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086