Bug 1560741 - [Deployment][TLS] Neutron fails to read certificates in container due to UID mistmatch between host and container
Summary: [Deployment][TLS] Neutron fails to read certificates in container due to UID ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: beta
: 13.0 (Queens)
Assignee: Tim Rozet
QA Contact: Itzik Brown
URL:
Whiteboard: odl_deployment, odl_tls
: 1572236 (view as bug list)
Depends On:
Blocks: 1488826
TreeView+ depends on / blocked
 
Reported: 2018-03-26 21:51 UTC by Tim Rozet
Modified: 2018-10-18 07:24 UTC (History)
7 users (show)

Fixed In Version: puppet-tripleo-8.3.2-0.20180327181746, openstack-tripleo-heat-templates-8.0.2-8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
N/A
Last Closed: 2018-06-27 13:48:49 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1759049 None None None 2018-03-26 21:59:39 UTC
OpenStack gerrit 556673 None None None 2018-03-26 22:34:03 UTC
OpenStack gerrit 558664 None None None 2018-04-04 20:16:42 UTC
OpenStack gerrit 558667 None None None 2018-04-04 20:17:36 UTC
OpenStack gerrit 565011 None None None 2018-04-30 16:51:14 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:49:42 UTC

Description Tim Rozet 2018-03-26 21:51:02 UTC
Description of problem:
When deploying with TLS and OpenDaylight, neutron dhcp agent is configured with TLS certificate/key in order to be able to communicate with OVSDB (listening in passive ssl).  However, neutron dhcp agent fails to add the dhcp tap port to OVSDB because it cannot read the key/certificate.  The reason for this bug is because the key and certificate are generated on the host with the uid of neutron on the host.  They are then mounted into the container.  However, the UID of neutron in the container is not the same as the UID of the host.  The neutron packaging distgit spec does not specify a unique UID.


Version-Release number of selected component (if applicable):


How reproducible:
Everytime

Steps to Reproduce:
1.  Deploy with  TLS + ODL
2.  After deployment, check neutron dhcp agent log:
2018-03-26 12:10:36.150 77314 ERROR neutron.agent.linux.dhcp [req-4f78a1e5-d50e-4d93-a0de-e62897b3d8e5 - - - - -] Unable to plug DHCP port for network aa811fc4-5602-4a69-bf1e-abf754b1c251. Releasing port.: Error: [('system library', 'fopen', 'Permission denied'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'system lib')]


Actual results:
No tap port is created in OVS on the control nodes, nova instances fail to spawn in dhcp enabled tenant networks.

Expected results:
Tap interface should be created in ip netns and added into OVS.  Nova instances should spawn correctly.

Additional info:

Comment 1 Tim Rozet 2018-03-26 21:56:17 UTC
Nova distgit does specify a UID:

https://review.rdoproject.org/r/gitweb?p=openstack/nova-distgit.git;a=blob;f=openstack-nova.spec;h=319f92e99e534f41a6a0a609894ffab0c3e4ceaf;hb=refs/heads/queens-rdo#l632

Neutron does not:
https://review.rdoproject.org/r/gitweb?p=openstack/neutron-distgit.git;a=blob;f=openstack-neutron.spec;h=94b009ef6ca5caad050154216ba2525c7e5463ca;hb=refs/heads/queens-rdo#l488

The obvious solution here would be to make Neutron UID static.  However after talking to with some folks it looks like this path is more complicated.  Ideally we should have a common approach of either specifying or not specifying the UID for every service in RDO.  However, an easier fix for this is to implement a step to mount the cert/key as RW and then chmod the files with the neutron UID of the container.

Comment 7 Tim Rozet 2018-04-26 18:48:19 UTC
*** Bug 1572236 has been marked as a duplicate of this bug. ***

Comment 8 Tim Rozet 2018-04-26 18:50:35 UTC
The fix did not work:

https://review.openstack.org/#/c/558667/1/docker/services/neutron-dhcp.yaml

ERROR:__main__:Failed to change ownership of /etc/pki/tls/certs/neutron.crt to 42435:42435
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 345, in set_perms
    os.chown(path, uid, gid)
OSError: [Errno 30] Read-only file system: '/etc/pki/tls/certs/neutron.crt'
INFO:__main__:Setting permission for /etc/pki/tls/private/neutron.key
ERROR:__main__:Failed to change ownership of /etc/pki/tls/private/neutron.key to 42435:42435
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 345, in set_perms
    os.chown(path, uid, gid)
OSError: [Errno 30] Read-only file system: '/etc/pki/tls/private/neutron.key'

Comment 11 Itzik Brown 2018-05-03 08:19:37 UTC
Checked with:
puppet-tripleo-8.3.2-4.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-9.el7ost.noarch

Comment 13 errata-xmlrpc 2018-06-27 13:48:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.