Bug 1597379
| Summary: | docker manila-share container missing ca-trust | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | David Vallee Delisle <dvd> | |
| Component: | openstack-tripleo-heat-templates | Assignee: | Tom Barron <tbarron> | |
| Status: | CLOSED ERRATA | QA Contact: | Dustin Schoenbrun <dschoenb> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 13.0 (Queens) | CC: | abishop, aschultz, dhill, dschoenb, emacchi, gouthamr, joflynn, jschluet, marjones, mburns, pablo.iranzo, pgrist, skinjo, tbarron, vcojot | |
| Target Milestone: | z2 | Keywords: | Triaged, ZStream | |
| Target Release: | 13.0 (Queens) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-tripleo-heat-templates-8.0.4-3.el7ost puppet-tripleo-8.3.4-2.el7ost openstack-tripleo-heat-templates-8.0.4-16.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
The manila-share service container failed to bind-mount PKI trust stores from the controller host. As a result, connections from the manila-share service to the storage back end could not be encrypted using SSL.
Bind-mount the PKI trust stores from the controller host into the manila-share service container. The connections from the manila-share service to the storage back end can now be encrypted using SSL.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1598589 (view as bug list) | Environment: | ||
| Last Closed: | 2018-08-29 16:37:54 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1598589 | |||
manila_api and scheduler have the correct mounts: $ sudo docker inspect manila_api | jq .[0].HostConfig.Binds [ "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro", "/etc/puppet:/etc/puppet:ro", "/var/log/containers/httpd/manila-api:/var/log/httpd", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", "/var/lib/kolla/config_files/manila_api.json:/var/lib/kolla/config_files/config.json:ro", "/var/lib/config-data/puppet-generated/manila/:/var/lib/kolla/config_files/src:ro", "/var/log/containers/manila:/var/log/manila", "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro" ] Back to ON-DEV since we also need https://review.openstack.org/#/c/584949/ so the other fixes don't cause a regression when manila is deployed with CephFS back ends. Moving to POST as all upstream reviews have merged. *** Bug 1613532 has been marked as a duplicate of this bug. *** Installed OSP13z2 08-08.2 puddle. Per https://bugzilla.redhat.com/show_bug.cgi?id=1613532 with manila-share container starting: [heat-admin@controller-1 ~]$ sudo docker ps | grep manila 8ebb5550c0f4 192.168.24.1:8787/rhosp13/openstack-manila-share:pcmklatest "/bin/bash /usr/lo..." 2 hours ago Up 2 hours openstack-manila-share-docker-0 Issue is resolved. Per OP bug creation, no errors in manila-share.log and driver initialized: [heat-admin@controller-1 ~]$ sudo grep ERROR /var/log/containers/manila/manila-share.log [heat-admin@controller-1 ~]$ sudo grep netapp /var/log/containers/manila/manila-share.log | grep driver | grep initial 2018-08-09 14:05:23.494 44 INFO manila.share.manager [req-46f461e5-9677-4af9-aceb-4b365534b8d0 - - - - -] Finished initialization of driver: 'NetAppCmodeSingleSvmShareDriver@hostgroup@tripleo_netapp' Per OP's binds: [heat-admin@controller-1 ~]$ sudo docker inspect manila_api | jq .[0].HostConfig.Binds [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro", "/var/log/containers/manila:/var/log/manila", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", "/var/log/containers/httpd/manila-api:/var/log/httpd", "/var/lib/kolla/config_files/manila_api.json:/var/lib/kolla/config_files/config.json:ro", "/var/lib/config-data/puppet-generated/manila/:/var/lib/kolla/config_files/src:ro" ] Tested creating a share also, so marking bug as verified. This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible. If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-". To add draft documentation text: * Select the documentation type from the "Doc Type" drop down field. * A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2574 |
Description of problem: manila-share container deployed by OpenStack 13 does not bind-mount the PKI trust store resulting in failure to access NetApp device: ~~~ [heat-admin@controller1 ~]$ sudo docker inspect openstack-manila-share-docker-0 | jq .[0].HostConfig.Binds [ "/var/log/containers/manila:/var/log/manila:rw", "/etc/ceph:/etc/ceph:ro", "/var/lib/config-data/puppet-generated/manila/:/var/lib/kolla/config_files/src:ro", "/etc/hosts:/etc/hosts:ro", "/run:/run:rw", "/lib/modules:/lib/modules:ro", "/var/lib/manila:/var/lib/manila:rw", "/var/lib/kolla/config_files/manila_share.json:/var/lib/kolla/config_files/config.json:ro", "/etc/localtime:/etc/localtime:ro", "/dev:/dev:rw", "/sys:/sys:rw" ] ~~~ [heat-admin@controller1 ~]$ sudo grep ERROR /var/log/containers/manila/manila-share.log [...] 2018-07-02 18:51:18.493 44 ERROR manila.share.manager [req-cd91734d-d84c-4f13-8182-e10789cf5d01 - - - - -] Error encountered during initialization of driver NetAppCmodeMultiSvmShareDriver@hostgroup@tripleo_manila_netapp: StorageCommunicationException: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)> 2018-07-02 18:51:18.493 44 ERROR manila.share.manager Traceback (most recent call last): 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 305, in _driver_setup 2018-07-02 18:51:18.493 44 ERROR manila.share.manager self.driver.do_setup(ctxt) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py", line 40, in do_setup 2018-07-02 18:51:18.493 44 ERROR manila.share.manager self.library.do_setup(context) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper 2018-07-02 18:51:18.493 44 ERROR manila.share.manager result = f(self, *args, **kwargs) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 130, in do_setup 2018-07-02 18:51:18.493 44 ERROR manila.share.manager self._client = self._get_api_client() 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper 2018-07-02 18:51:18.493 44 ERROR manila.share.manager result = f(self, *args, **kwargs) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 167, in _get_api_client 2018-07-02 18:51:18.493 44 ERROR manila.share.manager api_trace_pattern=na_utils.API_TRACE_PATTERN) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 57, in __init__ 2018-07-02 18:51:18.493 44 ERROR manila.share.manager (major, minor) = self.get_ontapi_version(cached=False) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/client/client_base.py", line 46, in get_ontapi_version 2018-07-02 18:51:18.493 44 ERROR manila.share.manager enable_tunneling=False) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/client/client_base.py", line 88, in send_request 2018-07-02 18:51:18.493 44 ERROR manila.share.manager return self.connection.invoke_successfully(request, enable_tunneling) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/client/api.py", line 275, in invoke_successfully 2018-07-02 18:51:18.493 44 ERROR manila.share.manager result = self.invoke_elem(na_element, enable_tunneling) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager File "/usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/client/api.py", line 255, in invoke_elem 2018-07-02 18:51:18.493 44 ERROR manila.share.manager raise exception.StorageCommunicationException(six.text_type(e)) 2018-07-02 18:51:18.493 44 ERROR manila.share.manager StorageCommunicationException: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)> 2018-07-02 18:51:18.493 44 ERROR manila.share.manager ~~~ [heat-admin@controller1 ~]$ sudo docker exec -ti openstack-manila-share-docker-0 bash ()[root@controller1 /]# echo | openssl s_client -connect netapp-mgmt:443 -showcerts | openssl verify depth=0 C = US, ST = California, L = Somewhere, O = Company Name, OU = Blabla, CN = netapp-mgmt verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 C = US, ST = California, L = Somewhere, O = Company Name, OU = Blabla, CN = netapp-mgmt verify error:num=21:unable to verify the first certificate verify return:1 DONE stdin: C = US, ST = California, L = Somewhere, O = Company Name, OU = Blabla, CN = netapp-mgmt error 20 at 0 depth lookup:unable to get local issuer certificate ~~~ Certificate verifies good from host: ~~~ [heat-admin@controller1 ~]$ echo | openssl s_client -connect $(sudo crudini --get /var/lib/config-data/puppet-generated/manila/etc/manila/manila.conf tripleo_manila_netapp netapp_server_hostname):443 -showcerts | openssl verify depth=1 CN = SomeLab CA, C = US, ST = California, L = Somewhere, O = Company Name, OU = Blabla verify return:1 depth=0 C = US, ST = California, L = Somewhere, O = Company Name, OU = Blabla, CN = netapp-mgmt verify return:1 DONE stdin: OK Version-Release number of selected component (if applicable): ~~~ $ sudo docker inspect openstack-manila-share-docker-0 | jq .[0].Config.Labels { "version": "13.0", "vendor": "Red Hat, Inc.", "vcs-type": "git", "distribution-scope": "public", "description": "Red Hat OpenStack Platform 13.0 manila-share", "com.redhat.component": "openstack-manila-share-container", "com.redhat.build-host": "osbs-cpt-001.ocp.osbs.upshift.eng.rdu2.redhat.com", "build-date": "2018-06-15T17:41:48.394334", "batch": "20180615.1", "authoritative-source-url": "registry.access.redhat.com", "architecture": "x86_64", "io.k8s.description": "Red Hat OpenStack Platform 13.0 manila-share", "io.k8s.display-name": "Red Hat OpenStack Platform 13.0 manila-share", "io.openshift.tags": "rhosp osp openstack osp-13.0", "name": "rhosp13/openstack-manila-share", "release": "39", "summary": "Red Hat OpenStack Platform 13.0 manila-share", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp13/openstack-manila-share/images/13.0-39", "vcs-ref": "b830cd6e54b8fb36268258f6a04a8881194217d7" } ~~~ ~~~ stack@undercloud $ rpm -q openstack-tripleo-heat-templates puppet-manila openstack-tripleo-heat-templates-8.0.2-38.el7ost.noarch puppet-manila-12.4.0-0.20180329035214.6c18418.el7ost.noarch ~~~ How reproducible: All the time