Bug 1619819

Summary: [OSP 13] overcloud deployment breaks at ControllerDeployment_Step3 when manila and tls everywhere are deployed together
Product: Red Hat OpenStack Reporter: Matt Flusche <mflusche>
Component: openstack-tripleo-heat-templatesAssignee: Goutham Pacha Ravi <gouthamr>
Status: CLOSED CURRENTRELEASE QA Contact: Jason Grosso <jgrosso>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: abishop, aschultz, dbecker, gouthamr, lmarsh, mburns, morazi, pgrist, slinaber, tbarron
Target Milestone: z3Keywords: TestOnly, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.0.7-2.el7ost Doc Type: Bug Fix
Doc Text:
The Docker manifests for the manila-api did not contain a necessary bind mount to provide SSL options. As a result, Overcloud deployments failed when deploying with manila and TLS-Everywhere. With this release, the bind mounts for the bootstrap containers are fixed. Overcloud deployments succeed with manila and TLS-Everywhere.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-20 11:44:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1644747    
Bug Blocks:    

Description Matt Flusche 2018-08-21 19:59:28 UTC
Description of problem:
When manila and tls everywhere are deployed together the deployment fails at ControllerDeployment_Step3

manila_api_db_sync fails as it appear to use a non-ssl client connection to mariadb; however, ssl is required by the tls everywhere config.

I'll attach more detailed errors to the case.

Deployment error:

overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: c1773ea6-9eff-499d-936f-8894b948d8a2
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |

[...]

    TASK [Debug output for task which failed: Run puppet host configuration for step 3] ***
    ok: [localhost] => {
        "failed_when_result": false, 

[...]

            "Error running ['docker', 'run', '--name', 'manila_api_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=manila_api_db_sync', '--label', 'managed_by=paunc
h', '--label', 'config_data={\"command\": \"/usr/bin/bootstrap_host_exec manila_api su manila -s /bin/bash -c \\'/usr/bin/manila-manage db sync\\'\", \"user\": \"root\", \"volumes\": [\"/etc/
hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.cr
t:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ipa/ca.crt:/etc
/ipa/ca.crt:ro\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/manila/etc/manila/:/etc/manila/:ro\", \"/var/log/containers/
manila:/var/log/manila\", \"/var/log/containers/httpd/manila-api:/var/log/httpd\"], \"image\": \"hostname:5000/osp13_containers-manila-api:13.0-47\", \"detach\": false,
 \"net\": \"host\"}', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/ex
tracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--v
olume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts
:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/manila/etc/manila/:/etc/manila/:ro', '--volume=/var/log/containers/manila:/var/log/manila', '--volume=/var/log/cont
ainers/httpd/manila-api:/var/log/httpd', 'hostname:5000osp13_containers-manila-api:13.0-47', '/usr/bin/bootstrap_host_exec', 'manila_api', 'su', 'manila', '-s', '/bin/
bash', '-c', \"'/usr/bin/manila-manage\", 'db', \"sync'\"]. [1]", 

Error from /var/log/containers/manila/manila-manage.log on controller 0:

2018-08-16 23:32:43.351 11 ERROR manila OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'manila'@'X.X.X.X' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)


Version-Release number of selected component (if applicable):
OSP 13 current

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP 13 with manila and tls everywhere
2.
3.


Additional info:

Will attach links to additional logs

Comment 15 Jason Grosso 2018-12-19 19:11:00 UTC
verified on OSP 13 z4 install with TLS anywhere was successful verified once installed could mount a manila share, write and read a file with TLS enabled  

jenkins build https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/OSPD-Customized-Deployment-virt/8306/artifact/.sh/08-ir-tripleo-overcloud-deploy.log

Comment 16 Lon Hohberger 2018-12-20 11:44:43 UTC
According to our records, this should be resolved by openstack-tripleo-heat-templates-8.0.7-4.el7ost.  This build is available now.