In OSPd 13 deployment tempest VolumeBackup tests failing when Overcloud is using SSL, due to being unable to verify ssl certificate, cause seems to be that openstack-cinder-backup-docker-0 does not have access/config to /etc/pki/ content. snippet of output from failing test tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminTest.test_volume_backup_export_import[id-a99c54a1-dd80-4724-8a13-13bf58d4068d] > Response - Headers: {'status': '200', u'content-length': '922', 'content-location': 'https://10.0.0.101:13776/v2/d2523afb79544a4197fa79f2c5837ce6/backups/57af9d07-80f6-4706-add7-9337270dc950', > u'x-compute-request-id': 'req-0581536b-4bca-45d7-8d58-86c49ecbf825', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Tue, 30 Jan 2018 14:03:46 GMT', > u'content-type': 'application/json', u'x-openstack-request-id': 'req-0581536b-4bca-45d7-8d58-86c49ecbf825'} > Body: {"backup": {"status": "error", "object_count": 0, "container": "volumebackups", > "name": "tempest-VolumesBackupsAdminTest-Backup-1258019896", > "links": [{"href": "https://10.0.0.101:13776/v2/d2523afb79544a4197fa79f2c5837ce6/backups/57af9d07-80f6-4706-add7-9337270dc950", "rel": "self"}, > {"href": "https://10.0.0.101:13776/d2523afb79544a4197fa79f2c5837ce6/backups/57af9d07-80f6-4706-add7-9337270dc950", "rel": "bookmark"}], > "availability_zone": "nova", "created_at": "2018-01-30T14:03:23.000000", "description": null, > "updated_at": "2018-01-30T14:03:45.000000", "data_timestamp": "2018-01-30T14:03:23.000000", "has_dependent_backups": false, > "snapshot_id": null, "volume_id": "ca53f42d-d9c8-4776-95b3-b17f58c6c899", > > "fail_reason": "(\"bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)\",)", > > "is_incremental": false, "id": "57af9d07-80f6-4706-add7-9337270dc950", "size": 1}} docker inspect openstack-cinder-backup-docker-0 > "Binds": [ > "/run:/run:rw", > "/etc/iscsi:/var/lib/kolla/config_files/src-iscsid:ro", > "/var/lib/cinder:/var/lib/cinder:rw", > "/var/log/containers/cinder:/var/log/cinder:rw", > "/var/lib/kolla/config_files/cinder_backup.json:/var/lib/kolla/config_files/config.json:ro", > "/var/lib/config-data/puppet-generated/cinder/:/var/lib/kolla/config_files/src:ro", > "/etc/localtime:/etc/localtime:ro", > "/dev:/dev:rw", > "/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro", > "/etc/hosts:/etc/hosts:ro", > "/sys:/sys:rw", > "/lib/modules:/lib/modules:ro" > ], compared with e.g. docker inspect nova_api > "Binds": [ > > "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", > "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", > "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", > > "/dev/log:/dev/log", > "/var/log/containers/nova:/var/log/nova", > "/etc/localtime:/etc/localtime:ro", > "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", > "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro", > "/etc/puppet:/etc/puppet:ro", > "/var/log/containers/httpd/nova-api:/var/log/httpd", > "/var/lib/kolla/config_files/nova_api.json:/var/lib/kolla/config_files/config.json:ro", > "/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro", > "/etc/hosts:/etc/hosts:ro" > ], snip from rpm -qa list (mixed from uc and oc-ctl): > openstack-cinder.noarch 1:12.0.0-0.20180122233816.71b869c.el7ost > openstack-tripleo-common.noarch 8.3.1-0.20180123050218.el7ost > openstack-tripleo-common-containers.noarch 8.3.1-0.20180123050218.el7ost > openstack-tripleo-heat-templates.noarch 8.0.0-0.20180122224016.el7ost > openstack-tripleo-puppet-elements.noarch 8.0.0-0.20180117092204.120eca8.el7ost > puppet-cinder.noarch 12.2.0-0.20180123011607.277828c.el7ost > puppet-cinder.noarch 12.2.0-0.20180123011607.277828c.el7ost > puppet-tripleo.noarch 8.2.0-0.20180122224519.9fd3379.el7ost > puppet-tripleo.noarch 8.2.0-0.20180122224519.9fd3379.el7ost
Created attachment 1389115 [details] docker-container-startup-config-step_5.json
I have a local deployment with Cinder running in containers under pacemaker, and confirm that both cinder-volume and cinder-backup containers are missing the volume mounts that provide access to the SSL certs. These mounts are present on the other cinder containers (the ones not under pacemaker control). Looking further, on the controller in /var/lib/tripleo-config/docker-container-startup-config-step_5.json, the "cinder_volume_init_bundle" and "cinder_backup_init_bundle" configurations are missing several volume mounts that are supposed to be there. The THT [1] and [2] are supposed to include the ContainersCommon volume mounts [3], but this doesn't seem to be happening. [1] https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/pacemaker/cinder-volume.yaml#L180 [2] https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/pacemaker/cinder-backup.yaml#L186 [3] https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/containers-common.yaml#L121 I'd like DFG:DF to take a look at this.
It looks like your OSP build is missing this change: https://review.openstack.org/#/c/531261/ I can see in the THT rhos-13.0-patches branch this commit has not propagated through yet. Can you please reattempt when your copy of docker/services/containers-common.yaml has a docker_puppet_apply_volumes section?
(In reply to Steve Baker from comment #4) > It looks like your OSP build is missing this change: > > https://review.openstack.org/#/c/531261/ > > I can see in the THT rhos-13.0-patches branch this commit has not propagated > through yet. Can you please reattempt when your copy of > docker/services/containers-common.yaml has a docker_puppet_apply_volumes > section? the version of THT in original report of bz does appear to have that included but this is only from packaging and -patches branch quick check. openstack-tripleo-heat-templates.noarch 8.0.0-0.20180122224016.el7ost Which is based on upstream 2ebc2ee3af744bc1206fc710e0dc3bf68d59d20d from 2018-01-23 and that patch merged upstream on 2018-01-11.
I manually extracted containers-common.yaml and pacemaker/cinder-volume.yaml from openstack-tripleo-heat-templates.noarch 8.0.0-0.20180122224016.el7ost and they are identical to what's on upstream master. I also see the patches in my local osp-12 deployment, which is using the latest version of rhos/rhos-12.0-patches. This is the THT that resulted in the docker-container-startup-config-step_5.json attachment.
OK, two more pieces of information would be useful: - please also attach docker-container-startup-config-step_1.json, I want to see if the haproxy config has the same issue - can you provide the version of the PyYAML package on your undercloud
Created attachment 1389711 [details] docker-container-startup-config-step_1.json (undercloud) [stack@rhos-undercloud ~]$ rpm -q PyYAML PyYAML-3.10-11.el7.x86_64 Remember that this is my own local osp-12 deployment, using THT from rhos/rhos-12.0-patches.
Created attachment 1390339 [details] docker container info (step-config JSON and docker inspect output) OK, I have fresh data. First, I found an error in one of my environment files that caused the deployment to not use the latest rhos version of the pacemaker/cinder-X.yaml files. It was still deploying the GA version, which we know is missing the new docker_puppet_apply.sh code. So I fixed that and redeployed. What I'm seeing now is docker-container-startup-config-step_5.json looks good (it shows the CA cert mounts), but the mounts are missing in the running containers. Somehow the mounts in the config-step JSON are not present in the containers. I attached a full set of the /var/lib/tripleo JSON files, as well as the docker inspect output for the cinder-volume and cinder-backup containers.
OK, these cinder containers are not managed by paunch via docker-container-startup-config-step*.json. They are managed by pacemaker, and configured by the puppet-tripleo manifest tripleo::profile::pacemaker::cinder::*_bundle. It looks like this means that the missing volume mounts need to be added to the storage_maps blocks: http://git.openstack.org/cgit/openstack/puppet-tripleo/tree/manifests/profile/pacemaker/cinder/volume_bundle.pp#n81 http://git.openstack.org/cgit/openstack/puppet-tripleo/tree/manifests/profile/pacemaker/cinder/backup_bundle.pp#n81 I'll reassign this to puppet-tripleo. It might be more appropriate for PIDONE DFG to handle the fix, but I'll ask around first.
Upstream fix posted
According to our records, this should be resolved by openstack-tripleo-heat-templates-7.0.9-8.el7ost. This build is available now.
Test is passing on latest puddle 2018-05-15.2 https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/job/phase2-13_director-rhel-7.5-virthost-1cont_1comp-ipv4-vxlan-lvm-ssl-containers/42/testReport/