Bug 1540565 - containerized cinder pacemaker bundles are missing ssl CA certs
Summary: containerized cinder pacemaker bundles are missing ssl CA certs
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
Target Milestone: z3
: 12.0 (Pike)
Assignee: Steve Baker
QA Contact: Gurenko Alex
Depends On:
Blocks: 1543152
TreeView+ depends on / blocked
Reported: 2018-01-31 12:12 UTC by Pavel Sedlák
Modified: 2018-05-17 10:35 UTC (History)
16 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.3-23.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1543152 (view as bug list)
Last Closed: 2018-05-17 10:35:32 UTC
Target Upstream Version:

Attachments (Terms of Use)
docker-container-startup-config-step_5.json (3.56 KB, text/plain)
2018-01-31 17:33 UTC, Alan Bishop
no flags Details
docker-container-startup-config-step_1.json (9.31 KB, text/plain)
2018-02-01 19:13 UTC, Alan Bishop
no flags Details
docker container info (step-config JSON and docker inspect output) (8.53 KB, application/octet-stream)
2018-02-02 21:28 UTC, Alan Bishop
no flags Details

System ID Priority Status Summary Last Updated
Launchpad 1747326 None None None 2018-02-05 02:04:16 UTC
OpenStack gerrit 536547 None stable/pike: MERGED tripleo-heat-templates: Correct indentation of docker_puppet_apply.sh run (I80734ccc842d5be01555ee5a863d0de9a8101d33) 2018-02-12 15:05:20 UTC
OpenStack gerrit 541448 None stable/pike: MERGED puppet-tripleo: Add missing pacemaker cindier CA cert mounts (I199c03ba36a24e6b1caf535ed285047952ac9eb0) 2018-02-12 15:04:06 UTC

Description Pavel Sedlák 2018-01-31 12:12:58 UTC
In OSPd 13 deployment tempest VolumeBackup tests failing when Overcloud is using SSL,
due to being unable to verify ssl certificate,
cause seems to be that openstack-cinder-backup-docker-0 does not have access/config to /etc/pki/ content.

snippet of output from failing test tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminTest.test_volume_backup_export_import[id-a99c54a1-dd80-4724-8a13-13bf58d4068d]
> Response - Headers: {'status': '200', u'content-length': '922', 'content-location': '',
>     u'x-compute-request-id': 'req-0581536b-4bca-45d7-8d58-86c49ecbf825', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Tue, 30 Jan 2018 14:03:46 GMT',
>     u'content-type': 'application/json', u'x-openstack-request-id': 'req-0581536b-4bca-45d7-8d58-86c49ecbf825'}
> Body: {"backup": {"status": "error", "object_count": 0, "container": "volumebackups",
>     "name": "tempest-VolumesBackupsAdminTest-Backup-1258019896",
>     "links": [{"href": "", "rel": "self"},
>         {"href": "", "rel": "bookmark"}],
>   "availability_zone": "nova", "created_at": "2018-01-30T14:03:23.000000", "description": null,
>   "updated_at": "2018-01-30T14:03:45.000000", "data_timestamp": "2018-01-30T14:03:23.000000", "has_dependent_backups": false,
>   "snapshot_id": null, "volume_id": "ca53f42d-d9c8-4776-95b3-b17f58c6c899",
>   "fail_reason": "(\"bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)\",)",
>   "is_incremental": false, "id": "57af9d07-80f6-4706-add7-9337270dc950", "size": 1}}

docker inspect openstack-cinder-backup-docker-0
> "Binds": [
>     "/run:/run:rw",
>     "/etc/iscsi:/var/lib/kolla/config_files/src-iscsid:ro",
>     "/var/lib/cinder:/var/lib/cinder:rw",
>     "/var/log/containers/cinder:/var/log/cinder:rw",
>     "/var/lib/kolla/config_files/cinder_backup.json:/var/lib/kolla/config_files/config.json:ro",
>     "/var/lib/config-data/puppet-generated/cinder/:/var/lib/kolla/config_files/src:ro",
>     "/etc/localtime:/etc/localtime:ro",
>     "/dev:/dev:rw",
>     "/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro",
>     "/etc/hosts:/etc/hosts:ro",
>     "/sys:/sys:rw",
>     "/lib/modules:/lib/modules:ro"
> ],

compared with e.g.
docker inspect nova_api
> "Binds": [
>     "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
>     "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
>     "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
>     "/dev/log:/dev/log",
>     "/var/log/containers/nova:/var/log/nova",
>     "/etc/localtime:/etc/localtime:ro",
>     "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
>     "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
>     "/etc/puppet:/etc/puppet:ro",
>     "/var/log/containers/httpd/nova-api:/var/log/httpd",
>     "/var/lib/kolla/config_files/nova_api.json:/var/lib/kolla/config_files/config.json:ro",
>     "/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro",
>     "/etc/hosts:/etc/hosts:ro"
> ],

snip from rpm -qa list (mixed from uc and oc-ctl):
> openstack-cinder.noarch            1:12.0.0-0.20180122233816.71b869c.el7ost
> openstack-tripleo-common.noarch  8.3.1-0.20180123050218.el7ost
> openstack-tripleo-common-containers.noarch 8.3.1-0.20180123050218.el7ost
> openstack-tripleo-heat-templates.noarch 8.0.0-0.20180122224016.el7ost
> openstack-tripleo-puppet-elements.noarch 8.0.0-0.20180117092204.120eca8.el7ost
> puppet-cinder.noarch               12.2.0-0.20180123011607.277828c.el7ost
> puppet-cinder.noarch             12.2.0-0.20180123011607.277828c.el7ost
> puppet-tripleo.noarch              8.2.0-0.20180122224519.9fd3379.el7ost
> puppet-tripleo.noarch            8.2.0-0.20180122224519.9fd3379.el7ost

Comment 2 Alan Bishop 2018-01-31 17:33:36 UTC
Created attachment 1389115 [details]

Comment 3 Alan Bishop 2018-01-31 17:37:58 UTC
I have a local deployment with Cinder running in containers under pacemaker, and confirm that both cinder-volume and cinder-backup containers are missing the volume mounts that provide access to the SSL certs. These mounts are present on the other cinder containers (the ones not under pacemaker control).

Looking further, on the controller in /var/lib/tripleo-config/docker-container-startup-config-step_5.json, the "cinder_volume_init_bundle" and "cinder_backup_init_bundle" configurations are missing several volume mounts that are supposed to be there. The THT [1] and [2] are supposed to include the ContainersCommon volume mounts [3], but this doesn't seem to be happening.

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/pacemaker/cinder-volume.yaml#L180
[2] https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/pacemaker/cinder-backup.yaml#L186
[3] https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/containers-common.yaml#L121

I'd like DFG:DF to take a look at this.

Comment 4 Steve Baker 2018-02-01 01:37:14 UTC
It looks like your OSP build is missing this change:


I can see in the THT rhos-13.0-patches branch this commit has not propagated through yet. Can you please reattempt when your copy of docker/services/containers-common.yaml has a docker_puppet_apply_volumes section?

Comment 5 Jon Schlueter 2018-02-01 03:26:05 UTC
(In reply to Steve Baker from comment #4)
> It looks like your OSP build is missing this change:
> https://review.openstack.org/#/c/531261/
> I can see in the THT rhos-13.0-patches branch this commit has not propagated
> through yet. Can you please reattempt when your copy of
> docker/services/containers-common.yaml has a docker_puppet_apply_volumes
> section?

the version of THT in original report of bz does appear to have that included but this is only from packaging and -patches branch quick check.

openstack-tripleo-heat-templates.noarch 8.0.0-0.20180122224016.el7ost
Which is based on upstream 2ebc2ee3af744bc1206fc710e0dc3bf68d59d20d from 2018-01-23 and that patch merged upstream on 2018-01-11.

Comment 6 Alan Bishop 2018-02-01 13:17:18 UTC
I manually extracted containers-common.yaml and pacemaker/cinder-volume.yaml from openstack-tripleo-heat-templates.noarch 8.0.0-0.20180122224016.el7ost and they are identical to what's on upstream master.

I also see the patches in my local osp-12 deployment, which is using the latest version of rhos/rhos-12.0-patches. This is the THT that resulted in the docker-container-startup-config-step_5.json attachment.

Comment 8 Steve Baker 2018-02-01 19:02:59 UTC
OK, two more pieces of information would be useful:
- please also attach docker-container-startup-config-step_1.json, I want to see if the haproxy config has the same issue
- can you provide the version of the PyYAML package on your undercloud

Comment 9 Alan Bishop 2018-02-01 19:13:59 UTC
Created attachment 1389711 [details]

(undercloud) [stack@rhos-undercloud ~]$ rpm -q PyYAML

Remember that this is my own local osp-12 deployment, using THT from rhos/rhos-12.0-patches.

Comment 14 Alan Bishop 2018-02-02 21:28:42 UTC
Created attachment 1390339 [details]
docker container info (step-config JSON and docker inspect output)

OK, I have fresh data. First, I found an error in one of my environment files
that caused the deployment to not use the latest rhos version of the
pacemaker/cinder-X.yaml files. It was still deploying the GA version, which we
know is missing the new docker_puppet_apply.sh code. So I fixed that and

What I'm seeing now is docker-container-startup-config-step_5.json looks good
(it shows the CA cert mounts), but the mounts are missing in the running
containers. Somehow the mounts in the config-step JSON are not present in the

I attached a full set of the /var/lib/tripleo JSON files, as well as the
docker inspect output for the cinder-volume and cinder-backup containers.

Comment 15 Steve Baker 2018-02-04 22:01:35 UTC
OK, these cinder containers are not managed by paunch via docker-container-startup-config-step*.json. They are managed by pacemaker, and configured by the puppet-tripleo manifest tripleo::profile::pacemaker::cinder::*_bundle.

It looks like this means that the missing volume mounts need to be added to the storage_maps blocks:

I'll reassign this to puppet-tripleo.

It might be more appropriate for PIDONE DFG to handle the fix, but I'll ask around first.

Comment 16 Steve Baker 2018-02-05 02:04:45 UTC
Upstream fix posted

Comment 22 Lon Hohberger 2018-04-06 10:33:55 UTC
According to our records, this should be resolved by openstack-tripleo-heat-templates-7.0.9-8.el7ost.  This build is available now.

Note You need to log in before you can comment on or make changes to this bug.