Bug 1320379

Summary: RFE: Allow Conversion from non-SSL to SSL
Product: Red Hat OpenStack Reporter: Andreas Karis <akaris>
Component: openstack-tripleoAssignee: Juan Antonio Osorio <josorior>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: jcoufal, jmelvin, josorior, jschluet, mburns, mcornea, michele, nkinder, rcritten, rhel-osp-director-maint, sasha, scohen, srevivo
Target Milestone: rcKeywords: FutureFeature, Triaged
Target Release: 10.0 (Newton)Flags: scohen: needinfo+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-0.0.1-0.20160916135259.4de13b3.el7ost, puppet-tripleo-5.3.0-5.el7ost Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 15:29:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Karis 2016-03-23 04:53:56 UTC
Description of problem:
This happens when trying to convert Director endpoints from non-SSL to SSL according to https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Advanced-Scenario_3_Using_the_CLI_to_Create_an_Advanced_Overcloud_with_Ceph_Nodes.html#sect-Advanced-Enabling_SSL_TLS_on_the_Overcloud

Initial deployment with this step-by-step guide works. However, if I first deploy a new environment without SSL, and then try to convert it to SSL, the following happens:
- endpoints are badly configured

Version-Release number of selected component (if applicable):
7.3

How reproducible:
all of the time

Steps to Reproduce:
1. deploy Director without SSL
2. configure SSL
3. redeploy Director

Actual results:
+----------------------------------+-----------+--------------------------------------------+-----------------------------------------------+-----------------------------------------+----------------------------------+
|                id                |   region  |                 publicurl                  |                  internalurl                  |                 adminurl                |            service_id            |
+----------------------------------+-----------+--------------------------------------------+-----------------------------------------------+-----------------------------------------+----------------------------------+
| 1ac53a0aee6a4c84bf627ca5132bd7ab | regionOne |   http://10.0.0.4:8004/v1/%(tenant_id)s    |    http://172.16.2.4:8004/v1/%(tenant_id)s    | http://172.16.2.4:8004/v1/%(tenant_id)s | 3e36d03c27d64dc9a3e2fc8842a057ef |
| 2a007e52e36e4d5baddc52cff8951a8e | regionOne |           http://10.0.0.4:9696/            |            http://172.16.2.4:9696/            |         http://172.16.2.4:9696/         | c6cbcd1d39d34a119d189511541960b2 |
| 6150a642cc8c46b7a75e0b784ecf86f8 | regionOne |   http://10.0.0.4:8776/v1/%(tenant_id)s    |    http://172.16.2.4:8776/v1/%(tenant_id)s    | http://172.16.2.4:8776/v1/%(tenant_id)s | 95fc966bc34e48649ab11b5d5db9bb91 |
| 77841d30267c4bfc8f71545140fad4d9 | regionOne |           http://10.0.0.4:8777/            |            http://172.16.2.4:8777/            |         http://172.16.2.4:8777/         | 27a420a5a7cc40e18b33b50fda80f672 |
| a98aea83fe934149b99d9cbc451ba7fb | regionOne |           http://10.0.0.4:9292/            |            http://172.18.0.10:9292/           |         http://172.18.0.10:9292/        | 4aff5165a98a42d39765f33eceb8529c |
| abb934a1eb2d4889804a2b607746a46c | regionOne |   http://10.0.0.4:8774/v2/$(tenant_id)s    |    http://172.16.2.4:8774/v2/$(tenant_id)s    | http://172.16.2.4:8774/v2/$(tenant_id)s | 5a2c224b493c49be8ac189d2a0739f7c |
| b208bb10f6a14a06bae32e05cf0bff3c | regionOne |         http://10.0.0.4:5000/v2.0          |          http://172.16.2.4:5000/v2.0          |       http://192.0.2.19:35357/v2.0      | 2b1a9869752c462099c389bbda5bcddd |
| c2fc9d47aa704424addb734c4d079357 | regionOne |          http://10.0.0.4:8774/v3           |           http://172.16.2.4:8774/v3           |        http://172.16.2.4:8774/v3        | 021e736d8fda486dbff2f28ca39bc5b1 |
| e8c3f6bb31ee4bdb93f8e364949ae137 | regionOne | http://10.0.0.4:8080/v1/AUTH_%(tenant_id)s | http://172.18.0.10:8080/v1/AUTH_%(tenant_id)s |        http://172.18.0.10:8080/v1       | bc489c937e124a499de535da454f17f0 |
| f784e74c37b24c1bb2add4202fc93d75 | regionOne |   http://10.0.0.4:8776/v2/%(tenant_id)s    |    http://172.16.2.4:8776/v2/%(tenant_id)s    | http://172.16.2.4:8776/v2/%(tenant_id)s | d0cd68bc7908434caf509c0de8272f67 |
| fb27d5a10ef94b88adfcc392dd8dfa43 | regionOne |       http://10.0.0.4:80/dashboard/        |         http://10.0.0.4:80/dashboard/         |    http://10.0.0.4:80/dashboard/admin   | 6b2fa795cff1452db858fe2e27e12766 |
+----------------------------------+-----------+--------------------------------------------+-----------------------------------------------+-----------------------------------------+----------------------------------+


Expected results:
+----------------------------------+-----------+-----------------------------------------------------+-----------------------------------------------+-------------------------------------------+----------------------------------+
|                id                |   region  |                      publicurl                      |                  internalurl                  |                  adminurl                 |            service_id            |
+----------------------------------+-----------+-----------------------------------------------------+-----------------------------------------------+-------------------------------------------+----------------------------------+
| 088182a67c504d47ba0e0e29e4642c85 | regionOne | https://osp.example.net:13808/v1/AUTH_%(tenant_id)s | http://172.18.0.10:8080/v1/AUTH_%(tenant_id)s |         http://172.18.0.10:8080/v1        | f5ce3d3fc3d04fdc9bf59ea51211b629 |
| 0f352e5b58174b4489d34becfdc618f1 | regionOne |          https://osp.example.net:13000/v2.0         |          http://172.16.2.5:5000/v2.0          |        http://192.0.2.8:35357/v2.0        | 93d3a2f1655849abb4954ef2769f608a |
| 22b1f36030be47b4a685f9c35558cd50 | regionOne |            https://osp.example.net:13696/           |            http://172.16.2.5:9696/            |          http://172.16.2.5:9696/          | 4302251c1ad34db58afcd48f57eb1e5f |
| 3d3cc3fd12754d138d5297de9646cab3 | regionOne |    https://osp.example.net:13776/v2/%(tenant_id)s   |    http://172.16.2.5:8776/v2/%(tenant_id)s    |  http://172.16.2.5:8776/v2/%(tenant_id)s  | 52453f5a1d1943cc8d2f70f4dea63b6d |
| 40a1d0feb6374667857f333ccfaebe08 | regionOne |         http://osp.example.net:80/dashboard/        |      http://osp.example.net:80/dashboard/     | http://osp.example.net:80/dashboard/admin | 0c954194a95c471b9353b7e5c4803d0c |
| 6a789ea0bc2a40f3a1852e3d1b3aeb05 | regionOne |           https://osp.example.net:13774/v3          |           http://172.16.2.5:8774/v3           |         http://172.16.2.5:8774/v3         | 0c62e551fd9a4dd7926a1c5038ae75ca |
| 8cc0d3b9fc9f4801b418eddb438f8541 | regionOne |            https://osp.example.net:13292/           |            http://172.18.0.10:9292/           |          http://172.18.0.10:9292/         | 2d0e665d9e0e4da998aa526e3f5d78ea |
| a35e87a654814c319478c6f74c3d8ea1 | regionOne |    https://osp.example.net:13776/v1/%(tenant_id)s   |    http://172.16.2.5:8776/v1/%(tenant_id)s    |  http://172.16.2.5:8776/v1/%(tenant_id)s  | 7bff84c8f0754bcb894b31a46484ddca |
| b3ff4bdc519c4210a2653ea7fe9a1915 | regionOne |    https://osp.example.net:13774/v2/$(tenant_id)s   |    http://172.16.2.5:8774/v2/$(tenant_id)s    |  http://172.16.2.5:8774/v2/$(tenant_id)s  | 347d290e609d478aac7e97d9d2991ac3 |
| e408e6690f3643839e9b8d9b228e5532 | regionOne |    https://osp.example.net:13004/v1/%(tenant_id)s   |    http://172.16.2.5:8004/v1/%(tenant_id)s    |  http://172.16.2.5:8004/v1/%(tenant_id)s  | a64fc99cdc8b4ec98d12ad869388e9df |
| fb1b8bcc47bd4c4c8497067f28b77962 | regionOne |            https://osp.example.net:13777/           |            http://172.16.2.5:8777/            |          http://172.16.2.5:8777/          | 433fe43e0d254426b5ec1d4b6b0ac567 |
+----------------------------------+-----------+-----------------------------------------------------+-----------------------------------------------+-------------------------------------------+----------------------------------+


Additional info:

Comment 2 Andreas Karis 2016-03-23 05:21:19 UTC
So I think that the issue is here:

rdomanager_oscplugin/v1/overcloud_deploy.py
(...)
 keystone.setup_endpoints(
            services,
            client=keystone_client,
            os_auth_url=overcloud_endpoint,
            public_host=overcloud_ip_or_fqdn)
(...)

Which at some point creates endpoints by means of this helper function which will simply abandon if endpoints already exist:

os_cloud_config/keystone.py
(...)
def _create_endpoint(keystone, region, service_id, public_uri, admin_uri,
                     internal_uri):
    """Helper for idempotent creating of endpoint.

    :param keystone: keystone v2 client
    :param region: endpoint region
    :param service_id: id of associated service
    :param public_uri: endpoint public uri
    :param admin_uri: endpoint admin uri
    :param internal_uri: endpoint internal uri
    """
    if keystone.endpoints.findall(publicurl=public_uri):
        LOG.info('Endpoint for service %s and public uri %s '
                 'already exists.', service_id, public_uri)
    else:
        LOG.debug('Creating endpoint for service %s.', service_id)
        keystone.endpoints.create(
            region, service_id, public_uri, admin_uri, internal_uri)
(...)

Comment 3 Juan Antonio Osorio 2016-03-23 08:02:18 UTC
that is indeed the issue; os-cloud-config is only run once in the post config. It isn't run on updates... We need to stop using that and take puppet into use for creating/updating the keystone endpoints. Because, AFAIK, os-cloud-config doesn't even have the capabilities of updating the endpoints at all.

Comment 4 Mike Burns 2016-03-23 13:41:08 UTC
Currently, we do not support migrating from non-SSL to SSL without a complete re-deployment.  That makes this request an RFE.

Comment 14 Marius Cornea 2016-10-24 12:58:49 UTC
Tested on openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost  

It looks that we're missing a step to do 'pcs resource haproxy restart' while running the overcloud deploy command which enables ssl. After the overcloud update is complete I'm not able to reach the haproxy endpoints and a haproxy restart is required.

Comment 15 Juan Antonio Osorio 2016-10-26 16:03:42 UTC
Marius, could you check if doing systemctl reload haproxy works? instead of doing pcs resource haproxy restart. Doing a reload of the configuration should work, and I have the feeling this bug https://bugs.launchpad.net/tripleo/+bug/1627254 is hitting us there.

Comment 16 Juan Antonio Osorio 2016-10-27 15:22:09 UTC
So, it seems that the issue is that we're not restarting haproxy via pacemaker anymore, so I've submitted a fix to work around that issue.

Comment 21 errata-xmlrpc 2016-12-14 15:29:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html