Bug 1573322 - python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch breaks SSL enabled Undercloud
Summary: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch breaks SSL enabled U...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 10.0 (Newton)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-30 20:30 UTC by Matt Flusche
Modified: 2019-02-17 13:10 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-29 13:39:25 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Matt Flusche 2018-04-30 20:30:29 UTC
Description of problem:

After patching a OSP 10 SSL enabled undercloud I get the following fatal SSL errors.

[stack@undercloud10 ~]$ source stackrc 
[stack@undercloud10 ~]$ openstack server list
Certificate did not match expected hostname: 172.16.3.10. Certificate: {'subjectAltName': (('DNS', '172.16.3.10'),), 'notBefore': u'Jun  8 13:44:01 2017 GMT', 'serialNumber': u'D8FAEC41147248F09AB3D5D678B633FC', 'notAfter': 'Jun  8 13:43:56 2018 GMT', 'version': 3L, 'subject': ((('commonName', u'172.16.3.10'),),), 'issuer': ((('commonName', u'Local Signing Authority'),), (('commonName', u'd8faec41-147248f0-9ab3d5d6-78b633fb'),))}
SSL exception connecting to https://172.16.3.10:13000/v2.0/tokens: hostname '172.16.3.10' doesn't match '172.16.3.10'

I traced this down to the updated package:

python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch


If I downgrade this package, ssl completes successfully.

# fully updated env
[stack@undercloud10 ~]$ sudo yum repolist                                                                                                                  
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id                                                        repo name                                                                             status
rhel-7-server-extras-rpms/x86_64                               Red Hat Enterprise Linux 7 Server - Extras (RPMs)                                        814
rhel-7-server-openstack-10-rpms/7Server/x86_64                 Red Hat OpenStack Platform 10 for RHEL 7 (RPMs)                                        1,910
rhel-7-server-rh-common-rpms/7Server/x86_64                    Red Hat Enterprise Linux 7 Server - RH Common (RPMs)                                     231
rhel-7-server-rpms/7Server/x86_64                              Red Hat Enterprise Linux 7 Server (RPMs)                                              20,127
rhel-ha-for-rhel-7-server-rpms/7Server/x86_64                  Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs)                    468
repolist: 23,550
[stack@undercloud10 ~]$ sudo yum list updates                                                                                                              
Loaded plugins: product-id, search-disabled-repos, subscription-manager
[stack@undercloud10 ~]$ 


[stack@undercloud10 ~]$ openstack token issue
Certificate did not match expected hostname: 172.16.3.10. Certificate: {'subjectAltName': (('DNS', '172.16.3.10'),), 'notBefore': u'Jun  8 13:44:01 2017 GMT', 'serialNumber': u'D8FAEC41147248F09AB3D5D678B633FC', 'notAfter': 'Jun  8 13:43:56 2018 GMT', 'version': 3L, 'subject': ((('commonName', u'172.16.3.10'),),), 'issuer': ((('commonName', u'Local Signing Authority'),), (('commonName', u'd8faec41-147248f0-9ab3d5d6-78b633fb'),))}
SSL exception connecting to https://172.16.3.10:13000/v2.0/tokens: hostname '172.16.3.10' doesn't match '172.16.3.10'

# downgrade to previous version of python-backports-ssl_match_hostname

[stack@undercloud10 ~]$ sudo yum -y downgrade python-backports-ssl_match_hostname-3.4.0.2-4.el7.noarch                                                     
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package python-backports-ssl_match_hostname.noarch 0:3.4.0.2-4.el7 will be a downgrade
---> Package python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================================
 Package                                               Arch                     Version                         Repository                            Size
===========================================================================================================================================================
Downgrading:
 python-backports-ssl_match_hostname                   noarch                   3.4.0.2-4.el7                   rhel-7-server-rpms                    12 k

Transaction Summary
===========================================================================================================================================================
Downgrade  1 Package

Total download size: 12 k
Downloading packages:
python-backports-ssl_match_hostname-3.4.0.2-4.el7.noarch.rpm                                                                        |  12 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python-backports-ssl_match_hostname-3.4.0.2-4.el7.noarch                                                                                1/2 
  Cleanup    : python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch                                                                                2/2 
  Verifying  : python-backports-ssl_match_hostname-3.4.0.2-4.el7.noarch                                                                                1/2 
  Verifying  : python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch                                                                                2/2 

Removed:
  python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7                                                                                               

Installed:
  python-backports-ssl_match_hostname.noarch 0:3.4.0.2-4.el7                                                                                               

Complete!
[stack@undercloud10 ~]$ 

# and test again, it works

[stack@undercloud10 ~]$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2018-05-01 00:25:37+00:00        |
| id         | 1aa4dffa16aa4f7d9436542b6050288c |
| project_id | 07d75a6688ec43459caa1f6049e90e7a |
| user_id    | d08262252506426a8785cae385795c8c |
+------------+----------------------------------+




Version-Release number of selected component (if applicable):


How reproducible:
100% in my env.


Steps to Reproduce:
1. Patch OSP 10 env with SSL enabled in Undercloud.

My undercloud.conf

[DEFAULT]
local_ip = 172.16.3.1/24
undercloud_public_vip = 172.16.3.10
undercloud_admin_vip = 172.16.3.11
local_interface = eth0
masquerade_network = 172.16.3.0/24
dhcp_start = 172.16.3.20
dhcp_end = 172.16.3.40
network_cidr = 172.16.3.0/24
network_gateway = 172.16.3.254
inspection_iprange = 172.16.3.150,172.16.3.180
hieradata_override = /home/stack/custom_hiera.yaml

#ssl testing
generate_service_certificate = true
certificate_generation_ca = local

2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Matt Flusche 2018-04-30 20:34:55 UTC
debug output

[stack@undercloud10 ~]$ openstack --debug token issue
START with options: [u'--debug', u'token', u'issue']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', auth_type='', auth_url='http://172.16.3.1:5000/v2.0', authorization_code='', cacert=None, cert='', client_id='', client_secret='***', cloud='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, murano_url='', old_profile=None, openid_scope='', os_alarming_api_version='2', os_application_catalog_api_version='1', os_baremetal_api_version='1.15', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_dns_api_version='2', os_identity_api_version='', os_image_api_version='1', os_key_manager_api_version='1', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', roles='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.3.1:5000/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.15', 'queues_api_version': '2', 'auth': {'project_name': 'admin'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.3.1:5000/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.15', 'queues_api_version': '2', 'auth': {'username': 'admin', 'project_name': 'admin', 'password': '***', 'auth_url': 'http://172.16.3.1:5000/v2.0'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 1, cmd group openstack.image.v1
volume API version 2, cmd group openstack.volume.v2
identity API version 2.0, cmd group openstack.identity.v2
object_store API version 1, cmd group openstack.object_store.v1
messaging API version 2, cmd group openstack.messaging.v2
key_manager API version 1, cmd group openstack.key_manager.v1
alarming API version 2, cmd group openstack.alarming.v2
application_catalog API version 1, cmd group openstack.application_catalog.v1
container_infra API version 1, cmd group openstack.container_infra.v1
dns API version 2, cmd group openstack.dns.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
data_processing API version 1.1, cmd group openstack.data_processing.v1
metric API version 1, cmd group openstack.metric.v1
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
baremetal API version 1.15, cmd group openstack.baremetal.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
orchestration API version 1, cmd group openstack.orchestration.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.3.1:5000/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.15', 'queues_api_version': '2', 'auth': {'project_name': 'admin'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.3.1:5000/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.15', 'queues_api_version': '2', 'auth': {'project_name': 'admin'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
command: token issue -> openstackclient.identity.v2_0.token.IssueToken
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'password': '***', 'auth_url': 'http://172.16.3.1:5000/v2.0'}
Get auth_ref
REQ: curl -g -i -X GET http://172.16.3.1:5000/v2.0 -H "Accept: application/json" -H "User-Agent: osc-lib keystoneauth1/2.12.3 python-requests/2.11.1 CPython/2.7.5"
Starting new HTTP connection (1): 172.16.3.1
"GET /v2.0 HTTP/1.1" 200 232
RESP: [200] Date: Mon, 30 Apr 2018 20:33:49 GMT Server: Apache/2.4.6 (Red Hat Enterprise Linux) Vary: X-Auth-Token,Accept-Encoding x-openstack-request-id: req-d7803890-15ca-4a3a-8d9b-3afa53b1539c Content-Encoding: gzip Content-Length: 232 Connection: close Content-Type: application/json 
RESP BODY: {"version": {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://172.16.3.10:13000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

Making authentication request to https://172.16.3.10:13000/v2.0/tokens
Starting new HTTPS connection (1): 172.16.3.10
Certificate did not match expected hostname: 172.16.3.10. Certificate: {'subjectAltName': (('DNS', '172.16.3.10'),), 'notBefore': u'Jun  8 13:44:01 2017 GMT', 'serialNumber': u'D8FAEC41147248F09AB3D5D678B633FC', 'notAfter': 'Jun  8 13:43:56 2018 GMT', 'version': 3L, 'subject': ((('commonName', u'172.16.3.10'),),), 'issuer': ((('commonName', u'Local Signing Authority'),), (('commonName', u'd8faec41-147248f0-9ab3d5d6-78b633fb'),))}
SSL exception connecting to https://172.16.3.10:13000/v2.0/tokens: hostname '172.16.3.10' doesn't match '172.16.3.10'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 380, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 196, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 434, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 198, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 705, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 579, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 617, in _send_request
    raise exceptions.SSLError(msg)
SSLError: SSL exception connecting to https://172.16.3.10:13000/v2.0/tokens: hostname '172.16.3.10' doesn't match '172.16.3.10'
clean_up IssueToken: SSL exception connecting to https://172.16.3.10:13000/v2.0/tokens: hostname '172.16.3.10' doesn't match '172.16.3.10'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 267, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 380, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 196, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 434, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 198, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 705, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 579, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 617, in _send_request
    raise exceptions.SSLError(msg)
SSLError: SSL exception connecting to https://172.16.3.10:13000/v2.0/tokens: hostname '172.16.3.10' doesn't match '172.16.3.10'

END return value: 1

Comment 2 Matt Flusche 2018-04-30 20:39:01 UTC
The certificate is valid for the undercloud and openssl can validate it just fine.

[stack@undercloud10 ~]$ openssl s_client -connect 172.16.3.10:13000
CONNECTED(00000003)
depth=1 CN = Local Signing Authority, CN = d8faec41-147248f0-9ab3d5d6-78b633fb
verify return:1
depth=0 CN = 172.16.3.10
verify return:1
---
Certificate chain
 0 s:/CN=172.16.3.10
   i:/CN=Local Signing Authority/CN=d8faec41-147248f0-9ab3d5d6-78b633fb
 1 s:/CN=Local Signing Authority/CN=d8faec41-147248f0-9ab3d5d6-78b633fb
   i:/CN=Local Signing Authority/CN=d8faec41-147248f0-9ab3d5d6-78b633fb
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDXzCCAkegAwIBAgIRANj67EEUckjwmrPV1ni2M/wwDQYJKoZIhvcNAQELBQAw
UDEgMB4GA1UEAxMXTG9jYWwgU2lnbmluZyBBdXRob3JpdHkxLDAqBgNVBAMTI2Q4
ZmFlYzQxLTE0NzI0OGYwLTlhYjNkNWQ2LTc4YjYzM2ZiMB4XDTE3MDYwODEzNDQw
MVoXDTE4MDYwODEzNDM1NlowFjEUMBIGA1UEAxMLMTcyLjE2LjMuMTAwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCjYUala6WKceRTaSN6tMBPo2z9PdPw
SrkhacResTFYvE/rV8W+2FNZxxlc/RHaVci/rN0ml0KmpX2ywzmquynrvy/g1ILp
Xpmz6YMdDtESHPDwWUQjcbVYg8za+4Tuww8h33j1oLQyklGEisXuI7ZFc/ZxI6hi
lwP/QvKigqfqGEfS2ZMNuY0ijHEu6PAPnyDwCT0OErcMNB9OoYviHrTzEFjqPklE
DIysT4K7aEzR2fshI4PqINVj2sI6KzN/ffdTJcQSV7UDNU2Ylqm3ANA3yJrfj5vY
EbLDi2AH9wabxqZmTIDqbz3rbMNnIgDwxMZf+0jZ/1QTp9l//b/rcNR9AgMBAAGj
bjBsMBkGA1UdEQEBAAQPMA2CCzE3Mi4xNi4zLjEwMAwGA1UdEwEB/wQCMAAwIAYD
VR0OAQEABBYEFHJIC7AziS4UYvxcXTPWL+KlJ2QPMB8GA1UdIwQYMBaAFJoXQL8Z
Tbo23UehXX0DaQy1XlnyMA0GCSqGSIb3DQEBCwUAA4IBAQAmB/G/lf8Oq1GGXGhY
Itf59xKA1AS6EMEE7CLtKlkZBqj8N+sdK/cgfE5ogcR+vNrdKvmskiImrKU2B6kJ
u7nSZDl06WxOJKgObQnYwtnD4xj5xZ9AWtm/TCQeaA+Ger3HmC3F/9vEQBiqqY4t
1Alc9dx0TFvEXENtXaR05tU6ENGX5AZcS8IgfFtUXaGnZsWbyGjjOe6f3k2wOYyU
jUKVIqsXfxHdzGE6+jvVdv7gboo/R+JIgYXCpv2SmVG/qXwL2DX3DEuibxYg3VuZ
Kf5Zjpm633cfnrd+es+pX9Tj8faq2uC0nfu08jyBbaIAefe6lu45qGTQ8Vn5Swmh
kAUZ
-----END CERTIFICATE-----
subject=/CN=172.16.3.10
issuer=/CN=Local Signing Authority/CN=d8faec41-147248f0-9ab3d5d6-78b633fb
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2443 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 6F76DDE6324D79209A95FC17662D2912A38AE06E0E7824D2D72BF84C5684A3B8
    Session-ID-ctx: 
    Master-Key: A75A40BFFBE1486C5D8F1AFE615236554DFD04A026858B1C2ACD60FCAA16A17AEABEF378BB0043C806B7B3DD8B9B1FCE
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 9d 5f 4b b9 3b 69 2b 42-c5 45 df d2 0b 5e 29 16   ._K.;i+B.E...^).
    0010 - d5 25 a7 78 f2 4e db b8-84 04 f4 d6 26 4f 07 ce   .%.x.N......&O..
    0020 - 1e d7 00 f2 ed ec 8c 7f-ca fe 5f 9c 24 73 b0 9d   .........._.$s..
    0030 - cc 0a ea 17 13 84 25 ac-af 77 eb 70 8e 7f 5e f1   ......%..w.p..^.
    0040 - bc 0d 76 36 5b ad 6b 2d-cd ad bf b0 8d f2 33 83   ..v6[.k-......3.
    0050 - 40 fd 99 3a 15 f8 d4 57-d9 97 d9 b5 5c 6a 5f c2   @..:...W....\j_.
    0060 - cf 53 96 0f ab dc 29 c1-0c 48 14 76 60 89 ca c2   .S....)..H.v`...
    0070 - 1a 88 df 97 33 d0 95 e4-e8 e4 7f 2e df 3e e8 4d   ....3........>.M
    0080 - 26 3d a8 a2 00 4d f9 50-72 76 97 a2 ed bf d8 92   &=...M.Prv......
    0090 - 5e 69 37 9d 30 2d 8d ef-90 41 f3 80 98 5d 6e 6b   ^i7.0-...A...]nk

    Start Time: 1525120537
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---


[stack@undercloud10 ~]$ openssl x509 -noout -text
-----BEGIN CERTIFICATE-----
MIIDXzCCAkegAwIBAgIRANj67EEUckjwmrPV1ni2M/wwDQYJKoZIhvcNAQELBQAw
UDEgMB4GA1UEAxMXTG9jYWwgU2lnbmluZyBBdXRob3JpdHkxLDAqBgNVBAMTI2Q4
ZmFlYzQxLTE0NzI0OGYwLTlhYjNkNWQ2LTc4YjYzM2ZiMB4XDTE3MDYwODEzNDQw
MVoXDTE4MDYwODEzNDM1NlowFjEUMBIGA1UEAxMLMTcyLjE2LjMuMTAwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCjYUala6WKceRTaSN6tMBPo2z9PdPw
SrkhacResTFYvE/rV8W+2FNZxxlc/RHaVci/rN0ml0KmpX2ywzmquynrvy/g1ILp
Xpmz6YMdDtESHPDwWUQjcbVYg8za+4Tuww8h33j1oLQyklGEisXuI7ZFc/ZxI6hi
lwP/QvKigqfqGEfS2ZMNuY0ijHEu6PAPnyDwCT0OErcMNB9OoYviHrTzEFjqPklE
DIysT4K7aEzR2fshI4PqINVj2sI6KzN/ffdTJcQSV7UDNU2Ylqm3ANA3yJrfj5vY
EbLDi2AH9wabxqZmTIDqbz3rbMNnIgDwxMZf+0jZ/1QTp9l//b/rcNR9AgMBAAGj
bjBsMBkGA1UdEQEBAAQPMA2CCzE3Mi4xNi4zLjEwMAwGA1UdEwEB/wQCMAAwIAYD
VR0OAQEABBYEFHJIC7AziS4UYvxcXTPWL+KlJ2QPMB8GA1UdIwQYMBaAFJoXQL8Z
Tbo23UehXX0DaQy1XlnyMA0GCSqGSIb3DQEBCwUAA4IBAQAmB/G/lf8Oq1GGXGhY
Itf59xKA1AS6EMEE7CLtKlkZBqj8N+sdK/cgfE5ogcR+vNrdKvmskiImrKU2B6kJ
u7nSZDl06WxOJKgObQnYwtnD4xj5xZ9AWtm/TCQeaA+Ger3HmC3F/9vEQBiqqY4t
1Alc9dx0TFvEXENtXaR05tU6ENGX5AZcS8IgfFtUXaGnZsWbyGjjOe6f3k2wOYyU
jUKVIqsXfxHdzGE6+jvVdv7gboo/R+JIgYXCpv2SmVG/qXwL2DX3DEuibxYg3VuZ
Kf5Zjpm633cfnrd+es+pX9Tj8faq2uC0nfu08jyBbaIAefe6lu45qGTQ8Vn5Swmh
kAUZ
-----END CERTIFICATE-----
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            d8:fa:ec:41:14:72:48:f0:9a:b3:d5:d6:78:b6:33:fc
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Local Signing Authority, CN=d8faec41-147248f0-9ab3d5d6-78b633fb
        Validity
            Not Before: Jun  8 13:44:01 2017 GMT
            Not After : Jun  8 13:43:56 2018 GMT
        Subject: CN=172.16.3.10
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a3:61:46:a5:6b:a5:8a:71:e4:53:69:23:7a:b4:
                    c0:4f:a3:6c:fd:3d:d3:f0:4a:b9:21:69:c4:5e:b1:
                    31:58:bc:4f:eb:57:c5:be:d8:53:59:c7:19:5c:fd:
                    11:da:55:c8:bf:ac:dd:26:97:42:a6:a5:7d:b2:c3:
                    39:aa:bb:29:eb:bf:2f:e0:d4:82:e9:5e:99:b3:e9:
                    83:1d:0e:d1:12:1c:f0:f0:59:44:23:71:b5:58:83:
                    cc:da:fb:84:ee:c3:0f:21:df:78:f5:a0:b4:32:92:
                    51:84:8a:c5:ee:23:b6:45:73:f6:71:23:a8:62:97:
                    03:ff:42:f2:a2:82:a7:ea:18:47:d2:d9:93:0d:b9:
                    8d:22:8c:71:2e:e8:f0:0f:9f:20:f0:09:3d:0e:12:
                    b7:0c:34:1f:4e:a1:8b:e2:1e:b4:f3:10:58:ea:3e:
                    49:44:0c:8c:ac:4f:82:bb:68:4c:d1:d9:fb:21:23:
                    83:ea:20:d5:63:da:c2:3a:2b:33:7f:7d:f7:53:25:
                    c4:12:57:b5:03:35:4d:98:96:a9:b7:00:d0:37:c8:
                    9a:df:8f:9b:d8:11:b2:c3:8b:60:07:f7:06:9b:c6:
                    a6:66:4c:80:ea:6f:3d:eb:6c:c3:67:22:00:f0:c4:
                    c6:5f:fb:48:d9:ff:54:13:a7:d9:7f:fd:bf:eb:70:
                    d4:7d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:172.16.3.10
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                72:48:0B:B0:33:89:2E:14:62:FC:5C:5D:33:D6:2F:E2:A5:27:64:0F
            X509v3 Authority Key Identifier: 
                keyid:9A:17:40:BF:19:4D:BA:36:DD:47:A1:5D:7D:03:69:0C:B5:5E:59:F2

    Signature Algorithm: sha256WithRSAEncryption
         26:07:f1:bf:95:ff:0e:ab:51:86:5c:68:58:22:d7:f9:f7:12:
         80:d4:04:ba:10:c1:04:ec:22:ed:2a:59:19:06:a8:fc:37:eb:
         1d:2b:f7:20:7c:4e:68:81:c4:7e:bc:da:dd:2a:f9:ac:92:22:
         26:ac:a5:36:07:a9:09:bb:b9:d2:64:39:74:e9:6c:4e:24:a8:
         0e:6d:09:d8:c2:d9:c3:e3:18:f9:c5:9f:40:5a:d9:bf:4c:24:
         1e:68:0f:86:7a:bd:c7:98:2d:c5:ff:db:c4:40:18:aa:a9:8e:
         2d:d4:09:5c:f5:dc:74:4c:5b:c4:5c:43:6d:5d:a4:74:e6:d5:
         3a:10:d1:97:e4:06:5c:4b:c2:20:7c:5b:54:5d:a1:a7:66:c5:
         9b:c8:68:e3:39:ee:9f:de:4d:b0:39:8c:94:8d:42:95:22:ab:
         17:7f:11:dd:cc:61:3a:fa:3b:d5:76:fe:e0:6e:8a:3f:47:e2:
         48:81:85:c2:a6:fd:92:99:51:bf:a9:7c:0b:d8:35:f7:0c:4b:
         a2:6f:16:20:dd:5b:99:29:fe:59:8e:99:ba:df:77:1f:9e:b7:
         7e:7a:cf:a9:5f:d4:e3:f1:f6:aa:da:e0:b4:9d:fb:b4:f2:3c:
         81:6d:a2:00:79:f7:ba:96:ee:39:a8:64:d0:f1:59:f9:4b:09:
         a1:90:05:19

Comment 3 Matt Flusche 2018-05-01 17:51:03 UTC
It looks like python-backports-ssl_match_hostname-3.5.0.1-1.el7 will no longer validate an IP address used as a "DNS" subject alternative name as set up by the undercloud installation.

From version 3.5.0.1-1,  /usr/lib/python2.7/site-packages/backports/ssl_match_hostname/__init__.py

    122     dnsnames = []
    123     san = cert.get('subjectAltName', ())
    124     for key, value in san:
    125         if key == 'DNS':
    126             if host_ip is None and _dnsname_match(value, hostname):
    127                 return
    128             dnsnames.append(value)
    129         elif key == 'IP Address':
    130             if host_ip is not None and _ipaddress_match(value, host_ip):
    131                 return
    132             dnsnames.append(value)

Where the previous version doesn't care what is set as the "DNS" value.

From version 3.4.0.2-4:

     74     dnsnames = []
     75     san = cert.get('subjectAltName', ())
     76     for key, value in san:
     77         if key == 'DNS':
     78             if _dnsname_match(value, hostname):
     79                 return
     80             dnsnames.append(value)


And the tripleo puppet code set ups a cert with the IP address as the DNS SAN value.

[root@undercloud10 ~]# hiera tripleo::profile::base::haproxy::certificates_specs
{"undercloud-haproxy-public"=>
  {"service_pem"=>"/etc/pki/tls/certs/undercloud-172.16.3.10.pem",
   "service_certificate"=>"/etc/pki/tls/certs/undercloud-front.crt",
   "service_key"=>"/etc/pki/tls/private/undercloud-front.key",
   "hostname"=>"172.16.3.10",
   "postsave_cmd"=>
    "/usr/bin/instack-haproxy-cert-update '/etc/pki/tls/certs/undercloud-front.crt' '/etc/pki/tls/private/undercloud-front.key' /etc/pki/tls/certs/undercloud-172.16.3.10.pem",
   "principal"=>nil}}

[stack@undercloud10 ~]$ sudo openssl x509 -in /etc/pki/tls/certs/undercloud-172.16.3.10.pem  -text  |grep -A1 'Subject Alternative Name'
            X509v3 Subject Alternative Name: 
                DNS:172.16.3.10

Comment 4 Matt Flusche 2018-05-02 16:40:03 UTC
So this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1498196

Where the certificates are created by puppet-certmonger prior to version 1.1.1-2.1157a7egit.el7ost

The fix is to remove the old certs and allow them to be re-created.

As root:

  mkdir /etc/pki/tls/backup
  getcert stop-tracking -i undercloud-haproxy-public-cert
  mv /etc/pki/tls/certs/undercloud-front.crt /etc/pki/tls/private/undercloud-front.key /etc/pki/tls/certs/undercloud-*.pem  /etc/pki/tls/backup/

Then run undercloud upgrade again; it will fail again but will recreate the haproxy certs.

Restart haproxy:

  systemctl restart haproxy

Finally run the update process again and it should be successful.

After this the cert has the correct Subject Alt Name type.

 openssl x509 -in /etc/pki/tls/certs/undercloud-front.crt -text |grep -A1 'Subject Alternative Name'
            X509v3 Subject Alternative Name: 
                IP Address:172.16.3.10

Comment 6 Raildo Mascena de Sousa Filho 2018-05-28 14:18:24 UTC
Hi Matt, 

So based on your response in #comment4 looks like you have the necessary workaround for this issue. Do you think that we still need to keep with this bug opened? If so, do you have a suggestion to how to fix it based on your approach?

Comment 7 Matt Flusche 2018-05-29 13:39:25 UTC
Yes we can close this BZ; here is the related KB for our customers.

https://access.redhat.com/solutions/3430961


Note You need to log in before you can comment on or make changes to this bug.