Bug 1609493

Summary:	Upgrade from OSp11 to OSp12 with self sign SSL is failing
Product:	Red Hat OpenStack	Reporter:	Nilesh <nchandek>
Component:	puppet-certmonger	Assignee:	John Dennis <jdennis>
Status:	CLOSED ERRATA	QA Contact:	Pavan <pkesavar>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	11.0 (Ocata)	CC:	bnemec, dhill, hrybacki, jagee, jdennis, jjoyce, josorior, jschluet, nchandek, nkinder, rmascena, rrasouli, slinaber, srevivo, tvignaud
Target Milestone:	z3	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)	Flags:	rmascena: needinfo+ rmascena: needinfo+ rmascena: needinfo+
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	puppet-certmonger-2.3.0-2.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-11-13 22:27:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nilesh 2018-07-28 15:13:49 UTC

Doing Upgrading the Undercloud Node is failing with error messages 


~~~
[root@dir ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)
[root@dir ~]# 
~~~

~~~
undercloud.conf
~~~

~~~
[DEFAULT]
local_interface = eth1
local_ip = 192.168.24.1/24
network_gateway = 192.168.24.1
undercloud_public_vip = 192.168.24.2
undercloud_admin_vip = 192.168.24.3
network_cidr = 192.168.24.0/24
masquerade_network = 192.168.24.0/24
dhcp_start = 192.168.24.5
dhcp_end = 192.168.24.24
inspection_iprange = 192.168.24.100,192.168.24.120
generate_service_certificate = true
certificate_generation_ca = local
~~~

~~~
2018-07-28 15:17:27,854 INFO: Created flavor "block-storage" with profile "block-storage"
2018-07-28 15:17:28,008 INFO: Created flavor "swift-storage" with profile "swift-storage"
2018-07-28 15:18:26,895 INFO: 
#############################################################################
Undercloud install complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and should be
secured.

#############################################################################
[stack@dir ~]$ 
~~~


~~~
[stack@dir ~]$ openstack catalog list 
+------------------+-------------------------+----------------------------------------------------------------------------------+
| Name             | Type                    | Endpoints                                                                        |
+------------------+-------------------------+----------------------------------------------------------------------------------+
| gnocchi          | metric                  | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13041                                          |
|                  |                         |   internalURL: http://192.168.24.3:8041                                          |
|                  |                         |   adminURL: http://192.168.24.3:8041                                             |
|                  |                         |                                                                                  |
| placement        | placement               | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13778/placement                                |
|                  |                         |   internalURL: http://192.168.24.3:8778/placement                                |
|                  |                         |   adminURL: http://192.168.24.3:8778/placement                                   |
|                  |                         |                                                                                  |
| neutron          | network                 | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13696                                          |
|                  |                         |   internalURL: http://192.168.24.3:9696                                          |
|                  |                         |   adminURL: http://192.168.24.3:9696                                             |
|                  |                         |                                                                                  |
| aodh             | alarming                | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13042                                          |
|                  |                         |   internalURL: http://192.168.24.3:8042                                          |
|                  |                         |   adminURL: http://192.168.24.3:8042                                             |
|                  |                         |                                                                                  |
| glance           | image                   | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13292                                          |
|                  |                         |   internalURL: http://192.168.24.3:9292                                          |
|                  |                         |   adminURL: http://192.168.24.3:9292                                             |
|                  |                         |                                                                                  |
| ceilometer       | metering                | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13777                                          |
|                  |                         |   internalURL: http://192.168.24.3:8777                                          |
|                  |                         |   adminURL: http://192.168.24.3:8777                                             |
|                  |                         |                                                                                  |
| heat-cfn         | cloudformation          | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13800/v1/0b1e4fd331804d50b64e543ae8733b5d      |
|                  |                         |   internalURL: http://192.168.24.3:8000/v1/0b1e4fd331804d50b64e543ae8733b5d      |
|                  |                         |   adminURL: http://192.168.24.3:8000/v1/0b1e4fd331804d50b64e543ae8733b5d         |
|                  |                         |                                                                                  |
| ironic           | baremetal               | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13385                                          |
|                  |                         |   internalURL: http://192.168.24.3:6385                                          |
|                  |                         |   adminURL: http://192.168.24.3:6385                                             |
|                  |                         |                                                                                  |
| nova             | compute                 | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13774/v2.1                                     |
|                  |                         |   internalURL: http://192.168.24.3:8774/v2.1                                     |
|                  |                         |   adminURL: http://192.168.24.3:8774/v2.1                                        |
|                  |                         |                                                                                  |
| zaqar-websocket  | messaging-websocket     | regionOne                                                                        |
|                  |                         |   publicURL: wss://192.168.24.2:9000                                             |
|                  |                         |   internalURL: ws://192.168.24.3:9000                                            |
|                  |                         |   adminURL: ws://192.168.24.3:9000                                               |
|                  |                         |                                                                                  |
| heat             | orchestration           | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13004/v1/0b1e4fd331804d50b64e543ae8733b5d      |
|                  |                         |   internalURL: http://192.168.24.3:8004/v1/0b1e4fd331804d50b64e543ae8733b5d      |
|                  |                         |   adminURL: http://192.168.24.3:8004/v1/0b1e4fd331804d50b64e543ae8733b5d         |
|                  |                         |                                                                                  |
| mistral          | workflowv2              | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13989/v2                                       |
|                  |                         |   internalURL: http://192.168.24.3:8989/v2                                       |
|                  |                         |   adminURL: http://192.168.24.3:8989/v2                                          |
|                  |                         |                                                                                  |
| swift            | object-store            | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13808/v1/AUTH_0b1e4fd331804d50b64e543ae8733b5d |
|                  |                         |   internalURL: http://192.168.24.3:8080/v1/AUTH_0b1e4fd331804d50b64e543ae8733b5d |
|                  |                         |   adminURL: http://192.168.24.3:8080                                             |
|                  |                         |                                                                                  |
| zaqar            | messaging               | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13888                                          |
|                  |                         |   internalURL: http://192.168.24.3:8888                                          |
|                  |                         |   adminURL: http://192.168.24.3:8888                                             |
|                  |                         |                                                                                  |
| ironic-inspector | baremetal-introspection | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13050                                          |
|                  |                         |   internalURL: http://192.168.24.3:5050                                          |
|                  |                         |   adminURL: http://192.168.24.3:5050                                             |
|                  |                         |                                                                                  |
| panko            | event                   | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13779                                          |
|                  |                         |   internalURL: http://192.168.24.3:8779                                          |
|                  |                         |   adminURL: http://192.168.24.3:8779                                             |
|                  |                         |                                                                                  |
| keystone         | identity                | regionOne                                                                        |
|                  |                         |   publicURL: https://192.168.24.2:13000/v2.0                                     |
|                  |                         |   internalURL: http://192.168.24.3:5000/v2.0                                     |
|                  |                         |   adminURL: http://192.168.24.3:35357/v2.0                                       |
|                  |                         |                                                                                  |
+------------------+-------------------------+----------------------------------------------------------------------------------+
[stack@dir ~]$ 
~~~



++++++++++++
MINOR UPDATE 
++++++++++++

~~~
2018-07-28 15:33:05,732 INFO: Not creating default plan "overcloud" because it already exists.
2018-07-28 15:33:06,769 INFO: 
#############################################################################
Undercloud upgrade complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and should be
secured.

#############################################################################

[stack@dir ~]$ 
~~~



[stack@dir ~]$ 
[stack@dir ~]$ 


[stack@dir ~]$ sudo openssl x509 -text -noout -in  /etc/pki/tls/certs/undercloud-192.168.24.2.pem | grep -i Alter -C 4
                    f7:c0:61:dd:42:91:56:61:31:b6:dd:27:98:ff:a1:
                    51:eb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:192.168.24.2
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Basic Constraints: critical
[stack@dir ~]$ 




[stack@dir ~]$ rpm -qi puppet-certmonger-1.1.1-1.1157a7egit.el7ost.noarch
Name        : puppet-certmonger
Version     : 1.1.1
Release     : 1.1157a7egit.el7ost
Architecture: noarch
Install Date: Sat 28 Jul 2018 02:28:13 PM EDT
Group       : Unspecified
Size        : 49470
License     : Apache-2.0
Signature   : RSA/SHA256, Tue 29 Nov 2016 12:16:00 PM EST, Key ID 199e2f91fd431d51
Source RPM  : puppet-certmonger-1.1.1-1.1157a7egit.el7ost.src.rpm
Build Date  : Thu 27 Oct 2016 03:52:35 PM EDT
Build Host  : x86-038.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/earsdown/puppet-certmonger
Summary     : Certmonger Puppet Module
Description :
Certmonger puppet module for integration with IPA CAs.
[stack@dir ~]$ 


++++++++++++
MAJOR UPDATE 
++++++++++++



* While following 

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html-single/upgrading_red_hat_openstack_platform/#upgrading_the_undercloud_node

* Run the following command to upgrade the undercloud:

$ openstack undercloud upgrade


* Is failing with below error messages, 


~~~
2018-07-28 16:23:51,252 INFO: Error: /Stage[main]/Heat::Keystone::Domain/Keystone_user[heat_admin::heat_stack]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: Certificate did not match expected hostname: 192.168.24.2. Certificate: {'subjectAltName': [('DNS', '192.168.24.2')], 'subject': ((('commonName', u'192.168.24.2'),),)}
2018-07-28 16:23:51,252 INFO: SSL exception connecting to https://192.168.24.2:13000/v3/auth/tokens: hostname '192.168.24.2' doesn't match '192.168.24.2' (tried 44, for a total of 170 seconds)
2018-07-28 16:23:51,253 INFO: Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[heat_admin::heat_stack@::heat_stack]: Dependency Keystone_user[heat_admin::heat_stack] has failures: true
2018-07-28 16:23:51,253 INFO: Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[heat_admin::heat_stack@::heat_stack]: Skipping because of failed dependencies
2018-07-28 16:23:53,575 INFO: Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]/public_url: public_url changed 'https://192.168.24.2:13000/v2.0' to 'https://192.168.24.2:13000'
2018-07-28 16:23:53,576 INFO: Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]/internal_url: internal_url changed 'http://192.168.24.3:5000/v2.0' to 'http://192.168.24.3:5000'
2018-07-28 16:23:53,576 INFO: Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]/admin_url: admin_url changed 'http://192.168.24.3:35357/v2.0' to 'http://192.168.24.3:35357'
~~~

Comment 5 Juan Antonio Osorio 2018-08-07 07:10:29 UTC

This requires a newer version of puppet-certmonger. The one that's being used there has a bug where it requests the certificate with the erroneous subjectAltName type (DNS always, and not IP). Having done this, you need to remove the certificate (and the certificate request) from the undercloud, and try again the undercloud install. It should do a new request with the correct parameters.

Comment 9 Raildo Mascena de Sousa Filho 2018-08-28 12:07:31 UTC

We need to backport these RDO change for the downstream side, also we need to check how RDO are building certmonger to see what is necessary to change the downstream as well.

Comment 18 errata-xmlrpc 2018-11-13 22:27:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3587