Bug 1786314 - [IPI][OSP] Install fails on OpenStack with self-signed certs unless the node running the installer has the CA cert in its system trusts
Summary: [IPI][OSP] Install fails on OpenStack with self-signed certs unless the node ...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.8.0
Assignee: Pierre Prinetti
QA Contact: Gaoyun Pei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-24 10:56 UTC by David Sanz
Modified: 2021-04-13 13:09 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The Installer did not support the `cacert` entry of clouds.yaml Consequence: In order to install on an OpenStack cloud that exposed its API over a non-trusted HTTPS certificate, it was required to install the certificate on the host that run the Installer. Fix: The Installer now supports using the `cacert` property of clouds.yaml. Result: It is possible to install by just adding the OpenStack HTTPS certificate in the `cacert` property of clouds.yaml, without the need for the entire machine to have it in its trustbundle.
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4486 0 None closed Bug 1786314: bump gophercloud/utils 2021-02-18 01:51:33 UTC
Github openshift installer pull 4561 0 None closed Bug 1786314: Bump dependencies 2021-02-18 01:51:33 UTC
Github openshift installer pull 4647 0 None closed Bug 1786314: openstack: cacert does not require system trust 2021-02-18 01:51:32 UTC

Description David Sanz 2019-12-24 10:56:12 UTC
Description of problem:

When installing OSP16 with self-signed certs, OSPd generates a clouds.yaml file with cacert parameter like:

clouds:
  overcloud:
    auth:
        auth_url: https://10.0.0.101:13000
        password: admin
        project_domain_name: Default
        project_name: admin
        user_domain_name: Default
        username: admin
    cacert: /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem
    identity_api_version: '3'

Using this clouds.yaml generated by OSP16 installer cannot be used to install OCP 4.3:

# ./openshift-install create cluster
FATAL failed to fetch Terraform Variables: failed to load asset "Install Config": invalid "install-config.yaml" file: [platform.openstack.externalNetwork: Internal error: could not retrieve valid networks, platform.openstack.computeFlavor: Internal error: could not retrieve valid flavors, platform.openstack.trunkSupport: Internal error: could not retrieve networking extension aliases, platform.openstack.octaviaSupport: Internal error: could not retrieve service catalog] 

If cacert file is imported on the system and cacert parameter is removed from the clouds.yaml file, installer works.

Version-Release number of the following components:
# ./openshift-install version
./openshift-install v4.3.0
built from commit 93c78d09ed9e2badb4bf5dab708152fe6b3b6602
release image registry.svc.ci.openshift.org/ocp/release@sha256:4cb1574385e4dfcb9342f4041eb674b3df51e31e6294ba433ce814c1b864dc86

How reproducible:

Steps to Reproduce:
1.Install OSP16 using OSPd with self-signed SSL
2.Get clouds.yaml file generated by OSPd and use it to install OCP

Actual results:
OCP is not installed but clouds.yaml is valid and can be used to execute openstack commands

Expected results:
OCP gets clouds.yaml file and checks cacert parameter


Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 2 Eric Duen 2020-01-03 21:10:11 UTC
David see comment #1 for question

Comment 6 Martin André 2020-02-04 14:20:17 UTC
This is currently documented as a known limitation in the upstream docs: https://github.com/openshift/installer/blob/8c55eae/docs/user/openstack/README.md#self-signed-openstack-ca-certificates

The workaround is to add the CA cert to the trusts of the node running the installer:

sudo cp ca.crt.pem /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract
 
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/director_installation_and_usage/appe-ssltls_certificate_configuration#Adding_the_Certificate_Authority_to_Clients

Comment 7 Martin André 2020-02-20 15:38:00 UTC
Re-targeting to 4.5, because it's not blocking 4.4.

Comment 8 Pierre Prinetti 2020-05-07 14:41:33 UTC
The team considers this bug as valid. Considering this bug priority and our capacity, we are deferring this bug to an upcoming sprint. If there are reasons for us to reprioritise, please let us know.

Comment 9 Pierre Prinetti 2020-05-14 14:25:50 UTC
Considering the priority assigned to this bug and our team capacity, we are deferring this bug to an upcoming sprint. Please let us know if there are reasons for us to reprioritize.

Comment 11 behnamsaberi@hotmail.com 2020-06-20 07:36:04 UTC
I need to install okd4.4-beta4 on openstack (VIO 5.1 vmware integrated openstack). However, when I try to install okd4.4 on openstack by IPI, I see the below error. 

FATAL failed to fetch Install Config: failed to fetch dependency of "Install Config": failed to fetch dependency of "Base Domain": failed to generate asset "Platform": Post https://x.x.x.x:5000/v3/auth/tokens: x509: certificate signed by unknown authority

I use this link and do step by step.
https://docs.openshift.com/container-platform/4.4/installing/installing_openstack/installing-openstack-user.html

please guide me to solve the issue.

Comment 12 Martin André 2020-06-25 14:06:04 UTC
The team considers this bug as valid. Considering this bug priority and our capacity, we are deferring this bug to an upcoming sprint. If there are reasons for us to reprioritise, please let us know.

Comment 21 Mike Fedosin 2020-12-08 12:15:58 UTC
The latest version of gophercloud/utils is able to work with self-signed certificates: https://github.com/gophercloud/utils/pull/131. The library itself has been bumped in the installer with https://github.com/openshift/installer/pull/4457. It means that we can set the status to ON_QA.

Comment 22 weiwei jiang 2020-12-11 07:20:06 UTC
Checked with 4.7.0-0.nightly-2020-12-09-112139, and it should contain the fix but still got failed with SSC OSP.

./openshift-install 4.7.0-0.nightly-2020-12-09-112139
built from commit 35d7aa255a6a849aab00d60b8c406a06d25c495c
release image registry.svc.ci.openshift.org/ocp/release@sha256:235c68dd2e120be1eb65ddeb747e0a2cd241de5405b55797576e0393e618e00e


# clouds.yaml
---
clouds:
  openstack:
    auth:
      auth_url: https://10.46.22.24:13000/v3
      username: shiftstack_user
      password: HIDDEN
      project_id: 8669733d329842049f4da0b3c2ca0ae0
      project_name: shiftstack
      user_domain_name: Default
    region_name: regionOne
    interface: public
    identity_api_version: 3
    cacert: "/home/jenkins/ws/workspace/Launch Environment Flexy/workdir/cacert.crt.20201211-381-1bsxzdh"

# install-config.yaml
---
apiVersion: v1
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3
metadata:
  name: wj47ios1211b
platform:
  openstack:
    cloud: openstack
    computeFlavor: m4.xlarge
    region: regionOne
    trunkSupport: '1'
    octaviaSupport: '0'
    apiFloatingIP: 10.46.22.37
    ingressFloatingIP: 10.46.22.59
    externalNetwork: nova
pullSecret: HIDDEN
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
  machineNetwork:
  - cidr: 192.168.0.0/18
  networkType: Kuryr
publish: External
baseDomain: 1211-hfm.qe.rhcloud.com
sshKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWkwurd8TNAi+D7ffvyDdhGBSQtJx3/Yedlwvvha0q772vLlOAGlKCw4dajKy6qty1/GGQDgTJ17h3C9TEArI8ZqILnyydeY56DL+ELN3dtGBVof/N2qtW0+SmEnd1Mi7Qy5Tx4e/GVmB3NgX9szwNOVXhebzgBsXc9x+RtCVLPLC8J+qqSdTUZ0UfJsh2ptlQLGHmmTpF//QlJ1tngvAFeCOxJUhrLAa37P9MtFsiNk31EfKyBk3eIdZljTERmqFaoJCohsFFEdO7tVgU6p5NwniAyBGZVjZBzjELoI1aZ+/g9yReIScxl1R6PWqEzcU6lGo2hInnb6nuZFGb+90D
  openshift-qe@redhat.com

# Generate manifests files.......
#./openshift-install create manifests --dir '/home/jenkins/ws/workspace/Launch Environment Flexy/workdir/install-dir'
level=fatal msg=failed to fetch Master Machines: failed to load asset "Install Config": failed to create a network client: Post "https://10.46.22.24:13000/v3/auth/tokens": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "192.168.24.2")

Comment 23 weiwei jiang 2020-12-11 07:35:36 UTC
when using curl, it work well: 

$ curl --cacert ./cacert.crt https://10.46.22.24:13000/v3/ -L -X GET -i 
HTTP/1.1 200 OK
Date: Fri, 11 Dec 2020 07:15:09 GMT
Server: Apache
Vary: X-Auth-Token,Accept-Encoding
x-openstack-request-id: req-7d37e856-dbfb-48e2-a808-f1dfc07a8ae6
Content-Length: 253
Content-Type: application/json

{"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "https://10.46.22.24:13000/v3/", "rel": "self"}]}}

Comment 27 Pierre Prinetti 2021-01-04 15:29:55 UTC
Given the low severity, we will tackle this bug on a best-effort basis. A complete fix might not land before 4.8.

Comment 29 Martin André 2021-01-12 10:13:52 UTC
We'll need to bump the gophercloud/utils dependency once https://github.com/gophercloud/utils/pull/140 merges.

Comment 30 Martin André 2021-01-12 10:18:26 UTC
(In reply to Martin André from comment #29)
> We'll need to bump the gophercloud/utils dependency once
> https://github.com/gophercloud/utils/pull/140 merges.

*And* update the user documentation to remove the instruction to have the system trust the CA cert:

https://github.com/openshift/installer/tree/master/docs/user/openstack#self-signed-openstack-ca-certificates

Comment 34 weiwei jiang 2021-02-22 10:34:27 UTC
Checked with 4.8.0-0.nightly-2021-02-21-102854 and it got fixed.


$ ./openshift-install-4.8 version
./openshift-install-4.8 4.8.0-0.nightly-2021-02-21-102854
built from commit 76838621d9ad64ee41bcda3a434c7282bcdb18a1
release image registry.ci.openshift.org/ocp/release@sha256:493bb3457443791e628be0d7262bf92771d65952686c1fa412e2c6aba672d9d9

$ curl --cacert kuryr13.crt https://10.46.22.24:13000/v3
{"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "https://10.46.22.24:13000/v3/", "rel": "self"}]}}%                  
$ curl https://10.46.22.24:13000/v3                      
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.


$ ./openshift-install-4.8 create cluster --dir bz1786314 --log-level debug
$ cat bz1786314/.openshift_install.log
time="2021-02-22T17:42:48+08:00" level=info msg="Credentials loaded from file \"/home/wjiang/osp_remover/clouds.yaml\""
time="2021-02-22T17:43:00+08:00" level=info msg="Consuming Install Config from target directory"
time="2021-02-22T17:43:35+08:00" level=info msg="Creating infrastructure resources..."
time="2021-02-22T17:46:16+08:00" level=info msg="Waiting up to 20m0s for the Kubernetes API at https://api.wj48ios222az.10.46.22.42.nip.io:6443..."
time="2021-02-22T17:48:38+08:00" level=info msg="API v1.20.0+01ab7fd up"
time="2021-02-22T17:48:38+08:00" level=info msg="Waiting up to 30m0s for bootstrapping to complete..."
time="2021-02-22T17:58:51+08:00" level=info msg="Destroying the bootstrap resources..."
time="2021-02-22T17:59:51+08:00" level=info msg="Waiting up to 40m0s for the cluster at https://api.wj48ios222az.10.46.22.42.nip.io:6443 to initialize..."
time="2021-02-22T18:20:40+08:00" level=info msg="Waiting up to 10m0s for the openshift-console route to be created..."
time="2021-02-22T18:20:41+08:00" level=info msg="Install complete!"
time="2021-02-22T18:20:41+08:00" level=info msg="To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/wjiang/osp_remover/bz1786314/auth/kubeconfig'"
time="2021-02-22T18:20:41+08:00" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.wj48ios222az.10.46.22.42.nip.io"
time="2021-02-22T18:20:41+08:00" level=info msg="Login to the console with user: \"kubeadmin\", and password: \"xxxx\""
time="2021-02-22T18:20:41+08:00" level=info msg="Time elapsed: 38m15s"

$ oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS                                                                                                                                                                                          
version   4.8.0-0.nightly-2021-02-21-102854   True        False         3m40s   Cluster version is 4.8.0-0.nightly-2021-02-21-102854                                                                                                                                            
$ oc get co                                                                                                                                                                                                                                                                   
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE                                                                                                                                                       
authentication                             4.8.0-0.nightly-2021-02-21-102854   True        False         False      6m27s                                                                                                                                                       
baremetal                                  4.8.0-0.nightly-2021-02-21-102854   True        False         False      30m                                                                                                                                                         
cloud-credential                           4.8.0-0.nightly-2021-02-21-102854   True        False         False      33m                                                                                                                                                         
cluster-autoscaler                         4.8.0-0.nightly-2021-02-21-102854   True        False         False      30m                                                                                                                                                         
config-operator                            4.8.0-0.nightly-2021-02-21-102854   True        False         False      31m                                                                                                                                                         
console                                    4.8.0-0.nightly-2021-02-21-102854   True        False         False      12m                                                                                                                                                         
csi-snapshot-controller                    4.8.0-0.nightly-2021-02-21-102854   True        False         False      30m                                                                                                                                                         
dns                                        4.8.0-0.nightly-2021-02-21-102854   True        False         False      30m                                                                                                                                                         
etcd                                       4.8.0-0.nightly-2021-02-21-102854   True        False         False      29m                                                                                                                                                         
image-registry                             4.8.0-0.nightly-2021-02-21-102854   True        False         False      17m                                                                                                                                                         
ingress                                    4.8.0-0.nightly-2021-02-21-102854   True        False         False      16m                                                                                                                                                         
insights                                   4.8.0-0.nightly-2021-02-21-102854   True        False         False      24m                                                                                                                                                         
kube-apiserver                             4.8.0-0.nightly-2021-02-21-102854   True        False         False      27m                                                                                                                                                         
kube-controller-manager                    4.8.0-0.nightly-2021-02-21-102854   True        False         False      28m                                                                                                                                                         
kube-scheduler                             4.8.0-0.nightly-2021-02-21-102854   True        False         False      28m                                                                                                                                                         
kube-storage-version-migrator              4.8.0-0.nightly-2021-02-21-102854   True        False         False      16m                                                                                                                                                         
machine-api                                4.8.0-0.nightly-2021-02-21-102854   True        False         False      26m                                                                                                                                                         
machine-approver                           4.8.0-0.nightly-2021-02-21-102854   True        False         False      30m                                                                                                                                                         
machine-config                             4.8.0-0.nightly-2021-02-21-102854   True        False         False      29m                                                                                                                                                         
marketplace                                4.8.0-0.nightly-2021-02-21-102854   True        False         False      29m                                                                                                                                                         
monitoring                                 4.8.0-0.nightly-2021-02-21-102854   True        False         False      14m                                                                                                                                                         
network                                    4.8.0-0.nightly-2021-02-21-102854   True        False         False      31m                                                                                                                                                         
node-tuning                                4.8.0-0.nightly-2021-02-21-102854   True        False         False      30m                                                                                                                                                         
openshift-apiserver                        4.8.0-0.nightly-2021-02-21-102854   True        False         False      22m                                                                                                                                                         
openshift-controller-manager               4.8.0-0.nightly-2021-02-21-102854   True        False         False      28m                                                                                                                                                         
openshift-samples                          4.8.0-0.nightly-2021-02-21-102854   True        False         False      22m                                                                                                                                                         
operator-lifecycle-manager                 4.8.0-0.nightly-2021-02-21-102854   True        False         False      30m                                                                                                                                                         
operator-lifecycle-manager-catalog         4.8.0-0.nightly-2021-02-21-102854   True        False         False      30m                                                                                                                                                         
operator-lifecycle-manager-packageserver   4.8.0-0.nightly-2021-02-21-102854   True        False         False      24m
service-ca                                 4.8.0-0.nightly-2021-02-21-102854   True        False         False      31m
storage                                    4.8.0-0.nightly-2021-02-21-102854   True        False         False      29m


Note You need to log in before you can comment on or make changes to this bug.