Bug 1769879

Summary: [IPI][OSP] Machine-api cannot create workers on osp envs installed with self-signed certs
Product: OpenShift Container Platform Reporter: David Sanz <dsanzmor>
Component: InstallerAssignee: egarcia
Installer sub component: OpenShift on OpenStack QA Contact: David Sanz <dsanzmor>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: akaris, asimonel, dlbewley, egarcia, hfukumot, jappleii, joboyer, kholden, mifiedle, mtleilia, nsatsia, openshift-bugs-escalate, pprinett, racedoro, rheinzma, rhowe, rosingh, tsedovic, xtian
Version: 4.3.0Keywords: NeedsTestCase, TestBlocker
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1788072 (view as bug list) Environment:
Last Closed: 2020-05-13 21:52:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1735192, 1788072    

Description David Sanz 2019-11-07 16:29:19 UTC
Description of problem:
Machine-api cannot create workers on OSP installation with Self-signed certs

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-11-07-113138

How reproducible:

Steps to Reproduce:
1.Run installation of OCP on Openstack
2.Watch workers creation
3.oc logs -f pod/machine-api-controllers-xxxx-xxxx -c machine-controller

I1107 16:25:33.656976       1 controller.go:164] Reconciling Machine "morenod-ocp-mhxnq-worker-xv7p4"
I1107 16:25:33.657036       1 controller.go:376] Machine "morenod-ocp-mhxnq-worker-xv7p4" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
E1107 16:25:33.664586       1 controller.go:279] Failed to check if machine "morenod-ocp-mhxnq-worker-xv7p4" exists: Error checking if instance exists (machine/actuator.go 346): 
Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: Post https://10.0.0.101:13000//v3/auth/tokens: x509: certificate signed by unknown authority

Actual results:
Workers are not created

Expected results:
Workers are created

Additional info:

Comment 2 egarcia 2019-11-20 18:42:07 UTC
A proposed solution to CA bundle distribution within OpenShift clusters is being discussed here: https://github.com/openshift/enhancements/pull/115. Development cannot move forwards until this feature is implemented.

Comment 10 David Sanz 2020-01-31 16:42:40 UTC
Verified on 4.4.0-0.nightly-2020-01-31-144949

Workers are correctly created both by the installer and scaling using the machinest

NAME                             STATUS   ROLES    AGE   VERSION
morenod-ssl-htrmd-master-0	 Ready    master   43m   v1.17.1
morenod-ssl-htrmd-master-1	 Ready    master   43m   v1.17.1
morenod-ssl-htrmd-master-2	 Ready    master   43m   v1.17.1
morenod-ssl-htrmd-worker-hqxx5   Ready    worker   21m   v1.17.1
morenod-ssl-htrmd-worker-kckq2   Ready    worker   22m   v1.17.1
NAME                             PHASE     TYPE        REGION   ZONE   AGE
morenod-ssl-htrmd-master-0       Running   m4.xlarge            nova   45m
morenod-ssl-htrmd-master-1       Running   m4.xlarge            nova   45m
morenod-ssl-htrmd-master-2	 Running   m4.xlarge            nova   45m
morenod-ssl-htrmd-worker-hqxx5   Running   m4.xlarge            nova   38m
morenod-ssl-htrmd-worker-kckq2   Running   m4.xlarge            nova   38m
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
morenod-ssl-htrmd-worker   2         2         2       2           45m


[stack@undercloud-0 morenod]$ openstack server list
+--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+
| ID                                   | Name                           | Status | Networks                                | Image                   | Flavor    |
+--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+
| 5c979035-d762-4120-ba1e-ecb2c4ec6435 | morenod-ssl-htrmd-worker-tzgzn | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.13 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| 3b2fb4d5-5401-4025-9fbb-2e1640a5d4d2 | morenod-ssl-htrmd-worker-2kgbm | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.27 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| e47e3de6-b2b1-4833-90fc-60f69011ac2d | morenod-ssl-htrmd-worker-kckq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.20 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| c21436b6-fdf5-4305-b5a5-ae8040140123 | morenod-ssl-htrmd-worker-hqxx5 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.19 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| 5003b464-f8ce-42d3-8198-fc78e0c5d08f | morenod-ssl-htrmd-master-0     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.21 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| e8f364f4-0464-41bc-8b10-3609f5066654 | morenod-ssl-htrmd-master-2     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.16 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| b5313ee0-bbce-4772-986a-56250af47bcd | morenod-ssl-htrmd-master-1     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.24 | morenod-ssl-htrmd-rhcos | m4.xlarge |
+--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+

NAME                             STATUS   ROLES    AGE     VERSION
morenod-ssl-htrmd-master-0	 Ready    master   51m     v1.17.1
morenod-ssl-htrmd-master-1	 Ready    master   52m     v1.17.1
morenod-ssl-htrmd-master-2	 Ready    master   51m     v1.17.1
morenod-ssl-htrmd-worker-2kgbm   Ready    worker   3m47s   v1.17.1
morenod-ssl-htrmd-worker-hqxx5   Ready    worker   30m     v1.17.1
morenod-ssl-htrmd-worker-kckq2   Ready    worker   30m     v1.17.1
morenod-ssl-htrmd-worker-tzgzn   Ready    worker   2m47s   v1.17.1
NAME                             PHASE     TYPE        REGION   ZONE   AGE
morenod-ssl-htrmd-master-0	 Running   m4.xlarge            nova   53m
morenod-ssl-htrmd-master-1	 Running   m4.xlarge            nova   53m
morenod-ssl-htrmd-master-2	 Running   m4.xlarge            nova   53m
morenod-ssl-htrmd-worker-2kgbm   Running   m4.xlarge            nova   8m2s
morenod-ssl-htrmd-worker-hqxx5   Running   m4.xlarge            nova   46m
morenod-ssl-htrmd-worker-kckq2   Running   m4.xlarge            nova   46m
morenod-ssl-htrmd-worker-tzgzn   Running   m4.xlarge            nova   8m2s
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
morenod-ssl-htrmd-worker   4         4         4       4           53m

Comment 11 Scott Dodson 2020-02-03 18:19:37 UTC
All changes are in and should be moved to ON_QA

Comment 15 errata-xmlrpc 2020-05-13 21:52:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581