Description of problem: When a CoreOS image needs to request ignition files from behind an endpoint (swift, webserver, etc) that is encrypted with a self-signed certificate it fails with an error. Version-Release number of the following components: /home/stack/src/github.com/openshift/installer/bin/openshift-install unreleased-master-1469-ge5b5d5efd1d43f5d9449efc798b7f8cde12bb0bb built from commit e5b5d5efd1d43f5d9449efc798b7f8cde12bb0bb release image registry.svc.ci.openshift.org/origin/release:4.2 How reproducible: Place ignition files behind an encrypted endpoint that uses a self-signed or internal CA. Steps to Reproduce: 1. deploy OCP with terraform 2. attempt to retrieve ignition configs from behind encrypted location that uses a self-signed cert 3. Actual results: certificate signed by unknown authority Here is a specific example from an OSP install using swift: [ 49.236893] ignition[609]: GET error: Get https://192.168.1.20:13808/v1/AUTH_3dd52f403c0c477ba541c54f4055ea90/rhte-tprhz/load-balancer.ign?temp_url_sig=1dae57439c0a8c55cd737ee8cee3dddeabfe0b6a&temp_url_expires=1564598715: x509: certificate signed by unknown authority Expected results: Retrieval of ignition files should be possible even when using encryption with a self-signed certificate Additional info:
ignition has docs on how to provide ca when requesting... see docshttp://coreos.com/ignition/docs/latest/
(In reply to Abhinav Dahiya from comment #1) > ignition has docs on how to provide ca when requesting... see > docshttp://coreos.com/ignition/docs/latest/ Can you point me where it shows this? It can't be IN ignition as this is BEFORE we grab it. Are you saying COreOS has a way to ASK for ignition files without a CA? Can you link me?
or to specify the CA. we can't seem to make it work for OpenStack.
https://github.com/coreos/ignition/blob/master/doc/configuration-v3_0.md
We can see where to provide the cacert file in the ignition file. However, this is in relation to the v4 openshift-installer. How do we get a CA file injected into ignition from the config.yml
reopening as OpenStack specific for now.
Turns out that OpenStack uses one additional ignition config that other platforms might not. In the terraform it is called `data "ignition_config" "redirect"` and is a shim ignition file, pushed via user_data, that then tries to pull the rest of the contents from an object store where the actual bootstrap.ign (and others) are stored. If this object store (accessed via http) is encrypted with tls and uses a self signed certificate the "shim/redirect" ignition file fails with a tls x.509 error because the CA is not "trusted". The installer needs to push the certificate provided by the `additionalTrustBundle` configuration attribute, similar to the way it does for bootstrap.ign, master.ign and node.ign. This is the part that's causing the bug.
Assign to Emilio for initial review.
The terraform ignition provider only supports ignition v2.1.0 for now, while we need at least ignition format v2.2.0. https://github.com/openshift/installer/issues/2225 describes the issue in detail. There's a feature request to support ignition v2.2.0 in the terraform ignition provider issue tracker at https://github.com/terraform-providers/terraform-provider-ignition/issues/42 with the beginning of a patch. It has been opened for almost a year and has no recent activity. It's unlikely this will make it for the 4.2 release.
Design Doc: https://github.com/openshift/enhancements/pull/72 Code: https://github.com/openshift/installer/pull/2544 Awaiting Approval and Review. On target to be delivered in 4.3
Do we have a workaround for this? I am installing OCP 4.2 on RH openstack13( self signed TLS) and hitting the same error.
There is not currently a workaround.
Verified on 4.3.0-0.nightly-2019-11-07-113138 [ 8.001872] ignition[1096]: GET http://169.254.169.254/openstack/latest/user_data: attempt #5 [ 9.181010] ignition[1096]: GET result: OK [ 9.204317] ignition[1096]: Adding "10.0.0.101" to list of CAs [ 9.208204] ignition[1096]: GET https://10.0.0.101:13808/v1/AUTH_19b91a0c7bd44aac9489948c255ee0be/morenod-ocp-mhxnq/72k6kgxfmz4c795q: attempt #1 [...] [ 54.123584] ignition[1096]: GET result: OK [K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (51s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (51s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (52s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (53s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (53s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (54s / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (54s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (55s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (55s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (56s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (56s / no limit)[ 60.198298] ignition[1096]: Adding "10.0.0.101" to list of CAs [ 60.202079] ignition[1096]: Adding "10.0.0.101" to list of CAs [K[[0;32m OK[ 60.206515] systemd[1]: Started Ignition (fetch). [0m] Started Ignition (fetch). Startin[ 60.211563] ignition[1096]: fetch: fetch complete
Sorry, reopening Ignition file is correctly downloaded, but machine-api controller cannot create workers I1107 15:47:57.510026 1 controller.go:164] Reconciling Machine "morenod-ocp-mhxnq-worker-xv7p4" I1107 15:47:57.510361 1 controller.go:376] Machine "morenod-ocp-mhxnq-worker-xv7p4" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster E1107 15:47:57.524429 1 controller.go:279] Failed to check if machine "morenod-ocp-mhxnq-worker-xv7p4" exists: Error checking if instance exists (machine/actuator.go 346): Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: Post https://10.0.0.101:13000//v3/auth/tokens: x509: certificate signed by unknown authority I
For clarification, this issue is resolved. Instances can all retrieve their ignition configs from services that use self signed certificates in their authentication. There is a further issue that needs to be resolved in order to fully support self signed certificates that is being tracked in the following bug https://bugzilla.redhat.com/show_bug.cgi?id=1769879. This is a separate technical issue and so we chose to track it independently.
If I understood this is solved in 4.3. But at the present moment the latest release that we can get at RH is 4.2.9. So I'll just leave my workaround here if anyone else can't wait for the next release. First I've tried adding "interface: internal" to clouds.yaml to point to the http endoints, but that was ignored. What worked was create a new region using the openstack CLI and copy all internal endpoints from the other region (RegionOne) to this one but as public endpoints. With that the terraform openstack provider stopped returning the https URL to the ignition file. This is not very secure, but in a test environment where you can't afford to create signed certificates it works.
This feature is now code-complete.
> Ideally we would've linked all the PRs that addressed this against this bug, the bot is smart enough to move the PR to modified once the last PR linked to the bug merges OK! Nice to know, thanks. I moved to ON_QA by request by David, I am then going to retroactively link the PRs like you suggested.
Verified ignition downloaded from SSL endpoint with self-signed on 4.4.0-0.nightly-2020-01-31-144949 Verified also that workers are correctly created Verified also that image-registry swift container is created Verified also worker scalation using machineset [ 11.258522] ignition[1075]: GET https://10.0.0.101:13292/v2/images/2780452b-a6df-41da-a3b8-e350c13b7d8f/file: attempt #1 [[0m[0;31m* [0m] A start job is running for Ignition (fetch) (11s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (12s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (12s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (13s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (13s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (14s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (15s / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (15s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (16s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (16s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (17s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (17s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (18s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (19s / no limit)[K[[0m[0;31m* [0m] A start job is running for Ignition (fetch) (19s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (20s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (20s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (21s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (22s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (22s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (23s / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (23s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (24s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (24s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (25s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (26s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (26s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (27s / no limit)[K[[0m[0;31m* [0m] A start job is running for Ignition (fetch) (27s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (28s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (29s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (29s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (30s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (30s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (31s / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (32s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (32s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (33s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (33s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (34s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (34s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (35s / no limit)[K[[0m[0;31m* [0m] A start job is running for Ignition (fetch) (36s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (36s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (37s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (37s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (38s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (39s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (39s / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (40s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (40s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (41s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (41s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (42s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (42s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (43s / no limit)[K[[0m[0;31m* [0m] A start job is running for Ignition (fetch) (43s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (44s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (44s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (45s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (45s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (46s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (46s / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (47s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (47s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (48s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (48s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (49s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (49s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (50s / no limit)[K[[0m[0;31m* [0m] A start job is running for Ignition (fetch) (50s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (51s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (51s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (52s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (52s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (53s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (53s / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (54s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (54s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (55s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (55s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (56s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (56s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (57s / no limit)[K[[0m[0;31m* [0m] A start job is running for Ignition (fetch) (57s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (58s / no limit)[ 61.262408] random: crng init done [ 61.267303] random: 7 urandom warning(s) missed due to ratelimiting [K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (58s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (59s / no limit)[ 62.733167] ignition[1075]: GET result: OK [K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (59s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (1min / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (1min / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (1min 1s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (1min 1s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (1min 2s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (1min 2s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (1min 3s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (1min 3s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (1min 4s / no limit)[K[[0m[0;31m* [0m] A start job is running for Ignition (fetch) (1min 4s / no limit)[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (1min 5s / no limit)[ 68.336746] ignition[1075]: Adding "192.168.24.2" to list of CAs [ 68.340338] ignition[1075]: Adding "192.168.24.2" to list of CAs [ 68.343766] ignition[1075]: fetch: fetch complete [K[[0;32m OK [0m] Started Ignition (fetch). [ 68.348433] systemd[1]: Started Ignition (fetch). Starting Check for FIPS mode... [ 68.352481] ignition[1075]: fetch: fetch passed [ 68.355295] systemd[1]: Starting Check for FIPS mode... [ 68.358383] ignition[1075]: Ignition finished successfully [ 68.407576] rhcos-fips[1213]: Found /etc/ignition-machine-config-encapsulated.json in Ignition config [ 68.413052] rhcos-fips[1213]: FIPS mode not requested [ 68.418686] systemd[1]: Started Check for FIPS mode. [[0;32m OK [0m] Started Check for FIPS mode. Starting Ignition (disks)... [ 68.425113] systemd[1]: Starting Ignition (disks)... [ 68.454513] ignition[1231]: Ignition 0.35.0 [ 68.457187] ignition[1231]: Stage: disks [ 68.459718] ignition[1231]: reading system config file "/usr/lib/ignition/base.ign" [ 68.480285] ignition[1231]: Adding "192.168.24.2" to list of CAs [ 68.483681] ignition[1231]: disks: disks passed [ 68.486469] ignition[1231]: Ignition finished successfully [ 68.489632] systemd[1]: Started Ignition (disks). [[0;32m OK [0m] Started Ignition (disks). Starting CoreOS Firstboot encryption of root device... [ 68.496423] systemd[1]: Starting CoreOS Firstboot encryption of root device... [ 68.576459] loop: module loaded [ 68.583300] coreos-cryptfs[1241]: coreos-cryptfs: Fetching clevis config [ 68.593805] coreos-cryptfs[1241]: coreos-cryptfs: No Clevis config provided [ 68.597732] coreos-cryptfs[1241]: coreos-cryptfs: No Clevis pin provided, encryption is not enabled [[0;32m OK [0m] Started CoreOS Firstboot encryption of root device. [ 68.613041] systemd[1]: Started CoreOS Firstboot encryption of root device. [ 68.616634] systemd[1]: Starting CoreOS LUKS Opener... Starting CoreOS LUKS Opener... [ 68.696894] coreos-cryptfs[1263]: coreos-cryptfs: /dev/vda4 uses the 'cipher_null-ecb' [ 68.701763] coreos-cryptfs[1263]: coreos-cryptfs: /dev/vda4 is not encrypted. Device will be mounted as device-mapper linear target. [ 69.017632] vda: vda1 vda2 vda3 vda4 [ 69.023420] coreos-cryptfs[1263]: CHANGED: partition=4 start=1050624 old: size=5937152 end=6987776 new: size=82835423,end=83886047 [ 69.061945] systemd[1]: Started CoreOS LUKS Opener. [[0;32m OK [0m] Started CoreOS LUKS Opener. [ 69.088261] systemd[1]: Starting Ignition OSTree: Mount (firstboot) /sysroot... Starting Ignition OSTree: Mount (firstboot) /sysroot... [ 69.234469] SGI XFS with ACLs, security attributes, no debug enabled [ 69.241964] XFS (dm-0): Mounting V5 Filesystem [ 69.613513] XFS (dm-0): Ending clean mount [ 69.616634] XFS (dm-0): Quotacheck needed: Please wait. [ 69.708289] XFS (dm-0): Quotacheck: Done. [[0;32m OK [[ 69.714682] systemd[1]: Started Ignition OSTree: Mount (firstboot) /sysroot. 0m] Started Ignition OSTree: Mount (firstboot) /sysroot. Startin[ 69.722795] systemd[1]: Starting OSTree Prepare OS/... g OSTree Prepare OS/... [ 69.740395] ostree-prepare-root[1350]: Resolved OSTree target to: /sysroot/ostree/deploy/rhcos/deploy/f61524fda480c611dcd25629fd15eb6de27a306689261c211dbc8e88c19a5219.0 [[0;32m OK [[ 69.751063] systemd[1]: Started OSTree Prepare OS/. 0m] Started OSTree Prepare OS/. [[0;32m OK [[ 69.754981] systemd[1]: Reached target Initrd Root File System. 0m] Reached target Initrd Root File System. Startin[ 69.760379] systemd[1]: Starting Mount OSTree /var... g Mount OSTree /var... [ 69.770309] ignition-ostree-mount-var[1353]: Mounting /sysroot/sysroot/ostree/deploy/rhcos/var [[0;32m OK [[ 69.776407] systemd[1]: Started Mount OSTree /var. 0m] Started Mount OSTree /var. Startin[ 69.781154] systemd[1]: Starting Ignition (mount) [stub]... g Ignition (mount) [stub]... [[0;32m OK [[ 69.785899] systemd[1]: Started Ignition (mount) [stub]. 0m] Started Ignition (mount) [stub]. [[0;32m OK [0m] Reached target Ignition Complete. [ 69.791701] systemd[1]: Reached target Ignition Complete. [stack@undercloud-0 morenod]$ openstack server list +--------------------------------------+--------------------------------+--------+-----------------------------------------------------+-------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------------------------+--------+-----------------------------------------------------+-------------------------+-----------+ | e47e3de6-b2b1-4833-90fc-60f69011ac2d | morenod-ssl-htrmd-worker-kckq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.20 | morenod-ssl-htrmd-rhcos | m4.xlarge | | c21436b6-fdf5-4305-b5a5-ae8040140123 | morenod-ssl-htrmd-worker-hqxx5 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.19 | morenod-ssl-htrmd-rhcos | m4.xlarge | | 2e4b2056-2292-42b4-8e45-e4dbd0f155e4 | morenod-ssl-htrmd-worker-2ssq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.17 | morenod-ssl-htrmd-rhcos | m4.xlarge | | 5003b464-f8ce-42d3-8198-fc78e0c5d08f | morenod-ssl-htrmd-master-0 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.21 | morenod-ssl-htrmd-rhcos | m4.xlarge | | e8f364f4-0464-41bc-8b10-3609f5066654 | morenod-ssl-htrmd-master-2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.16 | morenod-ssl-htrmd-rhcos | m4.xlarge | | b5313ee0-bbce-4772-986a-56250af47bcd | morenod-ssl-htrmd-master-1 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.24 | morenod-ssl-htrmd-rhcos | m4.xlarge | | c77af594-3540-46af-b517-a894cc80c9eb | morenod-ssl-htrmd-bootstrap | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.14, 10.0.0.202 | morenod-ssl-htrmd-rhcos | m4.xlarge | +--------------------------------------+--------------------------------+--------+-----------------------------------------------------+-------------------------+-----------+ $ openstack container list +----------------------------------------------------------------+ | Name | +----------------------------------------------------------------+ | morenod-ssl-htrmd-image-registry-amdmgtavlccstxdnxsaoyepwbeyqx | +----------------------------------------------------------------+ NAME STATUS ROLES AGE VERSION morenod-ssl-htrmd-master-0 Ready master 38m v1.17.1 morenod-ssl-htrmd-master-1 Ready master 38m v1.17.1 morenod-ssl-htrmd-master-2 Ready master 38m v1.17.1 morenod-ssl-htrmd-worker-2ssq2 Ready worker 16m v1.17.1 morenod-ssl-htrmd-worker-42c8w Ready worker 3m21s v1.17.1 morenod-ssl-htrmd-worker-hqxx5 Ready worker 16m v1.17.1 morenod-ssl-htrmd-worker-kckq2 Ready worker 17m v1.17.1 NAME PHASE TYPE REGION ZONE AGE morenod-ssl-htrmd-master-0 Running m4.xlarge nova 40m morenod-ssl-htrmd-master-1 Running m4.xlarge nova 40m morenod-ssl-htrmd-master-2 Running m4.xlarge nova 40m morenod-ssl-htrmd-worker-2ssq2 Running m4.xlarge nova 33m morenod-ssl-htrmd-worker-42c8w Running m4.xlarge nova 8m5s morenod-ssl-htrmd-worker-hqxx5 Running m4.xlarge nova 33m morenod-ssl-htrmd-worker-kckq2 Running m4.xlarge nova 33m [stack@undercloud-0 morenod]$ openstack server list +--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+ | 83c515da-2719-4251-8270-57c5b93f20da | morenod-ssl-htrmd-worker-42c8w | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.14 | morenod-ssl-htrmd-rhcos | m4.xlarge | | e47e3de6-b2b1-4833-90fc-60f69011ac2d | morenod-ssl-htrmd-worker-kckq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.20 | morenod-ssl-htrmd-rhcos | m4.xlarge | | c21436b6-fdf5-4305-b5a5-ae8040140123 | morenod-ssl-htrmd-worker-hqxx5 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.19 | morenod-ssl-htrmd-rhcos | m4.xlarge | | 2e4b2056-2292-42b4-8e45-e4dbd0f155e4 | morenod-ssl-htrmd-worker-2ssq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.17 | morenod-ssl-htrmd-rhcos | m4.xlarge | | 5003b464-f8ce-42d3-8198-fc78e0c5d08f | morenod-ssl-htrmd-master-0 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.21 | morenod-ssl-htrmd-rhcos | m4.xlarge | | e8f364f4-0464-41bc-8b10-3609f5066654 | morenod-ssl-htrmd-master-2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.16 | morenod-ssl-htrmd-rhcos | m4.xlarge | | b5313ee0-bbce-4772-986a-56250af47bcd | morenod-ssl-htrmd-master-1 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.24 | morenod-ssl-htrmd-rhcos | m4.xlarge | +--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+
Did the location of the tls bundle in install-config.yaml change? Previously it worked at .additionalTrustBundle. I know am back to seeing this on bootstrap node. `[ 281.539052] ignition[1067]: GET error: Get https://openstack.domain.com:13292/v2/images/98cf7fd6-e2d8-4279-83e6-e7de32f3793b/file: x509: certificate signed by unknown authority` Testing with openshift-install v4.4.0 built from commit e1a6a984b73a818a48f800eecc05ebbc526fc5b9 release image quay.io/openshift-release-dev/ocp-release-nightly@sha256:e1f4190d3a28efb01edb3a42aa138651cab813dc92e8aeef0d5be67fadd0f987
Forgive me. I found https://github.com/openshift/installer/commit/2d9ae15a28981a01a6c7503458725c6a14f0ea4a#diff-cf67b77422d61ae5593ac620f6e9b054 and learned to place the cacert in my clouds-public.yaml. I now see worker nodes being created!
I'm hitting this on OSP13z11 with a Certificate signed by my own CA (obviously, I'm not with Verisign so my CA chain needs to be added somewhere). 1) after preparing the OSP enviroment (in a similar fashion to [1] and [2]), I ran: ./openshift-install create cluster --log-level debug 2) Terraform starts and then hangs after a few minutes: DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused 3) In the console of the bootstrap node, I see this: 532.452676] ignition[724]: GET error: Get https://10.0.130.77:13808/swift/v1/AUTH_f7e33616a3fe42729523b5d296a42111/ocp4-m5fgd/2fspx766mqj86crv: x509: certificate signed by unknown authority [K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (8min 44s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (8min 45s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (8min 45s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (8min 46s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (8min 46s / no limit)[K[ [0;31m*[0m] A start job is running for Ignition (fetch) (8min 47s / no limit)[K[ [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (8min 47s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (8min 48s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (8min 48s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (8min 49s / no limit)[ 537.453694] ignition[724]: GET https://10.0.130.77:13808/swift/v1/AUTH_f7e33616a3fe42729523b5d296a42111/ocp4-m5fgd/2fspx766mqj86crv: attempt #108 [ 537.495145] ignition[724]: GET error: Get https://10.0.130.77:13808/swift/v1/AUTH_f7e33616a3fe42729523b5d296a42111/ocp4-m5fgd/2fspx766mqj86crv: x509: certificate signed by unknown authority The problem here is that I seem to be using OCP 4.3.9 so this shouldn't be happening, right? (overcloud) [stack@osp13p osd-ocp-demo]$ ./openshift-install version ./openshift-install 4.3.9 built from commit 64fccd954517812eab166d38c7fc5bf71b219b7e release image quay.io/openshift-release-dev/ocp-release@sha256:f0fada3c8216dc17affdd3375ff845b838ef9f3d67787d3d42a88dcd0f328eea [1] : https://developers.redhat.com/blog/2020/02/06/red-hat-openshift-4-2-ipi-on-openstack-13-all-in-one-setup/ [2] : https://kdjlab.com/openshift-4-2-on-red-hat-openstack-platform-13
https://bugzilla.redhat.com/show_bug.cgi?id=1735192#c48 was missing the cacert reference in clouds.yaml. Resolved when that was added and self signed cert worked.
Thank you August. Yes, the missing cacert was my issue. For the record, here's a copy of my clouds.yaml: clouds: openstack: auth: auth_url: https://<some ip>:13000/v3 username: "ocp-user" password: "<password>" project_id: f7e3[..digits...]42111 project_name: "ocp-tenant" user_domain_name: "Default" cacert: /home//stack/OSP/ca.crt.pem <====================================== region_name: "regionOne" interface: "public" identity_api_version: 3
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581