Bug 1870728 - openshift-install creates expired ignition files from stale .openshift_install_state.json
Summary: openshift-install creates expired ignition files from stale .openshift_instal...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.7.0
Assignee: John Hixson
QA Contact: Gaoyun Pei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-20 17:19 UTC by rvanderp
Modified: 2024-02-04 00:07 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: installer does not check for expired certificates in ignition configuration Consequence: failed install Fix: check for expired certificates Result: installer prints out a warning if any certificates have expired
Clone Of:
Environment:
Last Closed: 2021-02-24 15:16:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4287 0 None closed bug 1870728: pkg/asset/ignition/bootstrap: Display warning if certificates are expired 2021-02-19 21:23:05 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:16:33 UTC

Description rvanderp 2020-08-20 17:19:16 UTC
Description of problem:
Unless the user is aware that .openshift_install_state.json exists, they often produce ignition files which are already expired due to stale certificates being pulled in from the file.  This results in failed installations.

Version-Release number of the following components:
4.5.5

How reproducible:
Consistently

Steps to Reproduce:
1. Create ignition files using the installer
2. Recreate ignition files and the certificates from the first creation are injected in to the ignition files
3.

Actual results:
Installer creates expired ignition content

Expected results:
Installer should warn the user about the expired content or fail to proceed with a warning

Additional info:

Checked certs befor and after with:
FILES=`jq '.storage.files[].path' bootstrap.ign -r | grep crt`
for FILE in $FILES; do echo $FILE;  jq ".storage.files[] | select(.path==\"$FILE\") | .contents.source" bootstrap.ign -r | sed "s/data\:text\/plain\;charset\=utf\-8;base64,//g" | base64 -d | openssl x509 -text -noout | grep "Not Before\: \| Not After \: \| $FILE"; done

Comment 1 Abhinav Dahiya 2020-08-20 17:22:35 UTC
This is the current known behaviour of the installer that it provides you the same assets from the state file as long as it exists.. it does not generate new ones until the inputs change.

Comment 2 rvanderp 2020-08-20 17:48:44 UTC
Understood, the problem is that users are unknowingly creating expired certs and not realizing it until after support has analyzed the installer-gather.  It is not common knowledge that the installer uses cached content.  It would seem that the installer could detect this condition and provide a warning.

Comment 3 Gaoyun Pei 2020-08-25 08:09:15 UTC
> It would seem that the installer could detect this condition and provide a warning.
Agree. A warning message is helpful for users to understand there maybe risk if old state file exists.
Hope we could get this addressed in 4.6, thanks.

Comment 5 Abhinav Dahiya 2020-08-27 18:11:28 UTC
(In reply to Gaoyun Pei from comment #3)
> > It would seem that the installer could detect this condition and provide a warning.
> Agree. A warning message is helpful for users to understand there maybe risk
> if old state file exists.
> Hope we could get this addressed in 4.6, thanks.

The current behaviour of the state file has been in place since the start and therefore we expect the users already have the right expectations. Changing the installer to allow validating and warning existing assets would need change in the core code-path and we can't fix it at this stage in 4.6 cycle. So we need to move it to 4.7 to work on it.

Comment 12 Scott Dodson 2020-09-09 20:29:11 UTC
This is by definition not a 4.6 blocker because it's not a regression and has been this way since GA. Moving to 4.7.0, we can evaluate backporting to 4.6.z once there's an agreed upon fix.

Comment 17 John Hixson 2020-10-21 03:05:38 UTC
PR: https://github.com/openshift/installer/pull/4287

Comment 19 Gaoyun Pei 2020-10-26 09:22:08 UTC
Verify this bug with openshift-install 4.7.0-0.nightly-2020-10-24-155529

Steps:
1. Create ignition files using the installer
./openshift-install create ignition-configs --dir gpei-01

2. Change the system time to several days later

3. Re-create ignition files in the same folder
# ./openshift-install create ignition-configs --dir gpei-01
WARNING Bootstrap Ignition-Config Certificate aggregator-ca.crt expired at 2020-10-27T08:07:29Z. 
WARNING Bootstrap Ignition-Config Certificate aggregator-ca-bundle.crt expired at 2020-10-27T08:07:29Z. 
WARNING Bootstrap Ignition-Config Certificate aggregator-client.crt expired at 2020-10-27T08:07:30Z. 
WARNING Bootstrap Ignition-Config Certificate aggregator-signer.crt expired at 2020-10-27T08:07:29Z. 
WARNING Bootstrap Ignition-Config Certificate apiserver-proxy.crt expired at 2020-10-27T08:07:30Z. 
WARNING Bootstrap Ignition-Config Certificate kube-apiserver-lb-server.crt expired at 2020-10-27T08:07:31Z. 
WARNING Bootstrap Ignition-Config Certificate kube-apiserver-internal-lb-server.crt expired at 2020-10-27T08:07:31Z. 
WARNING Bootstrap Ignition-Config Certificate kube-apiserver-localhost-server.crt expired at 2020-10-27T08:07:31Z. 
WARNING Bootstrap Ignition-Config Certificate kube-apiserver-service-network-server.crt expired at 2020-10-27T08:07:31Z. 
WARNING Bootstrap Ignition-Config Certificate kube-apiserver-complete-client-ca-bundle.crt expired at 2020-10-27T08:07:32Z. 
WARNING Bootstrap Ignition-Config Certificate kubelet-client-ca-bundle.crt expired at 2020-10-27T08:07:32Z. 
WARNING Bootstrap Ignition-Config Certificate kubelet-signer.crt expired at 2020-10-27T08:07:32Z. 
WARNING Bootstrap Ignition-Config Certificate kubelet-serving-ca-bundle.crt expired at 2020-10-27T08:07:32Z. 
WARNING Bootstrap Ignition-Config: 13 certificates expired. Installation attempts with the created Ignition-Configs will possibly fail. 
INFO Ignition-Configs created in: gpei-01 and gpei-01/auth

Comment 22 errata-xmlrpc 2021-02-24 15:16:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633

Comment 23 Simon Foley 2021-04-13 23:05:37 UTC
Thanks to whoever posted this. This is has and will catch people out causing a really bad experience of openshift.  
Its caching a dumb thing to do specially when certificates expire in 24hrs.
Whoever wrote the code that cashes content and certificates needs to be  supervised.
This is a basic 101 mistake.

Comment 24 Gerrit Slomma 2024-02-04 00:07:16 UTC
Nearly three years later:

# ./openshift-install --dir=ocp create single-node-ignition-config
WARNING Bootstrap Ignition-Config Certificate aggregator-ca.crt expired at 2024-02-03T23:23:03Z.
WARNING Bootstrap Ignition-Config Certificate aggregator-ca-bundle.crt expired at 2024-02-03T23:23:03Z.
WARNING Bootstrap Ignition-Config Certificate aggregator-client.crt expired at 2024-02-03T23:23:03Z.
WARNING Bootstrap Ignition-Config Certificate aggregator-signer.crt expired at 2024-02-03T23:23:03Z.
WARNING Bootstrap Ignition-Config Certificate apiserver-proxy.crt expired at 2024-02-03T23:23:03Z.
WARNING Bootstrap Ignition-Config Certificate kube-apiserver-localhost-server.crt expired at 2024-02-03T23:23:04Z.
WARNING Bootstrap Ignition-Config Certificate kube-apiserver-complete-client-ca-bundle.crt expired at 2024-02-03T23:23:04Z.
WARNING Bootstrap Ignition-Config Certificate kubelet-client-ca-bundle.crt expired at 2024-02-03T23:23:04Z.
WARNING Bootstrap Ignition-Config Certificate kubelet-signer.crt expired at 2024-02-03T23:23:04Z.
WARNING Bootstrap Ignition-Config Certificate kubelet-serving-ca-bundle.crt expired at 2024-02-03T23:23:04Z.
WARNING Bootstrap Ignition-Config: 10 certificates expired. Installation attempts with the created Ignition-Configs will possibly fail.
INFO Single-Node-Ignition-Config created in: ocp and ocp/auth

Followed instructions at https://docs.openshift.com/container-platform/4.14/installing/installing_sno/install-sno-installing-sno.html
Nothing came of this until yet, had four attempts that failed installation at various points between 550 and 600 packages of ~850 installed.
It is everything i feared it would be and i voiced in the DO280-seminar - overcomplex and hard to fix.


Note You need to log in before you can comment on or make changes to this bug.