Version: $ openshift-install version Customer was using 4.7.7 installer with 4.7.7rhcos live iso to boot the bootstrap node. Platform: baremetal UPI install What happened? Customer was attempting to install OpenShift 4.7.7 where the install-config included FIPS enabled and the user include ed25519 ssh keys using the command ssh-keygen -t ed25519 -N '' -f <path> and the public key was included in install-config. The bootstrap machine was configured with the bootstrap.ign file which had the public and the machine did boot up properly but the user was not able to ssh to the node using the private key as it reported permission denied. We disabled FIPS [0] in the install-config and asked for a fresh install for bootstrap node and it worked fine with ed25519 keys and the user was able to ssh to the node. [0] https://access.redhat.com/solutions/3643252 What did you expect to happen? It is expected that the installer reports a notification to the customer about FIPS being enabled when ed25519 keys are used. The user was not notified with any error and the install process went ahead which should have been restricted with the message like ed25519 keys are not supported when FIPS is enabled. How to reproduce it (as minimally and precisely as possible)? 1. Pull ocp 4.7 installer with 4.7.7 rhcos live iso 2. create ssh ed25519 keys using ssh-keygen -t ed25519 -N '' -f <path> 3. Enable FIPS to true in the install-config.yaml 4. bootup the rhcos node to check if ssh with the private key works or not. Anything else we need to know?
The installation is successful. The only part that does not work is ssh access to the nodes. I am cloning this to a docs BZ to handle documenting that only ecdsa and rsa ssh keys can be used when fips in enabled. This BZ will be kept to add validation in the installer for the same.
Hello, I forgot to put in the docs bug link I opened the same day I pulled this bug, https://bugzilla.redhat.com/show_bug.cgi?id=1962418 Just adding in here since I forgot to do it in the first place. Thanks, Jatan.
(In reply to Jatan Malde from comment #5) > Hello, > > I forgot to put in the docs bug link I opened the same day I pulled this bug, > https://bugzilla.redhat.com/show_bug.cgi?id=1962418 > > Just adding in here since I forgot to do it in the first place. > > Thanks, > Jatan. Oh. I created a docs BZ too. I will close the one I created.
Is this discovered on AWS? ed25519 doesn't seem to work with GCP/Azure: https://bugzilla.redhat.com/show_bug.cgi?id=1968364
(In reply to To Hung Sze from comment #8) > Is this discovered on AWS? > > ed25519 doesn't seem to work with GCP/Azure: > https://bugzilla.redhat.com/show_bug.cgi?id=1968364 The BZ that you linked is strictly for an Azure UPI installation. It is because the UPI ARM templates are attempting to add the ssh key to the VMs. This is not needed and is not done when doing an IPI installation. Are you seeing problems on GCP or with an Azure IPI installation?
Yes, IPI Azure install fails. ERROR Error: Error waiting for creation of Linux Virtual Machine "tszeaz062221-62pck-master-2" (Resource Group "tszeaz062221-62pck-rg"): Code="OSProvisioningTimedOut" Message="OS Provisioning for VM 'tszeaz062221-62pck-master-2' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. Also, make sure the image has been properly prepared (generalized).\r\n * Instructions for Windows: https://azure.microsoft.com/documentation/articles/virtual-machines-windows-upload-image/ \r\n * Instructions for Linux: https://azure.microsoft.com/documentation/articles/virtual-machines-linux-capture-image/ \r\n * If you are deploying more than 20 Virtual Machines concurrently, consider moving your custom image to shared image gallery. Please refer to https://aka.ms/movetosig for the same." ERROR ERROR on ../../../../../tmp/openshift-install-721971274/master/master.tf line 84, in resource "azurerm_linux_virtual_machine" "master": ERROR 84: resource "azurerm_linux_virtual_machine" "master" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change
(In reply to To Hung Sze from comment #10) > Yes, IPI Azure install fails. > ERROR Error: Error waiting for creation of Linux Virtual Machine > "tszeaz062221-62pck-master-2" (Resource Group "tszeaz062221-62pck-rg"): > Code="OSProvisioningTimedOut" Message="OS Provisioning for VM > 'tszeaz062221-62pck-master-2' did not finish in the allotted time. The VM > may still finish provisioning successfully. Please check provisioning state > later. Also, make sure the image has been properly prepared > (generalized).\r\n * Instructions for Windows: > https://azure.microsoft.com/documentation/articles/virtual-machines-windows- > upload-image/ \r\n * Instructions for Linux: > https://azure.microsoft.com/documentation/articles/virtual-machines-linux- > capture-image/ \r\n * If you are deploying more than 20 Virtual Machines > concurrently, consider moving your custom image to shared image gallery. > Please refer to https://aka.ms/movetosig for the same." > ERROR > ERROR on ../../../../../tmp/openshift-install-721971274/master/master.tf > line 84, in resource "azurerm_linux_virtual_machine" "master": > ERROR 84: resource "azurerm_linux_virtual_machine" "master" { > ERROR > ERROR > FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to > create cluster: failed to apply Terraform: failed to complete the change @tsze I think that we are conflating two issues here. There is an issue with using ed25519 ssh keys in conjunction with fips. That configuration is not supported, independent of the platform used. The BZ that you linked is for using ed25519 ssh keys generally with an Azure UPI install, regardless of the fips mode.
Yes, please ignore my comment above. Sorry.
(In reply to To Hung Sze from comment #15) > Together with this fix, we now validate that supplied key is properly > formatted > FATAL failed to fetch Master Machines: failed to load asset "Install > Config": invalid "install-config.yaml" file: sshKey: Invalid value: > "ssh-ed25519 AAAAC3Nza........2MChd7rH0a+fa1cIqTVjgiYAempLLW > tsze.csb\n": ssh: no key found This was an existing validation, but it also works with the new one.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759