Bug 1962414 - ed25519 keys do not work when FIPS is enabled
Summary: ed25519 keys do not work when FIPS is enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.7
Hardware: x86_64
OS: Linux
low
low
Target Milestone: ---
: 4.9.0
Assignee: Etienne Simard
QA Contact: Pedro Amoedo
URL:
Whiteboard:
Depends On:
Blocks: 1969244
TreeView+ depends on / blocked
 
Reported: 2021-05-20 02:07 UTC by Jatan Malde
Modified: 2021-10-18 17:31 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Installer did not output any error if using ed25519 SSH key type with FIPS enabled. Consequence: The ed25519 SSH key could not be used when configured in install-config.yaml with FIPS enabled. Fix: Validation of the SSH key type when FIPS is enabled in the install configuration. Result: Only FIPS approved SSH key types (rsa and ecdsa) are allowed by the installer when FIPS is enabled.
Clone Of:
: 1969244 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:31:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5029 0 None open Bug 1962414: FIPS: validate ssh public key type compatibility 2021-06-24 19:52:24 UTC
Red Hat Knowledge Base (Solution) 6063171 0 None None None 2021-05-20 08:10:25 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:31:31 UTC

Description Jatan Malde 2021-05-20 02:07:51 UTC
Version:

$ openshift-install version
 
 Customer was using 4.7.7 installer with 4.7.7rhcos live iso to boot the bootstrap node.

Platform:

baremetal UPI install

What happened?

Customer was attempting to install OpenShift 4.7.7 where the install-config included FIPS enabled and the user include ed25519 ssh keys using the command ssh-keygen -t ed25519 -N '' -f <path> and the public key was included in install-config.

 The bootstrap machine was configured with the bootstrap.ign file which had the public and the machine did boot up properly but the user was not able to ssh to the node using the private key as it reported permission denied.

 We disabled FIPS [0] in the install-config and asked for a fresh install for bootstrap node and it worked fine with ed25519 keys and the user was able to ssh to the node.

[0] https://access.redhat.com/solutions/3643252

What did you expect to happen?

It is expected that the installer reports a notification to the customer about FIPS being enabled when ed25519 keys are used. The user was not notified with any error and the install process went ahead which should have been restricted with the message like ed25519 keys are not supported when FIPS is enabled.

How to reproduce it (as minimally and precisely as possible)?

1. Pull ocp 4.7 installer with 4.7.7 rhcos live iso
2. create ssh ed25519 keys using ssh-keygen -t ed25519 -N '' -f <path> 
3. Enable FIPS to true in the install-config.yaml
4. bootup the rhcos node to check if ssh with the private key works or not.

Anything else we need to know?

Comment 3 Matthew Staebler 2021-06-08 04:10:24 UTC
The installation is successful. The only part that does not work is ssh access to the nodes.

I am cloning this to a docs BZ to handle documenting that only ecdsa and rsa ssh keys can be used when fips in enabled. This BZ will be kept to add validation in the installer for the same.

Comment 5 Jatan Malde 2021-06-10 18:14:14 UTC
Hello, 

I forgot to put in the docs bug link I opened the same day I pulled this bug,
https://bugzilla.redhat.com/show_bug.cgi?id=1962418

Just adding in here since I forgot to do it in the first place.

Thanks,
Jatan.

Comment 7 Matthew Staebler 2021-06-10 18:16:39 UTC
(In reply to Jatan Malde from comment #5)
> Hello, 
> 
> I forgot to put in the docs bug link I opened the same day I pulled this bug,
> https://bugzilla.redhat.com/show_bug.cgi?id=1962418
> 
> Just adding in here since I forgot to do it in the first place.
> 
> Thanks,
> Jatan.

Oh. I created a docs BZ too. I will close the one I created.

Comment 8 To Hung Sze 2021-06-21 17:59:36 UTC
Is this discovered on AWS?

ed25519 doesn't seem to work with GCP/Azure: https://bugzilla.redhat.com/show_bug.cgi?id=1968364

Comment 9 Matthew Staebler 2021-06-21 19:35:33 UTC
(In reply to To Hung Sze from comment #8)
> Is this discovered on AWS?
> 
> ed25519 doesn't seem to work with GCP/Azure:
> https://bugzilla.redhat.com/show_bug.cgi?id=1968364

The BZ that you linked is strictly for an Azure UPI installation. It is because the UPI ARM templates are attempting to add the ssh key to the VMs. This is not needed and is not done when doing an IPI installation. Are you seeing problems on GCP or with an Azure IPI installation?

Comment 10 To Hung Sze 2021-06-22 15:02:14 UTC
Yes, IPI Azure install fails.
ERROR Error: Error waiting for creation of Linux Virtual Machine "tszeaz062221-62pck-master-2" (Resource Group "tszeaz062221-62pck-rg"): Code="OSProvisioningTimedOut" Message="OS Provisioning for VM 'tszeaz062221-62pck-master-2' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. Also, make sure the image has been properly prepared (generalized).\r\n * Instructions for Windows: https://azure.microsoft.com/documentation/articles/virtual-machines-windows-upload-image/ \r\n * Instructions for Linux: https://azure.microsoft.com/documentation/articles/virtual-machines-linux-capture-image/ \r\n * If you are deploying more than 20 Virtual Machines concurrently, consider moving your custom image to shared image gallery. Please refer to https://aka.ms/movetosig for the same." 
ERROR                                              
ERROR   on ../../../../../tmp/openshift-install-721971274/master/master.tf line 84, in resource "azurerm_linux_virtual_machine" "master": 
ERROR   84: resource "azurerm_linux_virtual_machine" "master" { 
ERROR                                              
ERROR                                              
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

Comment 11 Matthew Staebler 2021-06-22 16:15:10 UTC
(In reply to To Hung Sze from comment #10)
> Yes, IPI Azure install fails.
> ERROR Error: Error waiting for creation of Linux Virtual Machine
> "tszeaz062221-62pck-master-2" (Resource Group "tszeaz062221-62pck-rg"):
> Code="OSProvisioningTimedOut" Message="OS Provisioning for VM
> 'tszeaz062221-62pck-master-2' did not finish in the allotted time. The VM
> may still finish provisioning successfully. Please check provisioning state
> later. Also, make sure the image has been properly prepared
> (generalized).\r\n * Instructions for Windows:
> https://azure.microsoft.com/documentation/articles/virtual-machines-windows-
> upload-image/ \r\n * Instructions for Linux:
> https://azure.microsoft.com/documentation/articles/virtual-machines-linux-
> capture-image/ \r\n * If you are deploying more than 20 Virtual Machines
> concurrently, consider moving your custom image to shared image gallery.
> Please refer to https://aka.ms/movetosig for the same." 
> ERROR                                              
> ERROR   on ../../../../../tmp/openshift-install-721971274/master/master.tf
> line 84, in resource "azurerm_linux_virtual_machine" "master": 
> ERROR   84: resource "azurerm_linux_virtual_machine" "master" { 
> ERROR                                              
> ERROR                                              
> FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to
> create cluster: failed to apply Terraform: failed to complete the change

@tsze I think that we are conflating two issues here. There is an issue with using ed25519 ssh keys in conjunction with fips. That configuration is not supported, independent of the platform used. The BZ that you linked is for using ed25519 ssh keys generally with an Azure UPI install, regardless of the fips mode.

Comment 12 To Hung Sze 2021-06-23 01:41:20 UTC
Yes, please ignore my comment above. Sorry.

Comment 16 Etienne Simard 2021-07-19 12:16:58 UTC
(In reply to To Hung Sze from comment #15)
> Together with this fix, we now validate that supplied key is properly
> formatted
> FATAL failed to fetch Master Machines: failed to load asset "Install
> Config": invalid "install-config.yaml" file: sshKey: Invalid value:
> "ssh-ed25519 AAAAC3Nza........2MChd7rH0a+fa1cIqTVjgiYAempLLW
> tsze.csb\n": ssh: no key found

This was an existing validation, but it also works with the new one.

Comment 19 errata-xmlrpc 2021-10-18 17:31:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.