Bug 2151002

Summary: Undercloud creating 1024- bit SSH Public/Private Key - Enable upgrades from 16.x to 17.1 (which requires 2048+ bit keys)
Product: Red Hat OpenStack Reporter: Jamie Fargen <jfargen>
Component: tripleo-ansibleAssignee: Andre <afariasa>
Status: CLOSED ERRATA QA Contact: Khomesh Thakre <kthakre>
Severity: high Docs Contact:
Priority: medium    
Version: 16.2 (Train)CC: afariasa, alee, arcsingh, bshephar, cjeanner, dwilde, ggrasza, hrybacki, jbadiapa, jjung, jpodivin, jpretori, jschluet, lbezdick, lsvaty, mabdulsa, mburns, mgarciac, parthee, pgrist, pnavarro, pweeks, rdiazcam
Target Milestone: gaKeywords: Triaged, UpgradeBlocker, Upgrades
Target Release: 17.1Flags: jpretori: needinfo? (pnavarro)
astillma: needinfo? (afariasa)
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: tripleo-ansible-3.3.1-1.20230518201535.el9ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-16 01:12:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2129048, 2129049, 2016660    

Description Jamie Fargen 2022-12-05 20:02:01 UTC
Description of problem:
The ssh public/private keypair on the undercloud is just 1024-bits. Some OSs like RHEL9.1 now only support a minimum size keypair size of 2048-bits and will present errors and access. In the case of running an instance and using the undercloud default public|private keypair access will not be possible.

Version-Release number of selected component (if applicable):
Openstack 16.2.3


How reproducible:
This is reproducible 100% of the time.

Steps to Reproduce:
1. Deploy openstack 16.2.3.
2. Add the undercloud keypair to add instances.
3. Deploy an RHEL9.1 and assign undercloud keypair.

Actual results:
$ ssh-keygen -l -f ~/.ssh/id_rsa.pub
1024 SHA256

Expected results:
Expect the undercloud to generate a more secure keypair. 


Additional info:
Because the crypto policy that comes with RHEL9.1 requires 2048-bit keypairs if you upgrade an instance to RHEL9.1 while using the undercloud default generated keypair you will lose ssh access to the instance(s). In the case where a password is not known this will require the instances to be booted into single user mode to regain access. 

I wanted to maintain access to instances deployed with the old keypair and newer instances built with a more secure key so I added  ecdsa keypair. This ecdsa keypair can live in addition to the 1024 RSA keypair. 
$ find /home/stack/.ssh/| grep 'dsa\|rsa'
/home/stack/.ssh/id_rsa
/home/stack/.ssh/id_rsa.pub
/home/stack/.ssh/id_ecdsa
/home/stack/.ssh/id_ecdsa.pub


The down side or possible risk is that both keys will be tried during login and though there is a risk of the wrong key being used being logged or incrementing a counter towards locking the account, in this example it would only be encountered on instances with an extremely low threshold of login attemps.

Comment 1 Jamie Fargen 2022-12-05 20:04:15 UTC
This is the command I used to generate the ecdsa key on the undercloud, it is the largest size ecdsa key that is possible to create and though less than 2048 bits seems to be accepable for RHEL9.1 crypto policy.
$ ssh-keygen -t ecdsa -b 521

Comment 2 Jamie Fargen 2022-12-05 20:06:32 UTC
Is this the code responsible for creating the keypair on the undercloud?
/usr/share/openstack-tripleo-heat-templates/extraconfig/post_deploy/undercloud_post.sh:    ssh-keygen -b 1024 -N '' -f $HOMEDIR/.ssh/id_rsa

Comment 3 Jamie Fargen 2022-12-07 13:17:30 UTC
When OSP17.1 is released, which will be based on RHEL9.1+, it seems that this could block access to the overcloud nodes via the heat-user. This seems like a foreseeable blocker that could be encountered in upgrades to OSP17.1 (considering it is based on RHEL9.1). So before upgrading to OSP17.1 I would recommend that deployments add a 521 bit ecdca keypair to the stack user on the undercloud and push out this ecdca public key to the overcloud compute and controllers and ensure they can access the systems using this key.

Comment 4 Jamie Fargen 2022-12-07 13:21:13 UTC
Small typo in comment #3, it is actuall an ecdsa keypair.

Comment 5 Brendan Shephard 2022-12-08 10:25:31 UTC
(In reply to Jamie Fargen from comment #3)
> When OSP17.1 is released, which will be based on RHEL9.1+, it seems that
> this could block access to the overcloud nodes via the heat-user. This seems
> like a foreseeable blocker that could be encountered in upgrades to OSP17.1
> (considering it is based on RHEL9.1). So before upgrading to OSP17.1 I would
> recommend that deployments add a 521 bit ecdca keypair to the stack user on
> the undercloud and push out this ecdca public key to the overcloud compute
> and controllers and ensure they can access the systems using this key.

The code you're referring to was changed in Wallaby which is what OSP17 is based on. So this won't have any impact on 17.
https://github.com/openstack/tripleo-heat-templates/commit/11a79ab8f75f7661285590d64981f10525535da2

1024 bit keys were still fine with the version of openssh used in RHEL8 weren't they? When it comes to using a key for instances you want to deploy on the overcloud, generating your own key as per your requirements seems reasonable. The key we're generating during the deployment is intended to manage the overcloud nodes themselves rather than cater for instances deployed on the overcloud.

If dfg:security want to change it to increase the size, I see no problems with it. Other than we need to be sure to handle it properly during updates to avoid being locked out. But provided 1024bit keys are still supported on RHEL8, I don't think this is a OpenStack bug per se.

Comment 6 Ade Lee 2022-12-08 10:57:46 UTC
Brendan, 

Thanks for pointing out where the keys are generated.  I was looking someplace else (tripleo-ansible).

I agree that this is not a problem for fresh installs on OSP 17, and I'm inclined to not specify a key size
so that the crypto modules will pick the right one instead.  This is relevant for example if FIPS is enabled.

They may, however, be a problem for upgrades to 17.1 because you could be locked out of your nodes when
you update to RHEL 9 for OSP 17.1.  In that sense, this is an OpenStack (or at least an upgrade bug).

You can't just rerun the script above without the -b option because those keys are the ones used to talk to the
overcloud.  Fortunately, the security DFG recently wrote a playbook to allow you to rotate your keys.
It looks like we need to make that playbook more readily available as a second day operation - and make sure
that it is called prior to upgrade.

Comment 32 Lukas Svaty 2023-06-08 11:07:21 UTC
Raising severity due to the blocker+

Comment 61 errata-xmlrpc 2023-08-16 01:12:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577