Back to bug 2182941

Who When What Removed Added
Red Hat One Jira (issues.redhat.com) 2023-03-30 01:49:30 UTC Link ID Red Hat Issue Tracker RHCEPH-6346
Brad Hubbard 2023-05-08 03:57:23 UTC Flags needinfo?(jefbrown)
CC jefbrown
Flags needinfo?(jefbrown)
Flags needinfo?(jefbrown)
Flags needinfo?(jefbrown)
Flags needinfo?(gmeno)
Flags needinfo?(ceph-eng-bugs)
Flags needinfo?(adking)
CC adking, bhubbard
Adam King 2023-05-08 14:27:52 UTC Assignee jefbrown adking
Flags needinfo?(jefbrown) needinfo?(jefbrown) needinfo?(jefbrown) needinfo?(jefbrown) needinfo?(gmeno) needinfo?(ceph-eng-bugs) needinfo?(adking)
QA Contact msaini mobisht
Component Ceph-Installer Cephadm
Adam King 2023-05-08 14:50:03 UTC Status NEW ASSIGNED
Adam King 2023-05-16 19:55:37 UTC Flags needinfo?(adking)
Flags needinfo?(adking)
Adam King 2023-05-18 13:23:25 UTC Flags needinfo?(csharpe)
Adam King 2023-06-02 17:24:56 UTC Flags needinfo?(csharpe)
Flags needinfo?(adking)
Flags needinfo?(adking) needinfo?(csharpe)
Adam King 2023-06-09 13:47:49 UTC Flags needinfo?(csharpe)
Flags needinfo?(adking)
Flags needinfo?(adking)
Adam King 2023-06-20 23:27:01 UTC Flags needinfo?(adking)
Flags needinfo?(adking)
Sayalee 2023-06-26 11:23:02 UTC CC saraut
Vivek Das 2023-06-30 06:04:23 UTC CC vdas
Adam King 2023-07-14 17:51:05 UTC Status ASSIGNED POST
errata-xmlrpc 2023-07-15 04:57:39 UTC Fixed In Version ceph-17.2.6-98.el9cp
Status POST MODIFIED
CC tserlin
Status MODIFIED ON_QA
Mohit Bisht 2023-07-17 09:31:54 UTC Flags needinfo?(adking)
Status ON_QA VERIFIED
Adam King 2023-07-17 12:38:56 UTC Flags needinfo?(adking)
Akash Raj 2023-07-18 11:04:03 UTC Flags needinfo?(adking)
CC akraj
Blocks 2221020
Docs Contact akraj
Adam King 2023-07-18 18:06:55 UTC Doc Type If docs needed, set a value Enhancement
Doc Text Feature: Cephadm support for CA signed keys

Reason: CA signed keys worked as a deployment setup in RHCS 5, although it working was accidental and untested and was broken in changes from RHCS 5 -> 6. In order to support those who had been using this setup, official support for it is being added here

Result: Users can now have cephadm use CA signed keys rather than typical pubkeys for an ssh authentication scheme. Some notes on how to use it:

Setting up CA signed keys for the cluster
-----------------------------------------

Cephadm also supports using CA signed keys for SSH authentication
across cluster nodes. In this setup, instead of needing a private
key and public key, we instead need a private key and certificate
created by signing that private key with a CA key. Once you have your private
key and signed cert, they can setup for cephadm to use by running:

ceph config-key set mgr/cephadm/ssh_identity_key -i <private-key-file>
ceph config-key set mgr/cephadm/ssh_identity_cert -i <signed-cert-file>

and for bootstrapping a new cluster with this setup


Deployment with CA signed SSH keys
----------------------------------

As an alternative to standard public key authentication, cephadm also supports
deployment using CA signed keys. Before bootstrapping it's recommended to set up
the CA public key as a trusted CA key on hosts you'd like to eventually add to
the cluster. For example:

# we will act as our own CA, therefore we'll need to make a CA key
[root@host1 ~]# ssh-keygen -t rsa -f ca-key -N ""

# make the ca key trusted on the host we've generated it on
# this requires adding in a line in our /etc/sshd_config
# to mark this key as trusted
[root@host1 ~]# cp ca-key.pub /etc/ssh
[root@host1 ~]# vi /etc/ssh/sshd_config
[root@host1 ~]# cat /etc/ssh/sshd_config | grep ca-key
TrustedUserCAKeys /etc/ssh/ca-key.pub
# now restart sshd so it picks up the config change
[root@host1 ~]# systemctl restart sshd

# now, on all other hosts we want in the cluster, also install the CA key
[root@host1 ~]# scp /etc/ssh/ca-key.pub host2:/etc/ssh/

# on other hosts, make the same changes to the sshd_config
[root@host2 ~]# vi /etc/ssh/sshd_config
[root@host2 ~]# cat /etc/ssh/sshd_config | grep ca-key
TrustedUserCAKeys /etc/ssh/ca-key.pub
# and restart sshd so it picks up the config change
[root@host2 ~]# systemctl restart sshd

Once the CA key has been installed and marked as a trusted key, you are ready
to use a private key/CA signed cert combination for SSH. Continuing with our
current example, we will create a new key-pair for for host access and then
sign it with our CA key


# make a new key pair
[root@host1 ~]# ssh-keygen -t rsa -f cephadm-ssh-key -N ""
# sign the private key. This will create a new cephadm-ssh-key-cert.pub
# note here we're using user "root". If you'd like to use a non-root
# user the arguments to the -I and -n params would need to be adjusted
# Additionally, note the -V param indicates how long until the cert
# this creates will expire
[root@host1 ~]# ssh-keygen -s ca-key -I user_root -n root -V +52w cephadm-ssh-key
[root@host1 ~]# ls
ca-key ca-key.pub cephadm-ssh-key cephadm-ssh-key-cert.pub cephadm-ssh-key.pub

# verify our signed key is working. To do this, make sure the generated private
# key ("cephadm-ssh-key" in our example) and the newly signed cert are stored
# in the same directory. Then try to ssh using the private key
[root@host1 ~]# ssh -i cephadm-ssh-key host2

Once you have your private key and corresponding CA signed cert and have tested
SSH authentication using that key works, you can pass those keys to bootstrap
in order to have cephadm use them for SSHing between cluster hosts


[root@host1 ~]# cephadm bootstrap --mon-ip <ip-addr> --ssh-private-key cephadm-ssh-key --ssh-signed-cert cephadm-ssh-key-cert.pub

Note that this setup does not require installing the corresponding public key
from the private key passed to bootstrap on other nodes. In fact, cephadm will
reject the ``--ssh-public-key`` argument when passed along with ``--ssh-signed-cert``.
Not because having the public key breaks anything, but because it is not at all needed
for this setup and it helps bootstrap differentiate if the user wants the CA signed
keys setup or standard pubkey encryption. What this means is, SSH key rotation
would simply be a matter of getting another key signed by the same CA and providing
cephadm with the new private key and signed cert. No additional distribution of
keys to cluster nodes is needed after the initial setup of the CA key as a trusted key,
no matter how many new private key/signed cert pairs are rotated in.
Flags needinfo?(adking)
Akash Raj 2023-07-25 04:41:01 UTC Doc Text Feature: Cephadm support for CA signed keys

Reason: CA signed keys worked as a deployment setup in RHCS 5, although it working was accidental and untested and was broken in changes from RHCS 5 -> 6. In order to support those who had been using this setup, official support for it is being added here

Result: Users can now have cephadm use CA signed keys rather than typical pubkeys for an ssh authentication scheme. Some notes on how to use it:

Setting up CA signed keys for the cluster
-----------------------------------------

Cephadm also supports using CA signed keys for SSH authentication
across cluster nodes. In this setup, instead of needing a private
key and public key, we instead need a private key and certificate
created by signing that private key with a CA key. Once you have your private
key and signed cert, they can setup for cephadm to use by running:

ceph config-key set mgr/cephadm/ssh_identity_key -i <private-key-file>
ceph config-key set mgr/cephadm/ssh_identity_cert -i <signed-cert-file>

and for bootstrapping a new cluster with this setup


Deployment with CA signed SSH keys
----------------------------------

As an alternative to standard public key authentication, cephadm also supports
deployment using CA signed keys. Before bootstrapping it's recommended to set up
the CA public key as a trusted CA key on hosts you'd like to eventually add to
the cluster. For example:

# we will act as our own CA, therefore we'll need to make a CA key
[root@host1 ~]# ssh-keygen -t rsa -f ca-key -N ""

# make the ca key trusted on the host we've generated it on
# this requires adding in a line in our /etc/sshd_config
# to mark this key as trusted
[root@host1 ~]# cp ca-key.pub /etc/ssh
[root@host1 ~]# vi /etc/ssh/sshd_config
[root@host1 ~]# cat /etc/ssh/sshd_config | grep ca-key
TrustedUserCAKeys /etc/ssh/ca-key.pub
# now restart sshd so it picks up the config change
[root@host1 ~]# systemctl restart sshd

# now, on all other hosts we want in the cluster, also install the CA key
[root@host1 ~]# scp /etc/ssh/ca-key.pub host2:/etc/ssh/

# on other hosts, make the same changes to the sshd_config
[root@host2 ~]# vi /etc/ssh/sshd_config
[root@host2 ~]# cat /etc/ssh/sshd_config | grep ca-key
TrustedUserCAKeys /etc/ssh/ca-key.pub
# and restart sshd so it picks up the config change
[root@host2 ~]# systemctl restart sshd

Once the CA key has been installed and marked as a trusted key, you are ready
to use a private key/CA signed cert combination for SSH. Continuing with our
current example, we will create a new key-pair for for host access and then
sign it with our CA key


# make a new key pair
[root@host1 ~]# ssh-keygen -t rsa -f cephadm-ssh-key -N ""
# sign the private key. This will create a new cephadm-ssh-key-cert.pub
# note here we're using user "root". If you'd like to use a non-root
# user the arguments to the -I and -n params would need to be adjusted
# Additionally, note the -V param indicates how long until the cert
# this creates will expire
[root@host1 ~]# ssh-keygen -s ca-key -I user_root -n root -V +52w cephadm-ssh-key
[root@host1 ~]# ls
ca-key ca-key.pub cephadm-ssh-key cephadm-ssh-key-cert.pub cephadm-ssh-key.pub

# verify our signed key is working. To do this, make sure the generated private
# key ("cephadm-ssh-key" in our example) and the newly signed cert are stored
# in the same directory. Then try to ssh using the private key
[root@host1 ~]# ssh -i cephadm-ssh-key host2

Once you have your private key and corresponding CA signed cert and have tested
SSH authentication using that key works, you can pass those keys to bootstrap
in order to have cephadm use them for SSHing between cluster hosts


[root@host1 ~]# cephadm bootstrap --mon-ip <ip-addr> --ssh-private-key cephadm-ssh-key --ssh-signed-cert cephadm-ssh-key-cert.pub

Note that this setup does not require installing the corresponding public key
from the private key passed to bootstrap on other nodes. In fact, cephadm will
reject the ``--ssh-public-key`` argument when passed along with ``--ssh-signed-cert``.
Not because having the public key breaks anything, but because it is not at all needed
for this setup and it helps bootstrap differentiate if the user wants the CA signed
keys setup or standard pubkey encryption. What this means is, SSH key rotation
would simply be a matter of getting another key signed by the same CA and providing
cephadm with the new private key and signed cert. No additional distribution of
keys to cluster nodes is needed after the initial setup of the CA key as a trusted key,
no matter how many new private key/signed cert pairs are rotated in. 		.`cephadm` support for CA signed keys is implemented

Previously, CA signed keys worked as a deployment setup in {storage-product} 5, although their working was accidental, untested, and broken in changes from {storage-product} 5 to {storage-product} 6.

With this enhancement, `cephadm` support for CA signed keys is implemented. Users can now use CA signed keys rather than typical _pubkeys_ for SSH authentication scheme.
errata-xmlrpc 2023-08-03 16:32:24 UTC Group redhat, ibm_storage
Status VERIFIED RELEASE_PENDING
errata-xmlrpc 2023-08-03 16:45:09 UTC Resolution --- ERRATA
Status RELEASE_PENDING CLOSED
Last Closed 2023-08-03 16:45:09 UTC
errata-xmlrpc 2023-08-03 16:46:18 UTC Link ID Red Hat Product Errata RHBA-2023:4473

Back to bug 2182941