Bug 1783271 - [RFE] support for key rotation
Summary: [RFE] support for key rotation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Cephadm
Version: 3.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 6.1
Assignee: Adam King
QA Contact: Vinayak Papnoi
Akash Raj
URL:
Whiteboard:
: 1943506 (view as bug list)
Depends On: 2180567
Blocks: 2192813
TreeView+ depends on / blocked
 
Reported: 2019-12-13 13:45 UTC by John Fulton
Modified: 2024-05-14 06:26 UTC (History)
20 users (show)

Fixed In Version: ceph-17.2.6-5.el9cp
Doc Type: Enhancement
Doc Text:
.Users can now rotate the authentication key for Ceph daemons For security reasons, some users might desire to occasionally rotate the authentication key used for daemons in the storage cluster. With this release, the ability to rotate the authentication key for ceph daemons using the `ceph orch daemon rotate-key _DAEMON_NAME_` command is introduced. For MDS, OSD, and MGR daemons, this does not require a daemon restart. However, for other daemons, such as Ceph Object Gateway daemons, the daemon might require restarting to switch to the new key.
Clone Of:
Environment:
Last Closed: 2023-06-15 09:15:28 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 44869 0 None None None 2021-07-07 15:07:30 UTC
Github ceph ceph pull 48093 0 None open quincy: mon,auth,cephadm: support auth key rotation 2022-09-14 19:24:51 UTC
Red Hat Product Errata RHSA-2023:3623 0 None None None 2023-06-15 09:16:16 UTC

Description John Fulton 2019-12-13 13:45:12 UTC
This is a request for ceph-ansible to support the usecase described below.

If I want to change any Ceph security key, e.g "AQDC2UxZH4yeLhAAgTaZb+4wDUlYOsr1OfZSpQ==", of an existing keyring of a deployed Ceph cluster with minimal service interruption, then there is no playbook to do that. This is a request for such a playbook to be created.

A part of this request that should also be considered in scope is that if there are nodes in the Ceph client role and they have a key that is being changed with the new playbook, then their key should also be updated. E.g. if I want the key for the keyring ceph.client.openstack.keyring to be updated and the ansible Ceph client role configured an OpenStack Nova node with the same key, then the same playbook run should include updating the key on that node.

On the OpenStack side we acknowledge that changing the actual keyring on filesystem is not a complete solution because existing qemu guests on computes might be blocked from doing I/O when their token expires after the key is changed. So another BZ to track this aspect on the OpenStack side is required and would include the changing the token used by QEMU (this may not be possible without restarting the QEMU process). This other however would depend on this one. The combination of the two would allow a user to update the following in TripleO:

CephAdminKey
CephClientKey
CephManilaClientKey
CephMdsKey 
CephMonKey 
CephRgwKey

Comment 1 RHEL Program Management 2019-12-13 13:45:19 UTC
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 4 Federico Lucifredi 2020-04-07 19:36:32 UTC
candidate for 4.2.

Comment 10 Sebastian Wagner 2021-05-31 10:43:48 UTC
I think the desired behavior (as described in the BZ description) was implemented in https://github.com/ceph/ceph/pull/40941 . 

Please make sure you're not talking about https://tracker.ceph.com/issues/44869 here.

Comment 11 John Fulton 2021-06-09 12:05:01 UTC
It's more accurate to say that I'm talking about https://tracker.ceph.com/issues/44869.

Details:

The original description involved rotating the CephAdminKey (director's variable for the admin key) and which seems to be covered by [1] and not [2].

The original description also involved rotating client keys and I see that [2] gave us commands like `ceph orch client-keyring {ls,set,rm}` however, we're no longer using ceph-ansible to distribute client keys and we're not using cephadm to distribute client keys.

This RFE was requested in the context of OSP13/16, when ceph-ansible was controlling OpenStack cephx client keys, i.e. e.g. updating compute nodes ceph.conf and cephx keys. For OSP17/RHCSv5 director manages client keys [3] so the context is now different. Our process is now:

1. let cephadm create the admin key during bootstrap
2. use the ceph_key module to create OpenStack keys using the ceph_key module from ceph-ansible which is now in tripleo [4]
3. use the tripleo_ceph_client [3] to distribute the client cephx keys created in the previous step

Once we have [1] which rotates the CephAdminKey, we would follow a variation of the steps above for update, not create, letting [1] take care of step 1, and then use a variation of 2 and 3 to do the update. We could use [2] to implement step 2 above differently, but that's not sufficient to address the admin key rotation so we'd still need [1].


[1] https://tracker.ceph.com/issues/44869
[2] https://github.com/ceph/ceph/pull/40941
[3] https://docs.openstack.org/tripleo-ansible/latest/roles/role-tripleo_ceph_client.html
[4] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_cephadm/tasks/keys.yaml

Comment 14 Sebastian Wagner 2021-09-02 15:52:18 UTC
Neha, this needs a feature in RADOS to have two cephx keys for a brief period in time. Do you want to take it?

Comment 15 Neha Ojha 2021-09-09 19:27:59 UTC
(In reply to Sebastian Wagner from comment #14)
> Neha, this needs a feature in RADOS to have two cephx keys for a brief
> period in time. Do you want to take it?

Hi Sebastian,

IIRC, we discussed this at CDS and Sage added details in https://trello.com/c/dU24gHyD/302-automatic-key-rotation-for-daemons and here's the corresponding BZ https://bugzilla.redhat.com/show_bug.cgi?id=1943506. Is this what you are talking about?

Comment 16 Sebastian Wagner 2021-10-13 13:02:54 UTC
Moving to 5.2 as I don't think we can get this into 5.1 anymore

Comment 20 Federico Lucifredi 2022-08-30 21:24:45 UTC
*** Bug 1943506 has been marked as a duplicate of this bug. ***

Comment 24 Edith Stewart 2022-12-27 06:35:39 UTC Comment hidden (spam)
Comment 46 Mitchell 2023-05-25 11:18:18 UTC Comment hidden (spam)
Comment 47 errata-xmlrpc 2023-06-15 09:15:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 6.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:3623

Comment 48 AnjanetteRhymer 2023-07-31 09:40:24 UTC Comment hidden (spam)
Comment 49 carol1987grayson 2024-05-14 06:26:34 UTC Comment hidden (spam)

Note You need to log in before you can comment on or make changes to this bug.