Bug 2274107

Summary:	Failure in Cluster-Wide Encryption Key Rotation for NooBaa Secret 'noobaa-root-master-key-volume'.
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Parag Kamble <pakamble>
Component:	Multi-Cloud Object Gateway	Assignee:	Jacky Albo <jalbo>
Status:	CLOSED ERRATA	QA Contact:	Tiffany Nguyen <tunguyen>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.16	CC:	jalbo, nbecker, odf-bz-bot
Target Milestone:	---
Target Release:	ODF 4.16.0
Hardware:	All
OS:	Unspecified
Whiteboard:
Fixed In Version:	4.16.0-75	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-07-17 13:18:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Parag Kamble 2024-04-09 07:48:29 UTC

Description of problem (please be detailed as possible and provide log
snippests):
The cluster-wide encryption key rotation process is not functioning as expected for the NooBaa secret named 'noobaa-root-master-key-volume' though its function for the secret 'noobaa-root-master-key-backend'.


Version of all relevant components (if applicable): 4.16


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible? Y


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Install an ODF 4.16 cluster with clusterwide encryption enabled, without a Key Management Service (KMS) configured.
2. Record the existing keys from the following NooBaa secrets:
   - `noobaa-root-master-key-backend`
   - `noobaa-root-master-key-volume`
3. Change the default key rotation period from 'weekly' to 'every 5 minutes' using a cron job in the storage cluster specification.
4. Wait for the key rotation to occur according to the new cron job schedule (every 5 minutes).
5. Compare the keys retrieved from the NooBaa secrets after key rotation with the recorded keys from step 2.
6. Confirm that the keys have rotated successfully in both `noobaa-root-master-key-backend` and `noobaa-root-master-key-volume` secrets.



Actual results:
Keys from the noobaa secret `noobaa-root-master-key-volume` has not been rotated.

Expected results:

Keys rotation should happen for noobaa secrets.
   - `noobaa-root-master-key-backend`
   - `noobaa-root-master-key-volume` 

Additional info:


Cluster Details
-=-=-=-=-=-=-=
> ocs get csv
NAME                                        DISPLAY                            VERSION            REPLACES   PHASE
mcg-operator.v4.16.0-69.stable              NooBaa Operator                    4.16.0-69.stable              Succeeded
ocs-client-operator.v4.16.0-69.stable       OpenShift Data Foundation Client   4.16.0-69.stable              Succeeded
ocs-operator.v4.16.0-69.stable              OpenShift Container Storage        4.16.0-69.stable              Succeeded
odf-csi-addons-operator.v4.16.0-69.stable   CSI Addons                         4.16.0-69.stable              Succeeded
odf-operator.v4.16.0-69.stable              OpenShift Data Foundation          4.16.0-69.stable              Succeeded
odf-prometheus-operator.v4.16.0-69.stable   Prometheus Operator                4.16.0-69.stable              Succeeded
rook-ceph-operator.v4.16.0-69.stable        Rook-Ceph                          4.16.0-69.stable              Succeeded


Storagecluster keyrotation configuration
=====================
> ocs get storageclusters.ocs.openshift.io -ojsonpath='{.items[*].spec.encryption}'
{"clusterWide":true,"keyRotation":{"schedule":"*/5 * * * *"},"kms":{}}

Rook cephcluster keyrotation configuration
===================
> ocs get cephclusters.ceph.rook.io -o jsonpath='{.items[].spec.security}'
{"keyRotation":{"enabled":true,"schedule":"*/5 * * * *"},"kms":{}}

Noobaa keyrotation configuration
=====================
> ocs get noobaas.noobaa.io -o jsonpath='{.items[*].spec.security}'
{"kms":{"enableKeyRotation":true,"schedule":"*/5 * * * *"}}

NooBaa Secrets
-=-=-=-=-=-=-=-=-=-=-=
> ocs get secret noobaa-root-master-key-volume -o yaml
apiVersion: v1
data:
  active_root_key: a2V5LTE3MTI1NjM0NDkwMzE1NjY1MDI=
  key-1712563449031566502: UjRnUU14bGlNd0lDY3NwTkpnQi9zYWpXeDkvNnp5RnhWRTZEYmJ0Vis5Yz0=
kind: Secret
metadata:
  creationTimestamp: "2024-04-08T08:04:09Z"
  labels:
    app: noobaa
  name: noobaa-root-master-key-volume
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: noobaa.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: NooBaa
    name: noobaa
    uid: 6fb11e22-4f9d-4a89-bd5a-1c56b2d3d486
  resourceVersion: "44439"
  uid: 0f02a2b2-21ea-4415-b3d0-1ca6ca15fffb
type: Opaque


-------------------------------------------
> ocs get secret noobaa-root-master-key-backend -o yaml
apiVersion: v1
data:
  active_root_key: a2V5LTE3MTI2MjA4MDI3NTk3ODkyNzM=
  key-1712563449031566502: UjRnUU14bGlNd0lDY3NwTkpnQi9zYWpXeDkvNnp5RnhWRTZEYmJ0Vis5Yz0=
  key-1712600806071647216: SXlrcUgxNDZYVXErenYxTThDWjRmVkswa3Y3VUQwa0RqN25BZkFHVUVUZz0=
  key-1712601001965921492: SVMxcHNmTmUzUERjQU84eE1DQWJyOUg0eSs5WDRuRk9HT09MZjRrZkZZdz0=
  key-1712601302190304475: YjZveDd5TU5UOXJzM0VGcm4yWWNaZGxUQjJaQlo0M0tpdzBDaXdLMUxEYz0=
  key-1712601600334743630: UUpUbjBVb2FvandidEVOckFUWERWNHdHTWNWZFZqaFlYaUlMcHlrN3pXdz0=
  key-1712601902660350482: elZwanlEVWJFRmRRR0lUTW5ZSUdiN0UxSE1pQmw1Q1U5U09hSSsrUjUwMD0=
  key-1712602202895580299: bGR1aTFJRk85TlZZKzBZTkRuWmdMS1V6aFdiWDZGUnZxSERqMUNaSXlIYz0=
  key-1712602503111158828: c0w0bExWei9UUmtpU1J5NUYzUUJOMnp3R3lYRGRoOXpZek1KOWJrRjY2RT0=
  key-1712620802759789273: RzYyRSttUXBQV1BPajlSK2Y2cjFTZVRENDk4Yi9yUlIvdzJLOCs0VUc5WT0=
kind: Secret
metadata:
  creationTimestamp: "2024-04-08T08:04:09Z"
  name: noobaa-root-master-key-backend
  namespace: openshift-storage
  resourceVersion: "601456"
  uid: ffb9ef16-a9c3-4fee-80e0-09896171aa34
type: Opaque

Comment 8 Tiffany Nguyen 2024-04-17 16:31:37 UTC

Verified this issue with ODF 4.16.0-75
Issue is fixed and worked as expected.

$ oc get secret noobaa-root-master-key-volume -o yaml
apiVersion: v1
data:
  active_root_key: a2V5LTE3MTMyNTM1MDM3NTU0MTE1MTY=
  key-1713095320923357529: SHgrL21MTDZJVmQyT0xqaWcvMTVreW9SUmZseDYvMXFReU85dElqN3VsVT0=
  key-1713160076680614983: cFJDNXFudW9MZkJ3NHcyNFdZMTk3eVNCQkRiQ09XZGd3YTdRcnk4Y1d2cz0=
  key-1713160203225402218: UWNlZDlLb1g3b21aYXI3U0IyYjB3WklnbU4zRFpMamIxemhoODl4WVBKQT0=
  key-1713160503478375498: aEVPR04zM3hXZWp0b3d5ZFhsTTc3dUZvaWtxNjk5T2VZVHVrS2FnTXVNOD0=
  key-1713160801312564803: ZEljZXRoL3VFWngvVktiS0F5cThBYitrN3dIYjVFUUpjUFg2aUJvQTJvUT0=
  key-1713161103935223094: U0ZXcG03cDhGUFExS29KcXFaZ1FmYnZ2RG8vOG52MEZwK3dxK21GZHhZZz0=
  key-1713161404219746040: UjdRVEFuUURCakxTeFJ1V2s4UmUwSmpoUlR0N0VKb2k1M0Z1OTZTcG5QUT0=
  key-1713161704532982562: MGE4YklXcnQvQlhJQ3pTNGlydXI5RHFCY3QvWkFzMy9YakdISXg5MkUwUT0=
  key-1713162001822695533: ZjhKTVBBVjc3TlFGdENuREFCN285aE5USzBNYThWZFUrT3p5VytqSTExRT0=
  key-1713162305086394207: aTg0NCtXQVp3QmpsUFdMYk1hdW1vQkRBZjlicVo1ODBCNlNxSSs2VFNtND0=
  key-1713162605302220732: Q2pucU51QVFVUWwvMkd3TC81WnVLVDV3Yk5zLzY0Nk5heWlEOG5qODRKYz0=
  key-1713162905553887999: ajA0Q0prSUdaTHcyZnNGaVdadlRRd1dqRlYrOWhYUGJzS1VuNktMbDVrcz0=
  key-1713163205898289449: dDlUSVRrMi9UUDI0UmZJMkxPUi85Y3ZtYjlYb0dxZWRmY1pyWFlrTmxNST0=
  key-1713163506025301657: ZW95MFFoTkZJSEk1U0dUL2dYVG5XYnhVUW96SURDd1BXMDV0OWRKelVsZz0=
  key-1713163801239084765: RWlVcytWOXh0STJKZDg3U081a0lBY2l4MzdmRUc1bGN6dGE3QnB6TDhjbz0=
  key-1713164106543795232: QU9kanFOVEdrbzhUN3ZtMlhKR2YxWkt3UncyZjg0ejFPcnRxZEczditOdz0=
  key-1713164406949332266: ZDZSbXJ3Mi9tTHU3QkxoT21yb2diV2c5dTVNSkoxU1ZScEhqS3ZabVIrUT0=
  key-1713164703331975143: a1k5c2plNk1FYTg5R1ExcVc4WFprL2dqMEttellNK0gyVzZERVA5NHNzbz0=
  key-1713165007338147592: YWlabjRnU1RVVjdBcXNsVGZZNi85Y05hNlV5SVI5czhCaXZTQmluK1lpZz0=
  key-1713165307581033406: cFJzSW83WGhsbWJkVkR4blNZWDhEMVRZK280WTRPcXlLTWFoVjd0bXVvRT0=
  key-1713165606881306394: UUVCYUxWZ2xRZUZaNjBxS0dkeWFKdEQ5cnJDRkdVSXJLYWpnL010elRLYz0=
  key-1713165908115947008: SndaRUZ5cDZyRGd0NGV1V2M2ZEFlUXhYclFQTGUzOXBNY0ZuYTJ0WGY1bz0=
  key-1713166208361768431: MGpxRTJoTkJJQ1hjZDk0V3BqMnhqZ2JySVJ3SUVFaXVwclFzeUFKdGNJZz0=
  key-1713166503592829177: Z0hQZVRrNk4rWVUzaTFjZTVCZVNaeUYxNSt6cWdSOHo3QWFJTWFCWFBVYz0=
  key-1713166800254378171: VlpwZnRIMi9vMUFQVFJhN3Y0S0xMWDFNK2FtemVKMDN5ZjVhVURzOFVTcz0=
  key-1713167109192438562: RUtWKzJHNUdxZDl1d2Yyb2ZsdkdMZENhL1dOT013Z3BiZEk0dnd6VVhUQT0=
  key-1713209334105793928: MllEeGVVVTZleTU3ckg0VG9uVVRteitBZ1NsZ3FmdHFsM2ZoQ3NsNHNIbz0=
  key-1713209406689603341: SklhUHBpZVJ0Yks0U1MxQ1kzK25TREJ1L0E2ejUrZlVRb2t6VDE1cy9HOD0=
  key-1713225601231818386: bkhlVnorOE8waFRHTmpxYTNROHdpdmJvL1hWMnJDdGNHSUpXTlU4Yy84UT0=
  key-1713249158758679878: UjBxQkw3RFphN3dtYTNjUXJudWhjUVJkWGwxQVRuVXRxV2o4N2oweFdsRT0=
  key-1713249180198384766: ZmpYaExYMC80T3kvVTY5NnNXbFdpWklrYVV4aEJiSDI0d0hiTFk1Yjgzdz0=
  key-1713249360374648096: VTF0anMyL2J3SWZlYUlXYUo4Z1RJZzJ1TUxpSHVEbVJ5VlM0bG9ZeXNMVT0=
  key-1713251283107986532: VHlaYnczZHUyZUhDY3loR1VXdjU0VHRUcG9yUXl3aEhCenNLSHc4OGVCUT0=
  key-1713251341980442671: VlhhUDdjdTFxeHg5bSthNFZZei8wQUwrVkJKQ3VhTW9EZzZPS3FiT0U1WT0=
  key-1713251781582629422: SXphL2pWVk0zRUJjbUtQSFR1b1pBem1IYXVSMWwyMndlcHBYRWRSaEVzOD0=
  key-1713253356469280892: U09ZS0QzTEFYM2NBbmppTEkrVzhpNjVaUVJBRWNXM0tCakR6M0lITWswMD0=
  key-1713253503755411516: UWUydW5yOGZPRmRvbXAxK1djRS9pYnIxYUhQVTQ4a1VHVHpOelNsQ0FIbz0=
kind: Secret
metadata:
  creationTimestamp: "2024-04-14T11:48:40Z"
  labels:
    app: noobaa
  name: noobaa-root-master-key-volume
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: noobaa.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: NooBaa
    name: noobaa
    uid: 3782c0a7-0182-4095-85f5-623de91454fb
  resourceVersion: "1645217"
  uid: ca316c99-442e-449e-9ad8-46bf0b9e28a4
type: Opaque

$ oc get secret noobaa-root-master-key-backend -o yaml
apiVersion: v1
data:
  active_root_key: a2V5LTE3MTMyNTM1MDM3NTU0MTE1MTY=
  key-1713095320923357529: SHgrL21MTDZJVmQyT0xqaWcvMTVreW9SUmZseDYvMXFReU85dElqN3VsVT0=
  key-1713160076680614983: cFJDNXFudW9MZkJ3NHcyNFdZMTk3eVNCQkRiQ09XZGd3YTdRcnk4Y1d2cz0=
  key-1713160203225402218: UWNlZDlLb1g3b21aYXI3U0IyYjB3WklnbU4zRFpMamIxemhoODl4WVBKQT0=
  key-1713160503478375498: aEVPR04zM3hXZWp0b3d5ZFhsTTc3dUZvaWtxNjk5T2VZVHVrS2FnTXVNOD0=
  key-1713160801312564803: ZEljZXRoL3VFWngvVktiS0F5cThBYitrN3dIYjVFUUpjUFg2aUJvQTJvUT0=
  key-1713161103935223094: U0ZXcG03cDhGUFExS29KcXFaZ1FmYnZ2RG8vOG52MEZwK3dxK21GZHhZZz0=
  key-1713161404219746040: UjdRVEFuUURCakxTeFJ1V2s4UmUwSmpoUlR0N0VKb2k1M0Z1OTZTcG5QUT0=
  key-1713161704532982562: MGE4YklXcnQvQlhJQ3pTNGlydXI5RHFCY3QvWkFzMy9YakdISXg5MkUwUT0=
  key-1713162001822695533: ZjhKTVBBVjc3TlFGdENuREFCN285aE5USzBNYThWZFUrT3p5VytqSTExRT0=
  key-1713162305086394207: aTg0NCtXQVp3QmpsUFdMYk1hdW1vQkRBZjlicVo1ODBCNlNxSSs2VFNtND0=
  key-1713162605302220732: Q2pucU51QVFVUWwvMkd3TC81WnVLVDV3Yk5zLzY0Nk5heWlEOG5qODRKYz0=
  key-1713162905553887999: ajA0Q0prSUdaTHcyZnNGaVdadlRRd1dqRlYrOWhYUGJzS1VuNktMbDVrcz0=
  key-1713163205898289449: dDlUSVRrMi9UUDI0UmZJMkxPUi85Y3ZtYjlYb0dxZWRmY1pyWFlrTmxNST0=
  key-1713163506025301657: ZW95MFFoTkZJSEk1U0dUL2dYVG5XYnhVUW96SURDd1BXMDV0OWRKelVsZz0=
  key-1713163801239084765: RWlVcytWOXh0STJKZDg3U081a0lBY2l4MzdmRUc1bGN6dGE3QnB6TDhjbz0=
  key-1713164106543795232: QU9kanFOVEdrbzhUN3ZtMlhKR2YxWkt3UncyZjg0ejFPcnRxZEczditOdz0=
  key-1713164406949332266: ZDZSbXJ3Mi9tTHU3QkxoT21yb2diV2c5dTVNSkoxU1ZScEhqS3ZabVIrUT0=
  key-1713164703331975143: a1k5c2plNk1FYTg5R1ExcVc4WFprL2dqMEttellNK0gyVzZERVA5NHNzbz0=
  key-1713165007338147592: YWlabjRnU1RVVjdBcXNsVGZZNi85Y05hNlV5SVI5czhCaXZTQmluK1lpZz0=
  key-1713165307581033406: cFJzSW83WGhsbWJkVkR4blNZWDhEMVRZK280WTRPcXlLTWFoVjd0bXVvRT0=
  key-1713165606881306394: UUVCYUxWZ2xRZUZaNjBxS0dkeWFKdEQ5cnJDRkdVSXJLYWpnL010elRLYz0=
  key-1713165908115947008: SndaRUZ5cDZyRGd0NGV1V2M2ZEFlUXhYclFQTGUzOXBNY0ZuYTJ0WGY1bz0=
  key-1713166208361768431: MGpxRTJoTkJJQ1hjZDk0V3BqMnhqZ2JySVJ3SUVFaXVwclFzeUFKdGNJZz0=
  key-1713166503592829177: Z0hQZVRrNk4rWVUzaTFjZTVCZVNaeUYxNSt6cWdSOHo3QWFJTWFCWFBVYz0=
  key-1713166800254378171: VlpwZnRIMi9vMUFQVFJhN3Y0S0xMWDFNK2FtemVKMDN5ZjVhVURzOFVTcz0=
  key-1713167109192438562: RUtWKzJHNUdxZDl1d2Yyb2ZsdkdMZENhL1dOT013Z3BiZEk0dnd6VVhUQT0=
  key-1713209334105793928: MllEeGVVVTZleTU3ckg0VG9uVVRteitBZ1NsZ3FmdHFsM2ZoQ3NsNHNIbz0=
  key-1713209406689603341: SklhUHBpZVJ0Yks0U1MxQ1kzK25TREJ1L0E2ejUrZlVRb2t6VDE1cy9HOD0=
  key-1713225601231818386: bkhlVnorOE8waFRHTmpxYTNROHdpdmJvL1hWMnJDdGNHSUpXTlU4Yy84UT0=
  key-1713249158758679878: UjBxQkw3RFphN3dtYTNjUXJudWhjUVJkWGwxQVRuVXRxV2o4N2oweFdsRT0=
  key-1713249180198384766: ZmpYaExYMC80T3kvVTY5NnNXbFdpWklrYVV4aEJiSDI0d0hiTFk1Yjgzdz0=
  key-1713249360374648096: VTF0anMyL2J3SWZlYUlXYUo4Z1RJZzJ1TUxpSHVEbVJ5VlM0bG9ZeXNMVT0=
  key-1713251283107986532: VHlaYnczZHUyZUhDY3loR1VXdjU0VHRUcG9yUXl3aEhCenNLSHc4OGVCUT0=
  key-1713251341980442671: VlhhUDdjdTFxeHg5bSthNFZZei8wQUwrVkJKQ3VhTW9EZzZPS3FiT0U1WT0=
  key-1713251781582629422: SXphL2pWVk0zRUJjbUtQSFR1b1pBem1IYXVSMWwyMndlcHBYRWRSaEVzOD0=
  key-1713253356469280892: U09ZS0QzTEFYM2NBbmppTEkrVzhpNjVaUVJBRWNXM0tCakR6M0lITWswMD0=
  key-1713253503755411516: UWUydW5yOGZPRmRvbXAxK1djRS9pYnIxYUhQVTQ4a1VHVHpOelNsQ0FIbz0=
kind: Secret
metadata:
  creationTimestamp: "2024-04-14T11:48:40Z"
  name: noobaa-root-master-key-backend
  namespace: openshift-storage
  resourceVersion: "1645216"
  uid: 249dec3c-5107-46ad-81a8-fab901ceebf2
type: Opaque

Comment 10 Sunil Kumar Acharya 2024-06-18 06:45:26 UTC

Please update the RDT flag/text appropriately.

Comment 11 errata-xmlrpc 2024-07-17 13:18:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591

Comment 12 Red Hat Bugzilla 2024-11-15 04:25:30 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days