Description of problem (please be detailed as possible and provide log snippests): The cluster-wide encryption key rotation process is not functioning as expected for the NooBaa secret named 'noobaa-root-master-key-volume' though its function for the secret 'noobaa-root-master-key-backend'. Version of all relevant components (if applicable): 4.16 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Y Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Install an ODF 4.16 cluster with clusterwide encryption enabled, without a Key Management Service (KMS) configured. 2. Record the existing keys from the following NooBaa secrets: - `noobaa-root-master-key-backend` - `noobaa-root-master-key-volume` 3. Change the default key rotation period from 'weekly' to 'every 5 minutes' using a cron job in the storage cluster specification. 4. Wait for the key rotation to occur according to the new cron job schedule (every 5 minutes). 5. Compare the keys retrieved from the NooBaa secrets after key rotation with the recorded keys from step 2. 6. Confirm that the keys have rotated successfully in both `noobaa-root-master-key-backend` and `noobaa-root-master-key-volume` secrets. Actual results: Keys from the noobaa secret `noobaa-root-master-key-volume` has not been rotated. Expected results: Keys rotation should happen for noobaa secrets. - `noobaa-root-master-key-backend` - `noobaa-root-master-key-volume` Additional info: Cluster Details -=-=-=-=-=-=-= > ocs get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.16.0-69.stable NooBaa Operator 4.16.0-69.stable Succeeded ocs-client-operator.v4.16.0-69.stable OpenShift Data Foundation Client 4.16.0-69.stable Succeeded ocs-operator.v4.16.0-69.stable OpenShift Container Storage 4.16.0-69.stable Succeeded odf-csi-addons-operator.v4.16.0-69.stable CSI Addons 4.16.0-69.stable Succeeded odf-operator.v4.16.0-69.stable OpenShift Data Foundation 4.16.0-69.stable Succeeded odf-prometheus-operator.v4.16.0-69.stable Prometheus Operator 4.16.0-69.stable Succeeded rook-ceph-operator.v4.16.0-69.stable Rook-Ceph 4.16.0-69.stable Succeeded Storagecluster keyrotation configuration ===================== > ocs get storageclusters.ocs.openshift.io -ojsonpath='{.items[*].spec.encryption}' {"clusterWide":true,"keyRotation":{"schedule":"*/5 * * * *"},"kms":{}} Rook cephcluster keyrotation configuration =================== > ocs get cephclusters.ceph.rook.io -o jsonpath='{.items[].spec.security}' {"keyRotation":{"enabled":true,"schedule":"*/5 * * * *"},"kms":{}} Noobaa keyrotation configuration ===================== > ocs get noobaas.noobaa.io -o jsonpath='{.items[*].spec.security}' {"kms":{"enableKeyRotation":true,"schedule":"*/5 * * * *"}} NooBaa Secrets -=-=-=-=-=-=-=-=-=-=-= > ocs get secret noobaa-root-master-key-volume -o yaml apiVersion: v1 data: active_root_key: a2V5LTE3MTI1NjM0NDkwMzE1NjY1MDI= key-1712563449031566502: UjRnUU14bGlNd0lDY3NwTkpnQi9zYWpXeDkvNnp5RnhWRTZEYmJ0Vis5Yz0= kind: Secret metadata: creationTimestamp: "2024-04-08T08:04:09Z" labels: app: noobaa name: noobaa-root-master-key-volume namespace: openshift-storage ownerReferences: - apiVersion: noobaa.io/v1alpha1 blockOwnerDeletion: true controller: true kind: NooBaa name: noobaa uid: 6fb11e22-4f9d-4a89-bd5a-1c56b2d3d486 resourceVersion: "44439" uid: 0f02a2b2-21ea-4415-b3d0-1ca6ca15fffb type: Opaque ------------------------------------------- > ocs get secret noobaa-root-master-key-backend -o yaml apiVersion: v1 data: active_root_key: a2V5LTE3MTI2MjA4MDI3NTk3ODkyNzM= key-1712563449031566502: UjRnUU14bGlNd0lDY3NwTkpnQi9zYWpXeDkvNnp5RnhWRTZEYmJ0Vis5Yz0= key-1712600806071647216: SXlrcUgxNDZYVXErenYxTThDWjRmVkswa3Y3VUQwa0RqN25BZkFHVUVUZz0= key-1712601001965921492: SVMxcHNmTmUzUERjQU84eE1DQWJyOUg0eSs5WDRuRk9HT09MZjRrZkZZdz0= key-1712601302190304475: YjZveDd5TU5UOXJzM0VGcm4yWWNaZGxUQjJaQlo0M0tpdzBDaXdLMUxEYz0= key-1712601600334743630: UUpUbjBVb2FvandidEVOckFUWERWNHdHTWNWZFZqaFlYaUlMcHlrN3pXdz0= key-1712601902660350482: elZwanlEVWJFRmRRR0lUTW5ZSUdiN0UxSE1pQmw1Q1U5U09hSSsrUjUwMD0= key-1712602202895580299: bGR1aTFJRk85TlZZKzBZTkRuWmdMS1V6aFdiWDZGUnZxSERqMUNaSXlIYz0= key-1712602503111158828: c0w0bExWei9UUmtpU1J5NUYzUUJOMnp3R3lYRGRoOXpZek1KOWJrRjY2RT0= key-1712620802759789273: RzYyRSttUXBQV1BPajlSK2Y2cjFTZVRENDk4Yi9yUlIvdzJLOCs0VUc5WT0= kind: Secret metadata: creationTimestamp: "2024-04-08T08:04:09Z" name: noobaa-root-master-key-backend namespace: openshift-storage resourceVersion: "601456" uid: ffb9ef16-a9c3-4fee-80e0-09896171aa34 type: Opaque
Verified this issue with ODF 4.16.0-75 Issue is fixed and worked as expected. $ oc get secret noobaa-root-master-key-volume -o yaml apiVersion: v1 data: active_root_key: a2V5LTE3MTMyNTM1MDM3NTU0MTE1MTY= key-1713095320923357529: SHgrL21MTDZJVmQyT0xqaWcvMTVreW9SUmZseDYvMXFReU85dElqN3VsVT0= key-1713160076680614983: cFJDNXFudW9MZkJ3NHcyNFdZMTk3eVNCQkRiQ09XZGd3YTdRcnk4Y1d2cz0= key-1713160203225402218: UWNlZDlLb1g3b21aYXI3U0IyYjB3WklnbU4zRFpMamIxemhoODl4WVBKQT0= key-1713160503478375498: aEVPR04zM3hXZWp0b3d5ZFhsTTc3dUZvaWtxNjk5T2VZVHVrS2FnTXVNOD0= key-1713160801312564803: ZEljZXRoL3VFWngvVktiS0F5cThBYitrN3dIYjVFUUpjUFg2aUJvQTJvUT0= key-1713161103935223094: U0ZXcG03cDhGUFExS29KcXFaZ1FmYnZ2RG8vOG52MEZwK3dxK21GZHhZZz0= key-1713161404219746040: UjdRVEFuUURCakxTeFJ1V2s4UmUwSmpoUlR0N0VKb2k1M0Z1OTZTcG5QUT0= key-1713161704532982562: MGE4YklXcnQvQlhJQ3pTNGlydXI5RHFCY3QvWkFzMy9YakdISXg5MkUwUT0= key-1713162001822695533: ZjhKTVBBVjc3TlFGdENuREFCN285aE5USzBNYThWZFUrT3p5VytqSTExRT0= key-1713162305086394207: aTg0NCtXQVp3QmpsUFdMYk1hdW1vQkRBZjlicVo1ODBCNlNxSSs2VFNtND0= key-1713162605302220732: Q2pucU51QVFVUWwvMkd3TC81WnVLVDV3Yk5zLzY0Nk5heWlEOG5qODRKYz0= key-1713162905553887999: ajA0Q0prSUdaTHcyZnNGaVdadlRRd1dqRlYrOWhYUGJzS1VuNktMbDVrcz0= key-1713163205898289449: dDlUSVRrMi9UUDI0UmZJMkxPUi85Y3ZtYjlYb0dxZWRmY1pyWFlrTmxNST0= key-1713163506025301657: ZW95MFFoTkZJSEk1U0dUL2dYVG5XYnhVUW96SURDd1BXMDV0OWRKelVsZz0= key-1713163801239084765: RWlVcytWOXh0STJKZDg3U081a0lBY2l4MzdmRUc1bGN6dGE3QnB6TDhjbz0= key-1713164106543795232: QU9kanFOVEdrbzhUN3ZtMlhKR2YxWkt3UncyZjg0ejFPcnRxZEczditOdz0= key-1713164406949332266: ZDZSbXJ3Mi9tTHU3QkxoT21yb2diV2c5dTVNSkoxU1ZScEhqS3ZabVIrUT0= key-1713164703331975143: a1k5c2plNk1FYTg5R1ExcVc4WFprL2dqMEttellNK0gyVzZERVA5NHNzbz0= key-1713165007338147592: YWlabjRnU1RVVjdBcXNsVGZZNi85Y05hNlV5SVI5czhCaXZTQmluK1lpZz0= key-1713165307581033406: cFJzSW83WGhsbWJkVkR4blNZWDhEMVRZK280WTRPcXlLTWFoVjd0bXVvRT0= key-1713165606881306394: UUVCYUxWZ2xRZUZaNjBxS0dkeWFKdEQ5cnJDRkdVSXJLYWpnL010elRLYz0= key-1713165908115947008: SndaRUZ5cDZyRGd0NGV1V2M2ZEFlUXhYclFQTGUzOXBNY0ZuYTJ0WGY1bz0= key-1713166208361768431: MGpxRTJoTkJJQ1hjZDk0V3BqMnhqZ2JySVJ3SUVFaXVwclFzeUFKdGNJZz0= key-1713166503592829177: Z0hQZVRrNk4rWVUzaTFjZTVCZVNaeUYxNSt6cWdSOHo3QWFJTWFCWFBVYz0= key-1713166800254378171: VlpwZnRIMi9vMUFQVFJhN3Y0S0xMWDFNK2FtemVKMDN5ZjVhVURzOFVTcz0= key-1713167109192438562: RUtWKzJHNUdxZDl1d2Yyb2ZsdkdMZENhL1dOT013Z3BiZEk0dnd6VVhUQT0= key-1713209334105793928: MllEeGVVVTZleTU3ckg0VG9uVVRteitBZ1NsZ3FmdHFsM2ZoQ3NsNHNIbz0= key-1713209406689603341: SklhUHBpZVJ0Yks0U1MxQ1kzK25TREJ1L0E2ejUrZlVRb2t6VDE1cy9HOD0= key-1713225601231818386: bkhlVnorOE8waFRHTmpxYTNROHdpdmJvL1hWMnJDdGNHSUpXTlU4Yy84UT0= key-1713249158758679878: UjBxQkw3RFphN3dtYTNjUXJudWhjUVJkWGwxQVRuVXRxV2o4N2oweFdsRT0= key-1713249180198384766: ZmpYaExYMC80T3kvVTY5NnNXbFdpWklrYVV4aEJiSDI0d0hiTFk1Yjgzdz0= key-1713249360374648096: VTF0anMyL2J3SWZlYUlXYUo4Z1RJZzJ1TUxpSHVEbVJ5VlM0bG9ZeXNMVT0= key-1713251283107986532: VHlaYnczZHUyZUhDY3loR1VXdjU0VHRUcG9yUXl3aEhCenNLSHc4OGVCUT0= key-1713251341980442671: VlhhUDdjdTFxeHg5bSthNFZZei8wQUwrVkJKQ3VhTW9EZzZPS3FiT0U1WT0= key-1713251781582629422: SXphL2pWVk0zRUJjbUtQSFR1b1pBem1IYXVSMWwyMndlcHBYRWRSaEVzOD0= key-1713253356469280892: U09ZS0QzTEFYM2NBbmppTEkrVzhpNjVaUVJBRWNXM0tCakR6M0lITWswMD0= key-1713253503755411516: UWUydW5yOGZPRmRvbXAxK1djRS9pYnIxYUhQVTQ4a1VHVHpOelNsQ0FIbz0= kind: Secret metadata: creationTimestamp: "2024-04-14T11:48:40Z" labels: app: noobaa name: noobaa-root-master-key-volume namespace: openshift-storage ownerReferences: - apiVersion: noobaa.io/v1alpha1 blockOwnerDeletion: true controller: true kind: NooBaa name: noobaa uid: 3782c0a7-0182-4095-85f5-623de91454fb resourceVersion: "1645217" uid: ca316c99-442e-449e-9ad8-46bf0b9e28a4 type: Opaque $ oc get secret noobaa-root-master-key-backend -o yaml apiVersion: v1 data: active_root_key: a2V5LTE3MTMyNTM1MDM3NTU0MTE1MTY= key-1713095320923357529: SHgrL21MTDZJVmQyT0xqaWcvMTVreW9SUmZseDYvMXFReU85dElqN3VsVT0= key-1713160076680614983: cFJDNXFudW9MZkJ3NHcyNFdZMTk3eVNCQkRiQ09XZGd3YTdRcnk4Y1d2cz0= key-1713160203225402218: UWNlZDlLb1g3b21aYXI3U0IyYjB3WklnbU4zRFpMamIxemhoODl4WVBKQT0= key-1713160503478375498: aEVPR04zM3hXZWp0b3d5ZFhsTTc3dUZvaWtxNjk5T2VZVHVrS2FnTXVNOD0= key-1713160801312564803: ZEljZXRoL3VFWngvVktiS0F5cThBYitrN3dIYjVFUUpjUFg2aUJvQTJvUT0= key-1713161103935223094: U0ZXcG03cDhGUFExS29KcXFaZ1FmYnZ2RG8vOG52MEZwK3dxK21GZHhZZz0= key-1713161404219746040: UjdRVEFuUURCakxTeFJ1V2s4UmUwSmpoUlR0N0VKb2k1M0Z1OTZTcG5QUT0= key-1713161704532982562: MGE4YklXcnQvQlhJQ3pTNGlydXI5RHFCY3QvWkFzMy9YakdISXg5MkUwUT0= key-1713162001822695533: ZjhKTVBBVjc3TlFGdENuREFCN285aE5USzBNYThWZFUrT3p5VytqSTExRT0= key-1713162305086394207: aTg0NCtXQVp3QmpsUFdMYk1hdW1vQkRBZjlicVo1ODBCNlNxSSs2VFNtND0= key-1713162605302220732: Q2pucU51QVFVUWwvMkd3TC81WnVLVDV3Yk5zLzY0Nk5heWlEOG5qODRKYz0= key-1713162905553887999: ajA0Q0prSUdaTHcyZnNGaVdadlRRd1dqRlYrOWhYUGJzS1VuNktMbDVrcz0= key-1713163205898289449: dDlUSVRrMi9UUDI0UmZJMkxPUi85Y3ZtYjlYb0dxZWRmY1pyWFlrTmxNST0= key-1713163506025301657: ZW95MFFoTkZJSEk1U0dUL2dYVG5XYnhVUW96SURDd1BXMDV0OWRKelVsZz0= key-1713163801239084765: RWlVcytWOXh0STJKZDg3U081a0lBY2l4MzdmRUc1bGN6dGE3QnB6TDhjbz0= key-1713164106543795232: QU9kanFOVEdrbzhUN3ZtMlhKR2YxWkt3UncyZjg0ejFPcnRxZEczditOdz0= key-1713164406949332266: ZDZSbXJ3Mi9tTHU3QkxoT21yb2diV2c5dTVNSkoxU1ZScEhqS3ZabVIrUT0= key-1713164703331975143: a1k5c2plNk1FYTg5R1ExcVc4WFprL2dqMEttellNK0gyVzZERVA5NHNzbz0= key-1713165007338147592: YWlabjRnU1RVVjdBcXNsVGZZNi85Y05hNlV5SVI5czhCaXZTQmluK1lpZz0= key-1713165307581033406: cFJzSW83WGhsbWJkVkR4blNZWDhEMVRZK280WTRPcXlLTWFoVjd0bXVvRT0= key-1713165606881306394: UUVCYUxWZ2xRZUZaNjBxS0dkeWFKdEQ5cnJDRkdVSXJLYWpnL010elRLYz0= key-1713165908115947008: SndaRUZ5cDZyRGd0NGV1V2M2ZEFlUXhYclFQTGUzOXBNY0ZuYTJ0WGY1bz0= key-1713166208361768431: MGpxRTJoTkJJQ1hjZDk0V3BqMnhqZ2JySVJ3SUVFaXVwclFzeUFKdGNJZz0= key-1713166503592829177: Z0hQZVRrNk4rWVUzaTFjZTVCZVNaeUYxNSt6cWdSOHo3QWFJTWFCWFBVYz0= key-1713166800254378171: VlpwZnRIMi9vMUFQVFJhN3Y0S0xMWDFNK2FtemVKMDN5ZjVhVURzOFVTcz0= key-1713167109192438562: RUtWKzJHNUdxZDl1d2Yyb2ZsdkdMZENhL1dOT013Z3BiZEk0dnd6VVhUQT0= key-1713209334105793928: MllEeGVVVTZleTU3ckg0VG9uVVRteitBZ1NsZ3FmdHFsM2ZoQ3NsNHNIbz0= key-1713209406689603341: SklhUHBpZVJ0Yks0U1MxQ1kzK25TREJ1L0E2ejUrZlVRb2t6VDE1cy9HOD0= key-1713225601231818386: bkhlVnorOE8waFRHTmpxYTNROHdpdmJvL1hWMnJDdGNHSUpXTlU4Yy84UT0= key-1713249158758679878: UjBxQkw3RFphN3dtYTNjUXJudWhjUVJkWGwxQVRuVXRxV2o4N2oweFdsRT0= key-1713249180198384766: ZmpYaExYMC80T3kvVTY5NnNXbFdpWklrYVV4aEJiSDI0d0hiTFk1Yjgzdz0= key-1713249360374648096: VTF0anMyL2J3SWZlYUlXYUo4Z1RJZzJ1TUxpSHVEbVJ5VlM0bG9ZeXNMVT0= key-1713251283107986532: VHlaYnczZHUyZUhDY3loR1VXdjU0VHRUcG9yUXl3aEhCenNLSHc4OGVCUT0= key-1713251341980442671: VlhhUDdjdTFxeHg5bSthNFZZei8wQUwrVkJKQ3VhTW9EZzZPS3FiT0U1WT0= key-1713251781582629422: SXphL2pWVk0zRUJjbUtQSFR1b1pBem1IYXVSMWwyMndlcHBYRWRSaEVzOD0= key-1713253356469280892: U09ZS0QzTEFYM2NBbmppTEkrVzhpNjVaUVJBRWNXM0tCakR6M0lITWswMD0= key-1713253503755411516: UWUydW5yOGZPRmRvbXAxK1djRS9pYnIxYUhQVTQ4a1VHVHpOelNsQ0FIbz0= kind: Secret metadata: creationTimestamp: "2024-04-14T11:48:40Z" name: noobaa-root-master-key-backend namespace: openshift-storage resourceVersion: "1645216" uid: 249dec3c-5107-46ad-81a8-fab901ceebf2 type: Opaque
Please update the RDT flag/text appropriately.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:4591