Bug 2302842 - [MCG-4.17] noobaa-core pod doesn't dismount noobaa-bucket-logging-volume when disabling or changing Guaranteed Bucket Logs config at noobaa CR
Summary: [MCG-4.17] noobaa-core pod doesn't dismount noobaa-bucket-logging-volume when...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.16
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ODF 4.17.0
Assignee: Jacky Albo
QA Contact: Sagi Hirshfeld
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-08-05 08:31 UTC by Sagi Hirshfeld
Modified: 2024-10-30 14:30 UTC (History)
4 users (show)

Fixed In Version: 4.17.0-77
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-10-30 14:30:05 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github noobaa noobaa-operator pull 1406 0 None Merged Added a fix to update bucketLogVolumes and bucketLogVolumeMounts in noobaa core pod 2024-08-07 08:53:36 UTC
Github noobaa noobaa-operator pull 1409 0 None Merged backport into 5.17 2024-08-12 11:08:18 UTC
Red Hat Issue Tracker OCSBZM-8846 0 None None None 2024-08-26 06:25:43 UTC
Red Hat Product Errata RHSA-2024:8676 0 None None None 2024-10-30 14:30:08 UTC

Description Sagi Hirshfeld 2024-08-05 08:31:56 UTC
Description of problem (please be detailed as possible and provide log
snippests):
---------------------------------------------------------------------
Following the happy-path validation steps at https://issues.redhat.com/browse/RHSTOR-6133, we're provided a custom PVC at the first step:

```
bucketLogging: {  
  loggingType: guaranteed       
  bucketLoggingPVC: pvc-test-aed9c3ff7ea44069adbecaf05412b68
}
```

The rest of the steps are working as expected until step #11, where we attempt to disable the guaranteed bucket logs functionality at the NooBaa CR by setting an empty value under .spec.bucketLogging:

```
oc -n openshift-storage patch noobaa noobaa -n openshift-storage -p '[{"op": "replace", "path": "/spec/bucketLogging", "value": null}]'
```

The above patch command is accepted, and as a result, the noobaa-core and noobaa-endpoint pods restart. However, the noobaa-core pod is still mounting the PVC after restarting:

```
$ oc describe pod noobaa-core-0 | grep Mounts  -A5
    Mounts:
      ...
      ...
      /var/logs/bucket-logs from noobaa-bucket-logging-volume (rw)

$ oc describe pod noobaa-core-0 | grep Volumes -A30 | grep noobaa-bucket-logging-volume -A3
```
  noobaa-bucket-logging-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc-test-aed9c3ff7ea44069adbecaf05412b68
    ReadOnly:   false

```

Furthermore, attempting to use the default noobaa-bucket-logging-pvc by setting 
```
bucketLogging: {  
  loggingType: guaranteed       
}
```
leads only changes the mounts of the noobaa-endpoint pods, while noobaa-core keeps mounting the previous PVC that was provided.

In any such case where the noobaa-core pod mounts a different PVC for noobaa-bucket-logging-volum than the noobaa-endpoint pods, the logging also doesn't seem to work even after a long time.

Version of all relevant components (if applicable):
---------------------------------------------------------------------
OCP: 4.17.0-0.nightly-2024-08-01-213905
ODF: 4.17.0-63
ceph: 19.1.0-4.el9cp (b2c7ded5f7885ce1d488a241a30cba80f58d28bc) squid (rc)
rook: v4.17.0-0.b5d18bf0dd4ee4aacfe881ca3030ce10a156008f
noobaa core: 5.17.0-a7069fc
noobaa operator: 5.17.0-d3d80339937ee79af64a302e7870ae3c3c08127c
oc cli: 4.14.0-0.nightly-2023-09-02-132842
noobaa cli: 5.17.0


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
---------------------------------------------------------------------
1. Once the noobaa-core pod mounts a different PVC than the noobaa-endpoint pods the logging doesn't work anymore.
2. The user impact is that this can prevent the safe deletion of the provided PVC that remains mounted against the user's will.


Is there any workaround available to the best of your knowledge?
---------------------------------------------------------------------
None so far.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
---------------------------------------------------------------------
3


Can this issue reproducible?
---------------------------------------------------------------------
Yes


Can this issue reproduce from the UI?
---------------------------------------------------------------------
No


If this is a regression, please provide more details to justify this:
---------------------------------------------------------------------
No, since this is a new feature


Steps to Reproduce:
---------------------------------------------------------------------
1. Follow the happy path steps at https://issues.redhat.com/browse/RHSTOR-6133 up to step #10, using a provided PVC.
2. Check if the noobaa-core and noobaa-endpoint pods are still mounting the provided or default PVC:
```
$ oc describe pod noobaa-core-0 | grep Volumes -A30 | grep noobaa-bucket-logging-volume -A3
$ oc describe pod -l noobaa-s3=noobaa | grep Volumes -A30 | grep noobaa-bucket-logging-volume -A3
```
3. Follow the happy path steps again using the default noobaa PVC


Actual results:
---------------------------------------------------------------------
1. The steps work until #11
2. The noobaa-core pod still mounts the provided PVC for noobaa-bucket-logging-volume while the noobaa-endpoint pods do not.
3. The happy path validation fails at the second time because the operations never get logged.


Expected results:
---------------------------------------------------------------------
1. The steps work until #11
2. The noobaa-core and noobaa-endpoint pods successfully dismount the provided PVC
3. The happy path validation succeeds when using the default PVC


Additional info:
---------------------------------------------------------------------
1. Similar results were seen when we first set the feature with the default PVC:
```
bucketLogging: {  
  loggingType: guaranteed       
}
```
and then attempted to provide an alternative user-made PVC. The noobaa-core pod kept mounting noobaa-bucket-logging-pvc while the noobaa-endpoint pods did switch.

Comment 7 Sunil Kumar Acharya 2024-08-26 11:12:44 UTC
Please update the RDT flag/text appropriately.

Comment 10 errata-xmlrpc 2024-10-30 14:30:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.17.0 Security, Enhancement, & Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:8676


Note You need to log in before you can comment on or make changes to this bug.