Bug 2309700

Description of problem (please be detailed as possible and provide log
snippests):
-----------------------------------------------------------------------

In ODF 4.17 after setting up the feature as described in the happy path validation steps (https://url.corp.redhat.com/6059877), logs are generated as expected under the logs PVC:
```
$ oc rsh noobaa-core-0 ls -l /var/logs/bucket-logs
Defaulted container "core" out of: core, noobaa-log-processor
total 27
-rw-rw-rw-. 1 root 1000700000  1684 Sep  4 11:46 bucket_logging_shirshfe-417ibm92-6mbdc-worker-1-jbt9t.log
-rw-rw-rw-. 1 root 1000700000  7970 Sep  4 11:24 bucket_logging_shirshfe-417ibm92-6mbdc-worker-2-kptqv.log
-rw-rw-rw-. 1 root 1000700000 16752 Sep  4 12:02 bucket_logging_shirshfe-417ibm92-6mbdc-worker-3-zn9cj.log
```

However, these logs are never uploaded to the logs bucket even after we've set the logs uploader background worker's interval to one minute:
```
$ oc -n openshift-storage patch StatefulSet noobaa-core -n openshift-storage -p '[{"op": "add", "path": "/spec/template/spec/containers/0/env/-", "value": {"name": "CONFIG_JS_LOG_UPLOADER_INTERVAL", "value": "60000"}}]' --type json
```

Additionally, the following errors were spotted in the noobaa-core logs:
```
$ oc logs noobaa-core-0 --follow | grep core.util.persistent_logger
Defaulted container "core" out of: core, noobaa-log-processor

Sep-4 11:36:19.107 [BGWorkers/32]  [WARN] core.util.persistent_logger:: failed to rename active file: [Error: Permission denied] { code: 'EACCES', context: 'Rename _old_path=/var/logs/bucket-logs/bucket_logging_shirshfe-417ibm92-6mbdc-worker-3-zn9cj.log _new_path=/var/logs/bucket-logs/bucket_logging_shirshfe-417ibm92-6mbdc-worker-3-zn9cj.1725449779105.log ' }
...
...
...
Sep-4 11:46:19.125 [BGWorkers/32]  [WARN] core.util.persistent_logger:: failed to rename active file: [Error: Permission denied] { code: 'EACCES', context: 'Rename _old_path=/var/logs/bucket-logs/bucket_logging_shirshfe-417ibm92-6mbdc-worker-2-kptqv.log _new_path=/var/logs/bucket-logs/bucket_logging_shirshfe-417ibm92-6mbdc-worker-2-kptqv.1725450379123.log ' }
Sep-4 11:46:19.126 [BGWorkers/32]  [WARN] core.util.persistent_logger:: failed to rename active file: [Error: No such file or directory] { code: 'ENOENT', context: 'Rename _old_path=/var/logs/bucket-logs/bucket_logging_shirshfe-417ibm92-6mbdc-worker-3-zn9cj.failure.log _new_path=/var/logs/bucket-logs/bucket_logging_shirshfe-417ibm92-6mbdc-worker-3-zn9cj.failure.1725450379125.log ' }
Sep-4 11:46:19.126 [BGWorkers/32]  [WARN] core.util.persistent_logger:: failed to rename active file: [Error: No such file or directory] { code: 'ENOENT', context: 'Rename _old_path=/var/logs/bucket-logs/bucket_logging_shirshfe-417ibm92-6mbdc-worker-2-kptqv.failure.log _new_path=/var/logs/bucket-logs/bucket_logging_shirshfe-417ibm92-6mbdc-worker-2-kptqv.failure.1725450379125.log ' }
```

Version of all relevant components (if applicable):
-----------------------------------------------------------------------
OCP: 4.17.0-0.nightly-2024-09-03-024255
ODF: 4.17.0-90
ceph: 19.1.0-42.el9cp (03ae7f7ffec5e7796d2808064c4766b35c4b5ffb) squid (rc)
rook: v4.17.0-0.8d5e9a912402bbefb3e747920084744409291444
noobaa core: 5.17.0-267ee3a
noobaa operator: 5.17.0-825cbeac608a0dd2c9166ee866678f44813a365d
noobaa cli: 5.17.0


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
-----------------------------------------------------------------------
The logs are never delivered to the logs bucket


Is there any workaround available to the best of your knowledge?
-----------------------------------------------------------------------


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
-----------------------------------------------------------------------
2


Can this issue reproducible?
-----------------------------------------------------------------------
100%


Can this issue reproduce from the UI?
-----------------------------------------------------------------------
N/A


If this is a regression, please provide more details to justify this:
-----------------------------------------------------------------------
Seems like it - while validating the happy path about a month ago it didn't happen


Steps to Reproduce:
-----------------------------------------------------------------------
1. Follow the happy path validation steps at https://url.corp.redhat.com/6059877


Actual results:
-----------------------------------------------------------------------
Failure at steps #8 and #9:

"
8. wait for around 5 min. see that the logs were uploaded to the correct log-bucket with the correct prefix - the logs will be in the same format as in the mount
9. make sure the log file in the mount is deleted
"

The logs were never uploaded to the log-bucket, and log files are never deleted from the PVC (/var/logs/bucket-logs at the noobaa-core pod)


Expected results:
-----------------------------------------------------------------------
All the logs that were generated to the logs PVC should be copied to the logs bucket and then deleted from the volume.