Bug 2249781

Clone to track the fix in 4.14.1. 

+++ This bug was initially created as a clone of Bug #2247731 +++

Description of problem (please be detailed as possible and provide log
snippests):


Version of all relevant components (if applicable):


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:

--- Additional comment from Alexander on 2023-11-03 02:36:49 UTC ---

Cluster is in FIPS mode
ODF 4.13
OCP 4.13

The Nooba BackingStore object becomes "Rejected" when Noobaa-core pod is being rescheduled/restarted with a different pod IP. This results in a heartbeat RPC issue.

The expectation is to have the operator update the secret with the updated IP address.

However, when this secret is edited manually, the operator seeks a agent_conf.json file located within the PV through "/noobaa_init_files/noobaa_init.sh"

This means the only way to fix it is to go into each BackingStore pod and edit the PV agent_conf.json 

1. is this expected behaviour for noobaa operator? seems like an issue in the architecture as it is not prioritizing the secret.
2. why does it not use the Secret as precedence?


recommending that /noobaa_init_files/noobaa_init.sh should check for existing secret first and workflow to update the agent_conf.json in each PV this way. 






```
time="2023-11-02T16:24:49Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2023-11-02T16:24:49Z" level=info msg="✅ Exists: BucketClass \"it-local-bucket-class-prd\"\n"
time="2023-11-02T16:24:49Z" level=info msg="SetPhase: Verifying" bucketclass=openshift-storage/it-local-bucket-class-prd
time="2023-11-02T16:24:49Z" level=info msg="✅ Exists: ServiceAccount \"default\"\n"
time="2023-11-02T16:24:49Z" level=info msg="SetPhase: Rejected" backingstore=openshift-storage/it-local-backing-store-prd
time="2023-11-02T16:24:49Z" level=info msg="✅ Exists: BackingStore \"it-local-backing-store-prd\"\n"
time="2023-11-02T16:24:49Z" level=info msg="SetPhase: temporary error during phase \"Verifying\"" bucketclass=openshift-storage/it-local-bucket-class-prd
time="2023-11-02T16:24:49Z" level=warning msg="⏳ Temporary Error: NooBaa BackingStore \"it-local-backing-store-prd\" is not yet ready" bucketclass=openshift-storage/it-local-bucket-class-prd
time="2023-11-02T16:24:49Z" level=info msg="Update event detected for it-local-backing-store-prd (openshift-storage), queuing Reconcile"
time="2023-11-02T16:24:49Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore openshift-storage/it-local-backing-store-prd to bucketclasses"
time="2023-11-02T16:24:49Z" level=info msg="UpdateStatus: Done" backingstore=openshift-storage/it-local-backing-store-prd
time="2023-11-02T16:24:49Z" level=info msg="Start BackingStore Reconcile ..." backingstore=openshift-storage/noobaa-default-backing-store
```

--- Additional comment from RHEL Program Management on 2023-11-03 02:36:59 UTC ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.14.0' to '?', and so is being proposed to be fixed at the ODF 4.14.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from Liran Mauda on 2023-11-05 10:35:18 UTC ---

Adding Boris Ranto

Hi Alexander,

It seems that this is an issue with the kubectl itself, and it indeed related to FIPS

from the core logs we can see: 

Nov-2 16:20:57.043 [WebServer/38]  [WARN] core.util.os_utils:: discover_k8s_services: could not list OpenShift routes:  Error: Command failed: kubectl api-versions
FIPS mode is enabled, but the required OpenSSL library is not available

    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:517:28)
    at ChildProcess.emit (node:domain:489:12)
    at maybeClose (node:internal/child_process:1098:16)
    at Socket.<anonymous> (node:internal/child_process:450:11)
    at Socket.emit (node:events:517:28)
    at Socket.emit (node:domain:489:12)
    at Pipe.<anonymous> (node:net:350:12) {
  code: 1,
  killed: false,
  signal: null,
  cmd: 'kubectl api-versions ',
  stdout: '',
  stderr: 'FIPS mode is enabled, but the required OpenSSL library is not available\n'
}

The flow in the code is:
discover_k8s_services -> _list_openshift_routes -> kube_utils.api_exists('route.openshift.io'); -> exec_kubectl(`api-versions`, 'raw'); -> and then `kubectl`
The `kubectl` itself return the above error, which in turn we cannot list the openshift_routes

We can see from the logs that the noobaa upstream version is: 5.13.4-d296296 (first line in the logs), which translate to downstream 4.13.4

There was a CVE that lead us to bump the node version and with it the OpenSSL.

from the logs we can clearly see that FIPS is detected and the version of OpenSSL:

detect_fips_mode: found /proc/sys/crypto/fips_enabled with value 1
OpenSSL 3.0.7 1 Nov 2022 setting up

We should check if the correct version of kubectl was also updated in the 4.13 downstream build

@branto do we use the latest 4.13 kubectl? 

Best Regards,
Liran

--- Additional comment from Sunil Kumar Acharya on 2023-11-06 12:29:23 UTC ---

Moving the Non-blocker BZs out of ODF-4.14.0. If you think this is blocker issue for ODF-4.14.0, feel free to propose it as a blocker with justification note.

--- Additional comment from Boris Ranto on 2023-11-08 13:45:13 UTC ---

OK, I believe I know what the issue is. The issue is that we use RHEL 8 binary in the RHEL 9 container.

We are currently using ocs-cli containers to get the binary

https://gitlab.cee.redhat.com/ceph/rhodf/-/blob/rhodf-4.14-rhel-9/distgit/containers/mcg-core/Dockerfile.in?ref_type=heads#L96

However, these containers are rhel 8 only no matter what OCP version you use.

After some digging, I found the openshift-clients RPM that is built by OCP. This RPM contains both oc and kubectl binaries. After some more digging, I was able to find the content sets that have this RPM in them:

rhocp-4_DOT_14-for-rhel-9-aarch64-rpms
rhocp-4_DOT_14-for-rhel-9-s390x-rpms
rhocp-4_DOT_14-for-rhel-9-ppc64le-rpms
rhocp-4_DOT_14-for-rhel-9-x86_64-rpms

I have made the changes to in a private branch to test it out

https://pkgs.devel.redhat.com/cgit/containers/mcg-core/tree/?h=private-oc-clients

The scratch build did pass and it installed the oc/kubectl binaries in the final container:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=56829321

Based on the above, I'm providing devel_ack here. We should also back-port this all the way down to ODF 4.13 (no need to back-port it to the earlier releases as those are based on RHEL 8).

--- Additional comment from Liran Mauda on 2023-11-08 16:11:55 UTC ---

Based on Boris's analysis, moving to the build team.

--- Additional comment from Boris Ranto on 2023-11-09 21:54:43 UTC ---

This should fix this:

https://gitlab.cee.redhat.com/ceph/rhodf/-/commit/8f29799cd6ee3c11c9bf6c930f30e05e44a9dc46

This is technically a regression from 4.12 where FIPS mode worked fine as it was based on RHEL 8 and we could use the RHEL8-based ose-cli container to get the oc/kubectl binaries and it worked fine as it matched the RHEL version of final container. This is an issue from ODF 4.13 and we should back-port all the way there.

--- Additional comment from RHEL Program Management on 2023-11-09 21:54:53 UTC ---

This BZ is being approved for ODF 4.15.0 release, upon receipt of the 3 ACKs (PM,Devel,QA) for the release flag 'odf‑4.15.0

--- Additional comment from RHEL Program Management on 2023-11-09 21:54:53 UTC ---

Since this bug has been approved for ODF 4.15.0 release, through release flag 'odf-4.15.0+', the Target Release is being set to 'ODF 4.15.0

--- Additional comment from Bipin Kunal on 2023-11-15 06:21:21 UTC ---

Hi Alexander,

  We have a probable fix and we would like to get that tested in the non-production environment first before we allow customer to use it in the production. Can you check with the customer if they have a test environment where they can simulate the scenario/issue and test the fix? We will be providing a new image for mcg-core and steps to use that image. 

-Bipin Kunal