Description of problem (please be detailed as possible and provide log snippests): Using storage system wizard to create a storage system with external postgres. Noobaa-core and noobaa-db pods failed to create. Not all the pods are created and running: $ oc get pod NAME READY STATUS RESTARTS AGE csi-addons-controller-manager-5dbfb55df9-85wxv 2/2 Running 0 14h csi-cephfsplugin-9xnsz 2/2 Running 0 14h csi-cephfsplugin-provisioner-58c69cfb78-7h5tv 6/6 Running 0 14h csi-cephfsplugin-provisioner-58c69cfb78-ccnng 6/6 Running 1 (14h ago) 14h csi-cephfsplugin-s2ngm 2/2 Running 0 14h csi-cephfsplugin-vbf98 2/2 Running 0 14h csi-rbdplugin-8s279 3/3 Running 0 14h csi-rbdplugin-p9rn5 3/3 Running 0 14h csi-rbdplugin-provisioner-d65774655-d5vtk 6/6 Running 0 14h csi-rbdplugin-provisioner-d65774655-lq24w 6/6 Running 0 14h csi-rbdplugin-tbr2v 3/3 Running 0 14h noobaa-operator-68b69cd44b-vdszf 2/2 Running 0 14h ocs-operator-859d787c7-vzzgf 1/1 Running 0 14h odf-console-8485dc45db-wpv28 1/1 Running 0 14h odf-operator-controller-manager-64fbbbdc4d-j25c6 2/2 Running 0 14h rook-ceph-crashcollector-tunguyen-111p-szz92-worker-1-7qp5lv872 1/1 Running 0 14h rook-ceph-crashcollector-tunguyen-111p-szz92-worker-2-zfcrmjs8s 1/1 Running 0 14h rook-ceph-crashcollector-tunguyen-111p-szz92-worker-3-dxr7dhnjj 1/1 Running 0 14h rook-ceph-exporter-tunguyen-111p-szz92-worker-1-7qp5j-7769pzhlc 1/1 Running 0 14h rook-ceph-exporter-tunguyen-111p-szz92-worker-2-zfcrm-bf5689l5n 1/1 Running 0 14h rook-ceph-exporter-tunguyen-111p-szz92-worker-3-dxr7h-6b47b8jvd 1/1 Running 0 14h rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-fd986dd9sz762 2/2 Running 0 14h rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-57f86548mc7z7 2/2 Running 0 14h rook-ceph-mgr-a-56747bb7c5-drh59 3/3 Running 0 14h rook-ceph-mgr-b-59d95b9d88-h4d9h 3/3 Running 0 14h rook-ceph-mon-a-64d7c864cc-wctc5 2/2 Running 0 14h rook-ceph-mon-b-55f5c4696d-xrkqs 2/2 Running 0 14h rook-ceph-mon-c-7dddc877c5-z2857 2/2 Running 0 14h rook-ceph-operator-b8cf888cf-jldx9 1/1 Running 0 14h ux-backend-server-695548595d-mjtzc 2/2 Running 0 14h Version of all relevant components (if applicable): ODF 4.15 build 4.15.0-112 $ oc get csv -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.15.0-112.stable NooBaa Operator 4.15.0-112.stable Succeeded ocs-operator.v4.15.0-112.stable OpenShift Container Storage 4.15.0-112.stable Succeeded odf-csi-addons-operator.v4.15.0-112.stable CSI Addons 4.15.0-112.stable Succeeded odf-operator.v4.15.0-112.stable OpenShift Data Foundation 4.15.0-112.stable Succeeded Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes, happy path testing is failing for epic https://issues.redhat.com/browse/RHSTOR-4749 Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: No Steps to Reproduce: 1. Deploy an OCP cluster 2. Install ODF build 4.15.0-112 3. Create storage system using storage system wizard 4. Select external postgres and input the database connection info 5. Complete the wizard and check for the installation progress Actual results: Storage system failed to create, noobaa pods failed to create. Expected results: Storage system, noobaa pods should create and running without any issue. Additional info: Must gather logs: http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-2257982/ Failing cluster is available for investigate.
I installed CRD manually using below command: $ kubectl create -k github.com/kubernetes-sigs/container-object-storage-interface-api customresourcedefinition.apiextensions.k8s.io/bucketaccessclasses.objectstorage.k8s.io created customresourcedefinition.apiextensions.k8s.io/bucketaccesses.objectstorage.k8s.io created customresourcedefinition.apiextensions.k8s.io/bucketclaims.objectstorage.k8s.io created customresourcedefinition.apiextensions.k8s.io/bucketclasses.objectstorage.k8s.io created customresourcedefinition.apiextensions.k8s.io/buckets.objectstorage.k8s.io created However, noobaa-db and noobaa-core pods are not created: $ oc get pod | grep noobaa noobaa-operator-798cd44446-hgwpq 2/2 Running 0 46m
(In reply to Tiffany Nguyen from comment #6) > I installed CRD manually using below command: > > $ kubectl create -k > github.com/kubernetes-sigs/container-object-storage-interface-api > customresourcedefinition.apiextensions.k8s.io/bucketaccessclasses. > objectstorage.k8s.io created > customresourcedefinition.apiextensions.k8s.io/bucketaccesses.objectstorage. > k8s.io created > customresourcedefinition.apiextensions.k8s.io/bucketclaims.objectstorage.k8s. > io created > customresourcedefinition.apiextensions.k8s.io/bucketclasses.objectstorage. > k8s.io created > customresourcedefinition.apiextensions.k8s.io/buckets.objectstorage.k8s.io > created > > However, noobaa-db and noobaa-core pods are not created: > > $ oc get pod | grep noobaa > noobaa-operator-798cd44446-hgwpq 2/2 > Running 0 46m Jacky, could you pls take a look?
Verifing the fix using build 4.15.0-130. The storagecluster is now getting "externalPgConfig" and "pgSecretName". However, there are few more issues are seen when configure an external postgres. As result, storagecluster doesn't deploy correctly and noobaa is not get created. 1. In the secret, "db_url" is incorrect. Provide link: postgres://postgres:postgres.99.1.namespace.svc:5432/Tiffany Correct link: postgresql://postgres:postgres.99.1:5432/tiffany 2. externalPgSSLRequired is set to "true" in noobaa.yaml when there is no SecureSSL database selected. This is causing the database connection error.
https://bugzilla.redhat.com/show_bug.cgi?id=2262974 - Created clone for this bz on OCS operator as well. May need to move them both on QA
No doc changes required.
Test with ODF 4.15 build 139: $ oc get csv -A NAMESPACE NAME DISPLAY VERSION REPLACES PHASE openshift-operator-lifecycle-manager packageserver Package Server 0.0.1-snapshot Succeeded openshift-storage mcg-operator.v4.15.0-139.stable NooBaa Operator 4.15.0-139.stable Succeeded openshift-storage ocs-operator.v4.15.0-139.stable OpenShift Container Storage 4.15.0-139.stable Succeeded openshift-storage odf-csi-addons-operator.v4.15.0-139.stable CSI Addons 4.15.0-139.stable Succeeded openshift-storage odf-operator.v4.15.0-139.stable OpenShift Data Foundation 4.15.0-139.stable Succeeded $ oc get pod NAME READY STATUS RESTARTS AGE csi-addons-controller-manager-555bcf9c9d-n5v8g 2/2 Running 0 5m57s csi-cephfsplugin-dszbm 2/2 Running 0 4m11s csi-cephfsplugin-provisioner-5b5575d8d5-2j4kx 6/6 Running 0 4m11s csi-cephfsplugin-provisioner-5b5575d8d5-pjdtn 6/6 Running 0 4m11s csi-cephfsplugin-sfmss 2/2 Running 0 4m11s csi-cephfsplugin-v2q6l 2/2 Running 1 (3m22s ago) 4m11s csi-rbdplugin-64k5k 3/3 Running 0 4m11s csi-rbdplugin-fscs6 3/3 Running 0 4m11s csi-rbdplugin-k7w7n 3/3 Running 1 (3m22s ago) 4m11s csi-rbdplugin-provisioner-df8895f7b-qxmcb 6/6 Running 0 4m11s csi-rbdplugin-provisioner-df8895f7b-sc6f5 6/6 Running 4 (2m36s ago) 4m11s noobaa-operator-7cccc64c59-mf6nf 2/2 Running 0 6m17s ocs-operator-5bc895b594-p6mgh 1/1 Running 0 6m19s odf-console-7c7d845fb-qwc66 1/1 Running 0 6m19s odf-operator-controller-manager-5ccc94dd7b-skswv 2/2 Running 0 6m19s rook-ceph-crashcollector-compute-0-755b9c4cf4-dqhgf 1/1 Running 0 88s rook-ceph-crashcollector-compute-1-5698884fdc-f8z7s 1/1 Running 0 89s rook-ceph-crashcollector-compute-2-6dc4dd7b4-xqhhw 1/1 Running 0 57s rook-ceph-exporter-compute-0-5f6768cb7b-dk2zz 1/1 Running 0 88s rook-ceph-exporter-compute-1-85676cbdd5-dm5xf 1/1 Running 0 89s rook-ceph-exporter-compute-2-7cd9d965d-rjhwp 1/1 Running 0 54s rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-657ff949lnpns 2/2 Running 0 57s rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-54f46f58nsl76 2/2 Running 0 55s rook-ceph-mgr-a-7777dc7c77-7z4tb 3/3 Running 0 89s rook-ceph-mgr-b-5d4685d7f8-p46nt 3/3 Running 0 89s rook-ceph-mon-a-8584b5768-mfw55 2/2 Running 0 2m18s rook-ceph-mon-b-5b5f99bcbd-r6vrc 2/2 Running 0 113s rook-ceph-mon-c-6d8bfdc8b4-wk8jc 2/2 Running 0 102s rook-ceph-operator-94b6546d-72hrq 1/1 Running 0 4m10s ux-backend-server-687cddc8b7-ldf72 2/2 Running 0 6m18s $ oc get secret noobaa-external-pg -o yaml apiVersion: v1 data: db_url: cG9zdGdyZXNxbDovL3Bvc3RncmVzOnBvc3RncmVzQDEwLjAuOTkuMTo1NDMyL3RpZmZhbnk= kind: Secret metadata: creationTimestamp: "2024-02-12T23:12:13Z" name: noobaa-external-pg namespace: openshift-storage resourceVersion: "80145" uid: 79c27999-b4a1-4152-acdc-187b97f42503 type: Opaque Issue #1 in https://bugzilla.redhat.com/show_bug.cgi?id=2257982#c12 is now fixed. Secret now has correct db_url. However, noobaa still doesn't deploy due to issue #2 which is tracking in this bz https://bugzilla.redhat.com/show_bug.cgi?id=2262974
Marked this issue as verified since db_url is correct with latest build.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.15.0 security, enhancement, & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:1383