Bug 2257982

Summary: Noobaa system failed to create using external postgres - db_url link is not correct
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Tiffany Nguyen <tunguyen>
Component: management-consoleAssignee: Vineet <vbadrina>
Status: CLOSED ERRATA QA Contact: Tiffany Nguyen <tunguyen>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.15CC: jalbo, nbecker, ngowda, nigoyal, nthomas, odf-bz-bot, skatiyar, usrivast, vbadrina
Target Milestone: ---   
Target Release: ODF 4.15.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.15.0-135 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2262974 (view as bug list) Environment:
Last Closed: 2024-03-19 15:31:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2262974    

Description Tiffany Nguyen 2024-01-11 22:51:03 UTC
Description of problem (please be detailed as possible and provide log
snippests):

Using storage system wizard to create a storage system with external postgres.  Noobaa-core and noobaa-db pods failed to create. 

Not all the pods are created and running:
$ oc get pod
NAME                                                              READY   STATUS    RESTARTS      AGE
csi-addons-controller-manager-5dbfb55df9-85wxv                    2/2     Running   0             14h
csi-cephfsplugin-9xnsz                                            2/2     Running   0             14h
csi-cephfsplugin-provisioner-58c69cfb78-7h5tv                     6/6     Running   0             14h
csi-cephfsplugin-provisioner-58c69cfb78-ccnng                     6/6     Running   1 (14h ago)   14h
csi-cephfsplugin-s2ngm                                            2/2     Running   0             14h
csi-cephfsplugin-vbf98                                            2/2     Running   0             14h
csi-rbdplugin-8s279                                               3/3     Running   0             14h
csi-rbdplugin-p9rn5                                               3/3     Running   0             14h
csi-rbdplugin-provisioner-d65774655-d5vtk                         6/6     Running   0             14h
csi-rbdplugin-provisioner-d65774655-lq24w                         6/6     Running   0             14h
csi-rbdplugin-tbr2v                                               3/3     Running   0             14h
noobaa-operator-68b69cd44b-vdszf                                  2/2     Running   0             14h
ocs-operator-859d787c7-vzzgf                                      1/1     Running   0             14h
odf-console-8485dc45db-wpv28                                      1/1     Running   0             14h
odf-operator-controller-manager-64fbbbdc4d-j25c6                  2/2     Running   0             14h
rook-ceph-crashcollector-tunguyen-111p-szz92-worker-1-7qp5lv872   1/1     Running   0             14h
rook-ceph-crashcollector-tunguyen-111p-szz92-worker-2-zfcrmjs8s   1/1     Running   0             14h
rook-ceph-crashcollector-tunguyen-111p-szz92-worker-3-dxr7dhnjj   1/1     Running   0             14h
rook-ceph-exporter-tunguyen-111p-szz92-worker-1-7qp5j-7769pzhlc   1/1     Running   0             14h
rook-ceph-exporter-tunguyen-111p-szz92-worker-2-zfcrm-bf5689l5n   1/1     Running   0             14h
rook-ceph-exporter-tunguyen-111p-szz92-worker-3-dxr7h-6b47b8jvd   1/1     Running   0             14h
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-fd986dd9sz762   2/2     Running   0             14h
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-57f86548mc7z7   2/2     Running   0             14h
rook-ceph-mgr-a-56747bb7c5-drh59                                  3/3     Running   0             14h
rook-ceph-mgr-b-59d95b9d88-h4d9h                                  3/3     Running   0             14h
rook-ceph-mon-a-64d7c864cc-wctc5                                  2/2     Running   0             14h
rook-ceph-mon-b-55f5c4696d-xrkqs                                  2/2     Running   0             14h
rook-ceph-mon-c-7dddc877c5-z2857                                  2/2     Running   0             14h
rook-ceph-operator-b8cf888cf-jldx9                                1/1     Running   0             14h
ux-backend-server-695548595d-mjtzc                                2/2     Running   0             14h


Version of all relevant components (if applicable):
ODF 4.15 build 4.15.0-112
$ oc get csv -n openshift-storage
NAME                                         DISPLAY                       VERSION             REPLACES   PHASE
mcg-operator.v4.15.0-112.stable              NooBaa Operator               4.15.0-112.stable              Succeeded
ocs-operator.v4.15.0-112.stable              OpenShift Container Storage   4.15.0-112.stable              Succeeded
odf-csi-addons-operator.v4.15.0-112.stable   CSI Addons                    4.15.0-112.stable              Succeeded
odf-operator.v4.15.0-112.stable              OpenShift Data Foundation     4.15.0-112.stable              Succeeded


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, happy path testing is failing for epic https://issues.redhat.com/browse/RHSTOR-4749


Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
Yes


Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
No

Steps to Reproduce:
1. Deploy an OCP cluster
2. Install ODF build 4.15.0-112
3. Create storage system using storage system wizard
4. Select external postgres and input the database connection info
5. Complete the wizard and check for the installation progress


Actual results:
Storage system failed to create, noobaa pods failed to create.



Expected results:
Storage system, noobaa pods should create and running without any issue.


Additional info:
Must gather logs: http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-2257982/
Failing cluster is available for investigate.

Comment 6 Tiffany Nguyen 2024-01-17 23:00:34 UTC
I installed CRD manually using below command:

$ kubectl create -k github.com/kubernetes-sigs/container-object-storage-interface-api
customresourcedefinition.apiextensions.k8s.io/bucketaccessclasses.objectstorage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/bucketaccesses.objectstorage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/bucketclaims.objectstorage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/bucketclasses.objectstorage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/buckets.objectstorage.k8s.io created

However, noobaa-db and noobaa-core pods are not created:

$ oc get pod | grep noobaa
noobaa-operator-798cd44446-hgwpq                                  2/2     Running   0          46m

Comment 7 krishnaram Karthick 2024-01-22 08:08:53 UTC
(In reply to Tiffany Nguyen from comment #6)
> I installed CRD manually using below command:
> 
> $ kubectl create -k
> github.com/kubernetes-sigs/container-object-storage-interface-api
> customresourcedefinition.apiextensions.k8s.io/bucketaccessclasses.
> objectstorage.k8s.io created
> customresourcedefinition.apiextensions.k8s.io/bucketaccesses.objectstorage.
> k8s.io created
> customresourcedefinition.apiextensions.k8s.io/bucketclaims.objectstorage.k8s.
> io created
> customresourcedefinition.apiextensions.k8s.io/bucketclasses.objectstorage.
> k8s.io created
> customresourcedefinition.apiextensions.k8s.io/buckets.objectstorage.k8s.io
> created
> 
> However, noobaa-db and noobaa-core pods are not created:
> 
> $ oc get pod | grep noobaa
> noobaa-operator-798cd44446-hgwpq                                  2/2    
> Running   0          46m

Jacky, could you pls take a look?

Comment 12 Tiffany Nguyen 2024-02-05 23:11:34 UTC
Verifing the fix using build 4.15.0-130.  The storagecluster is now getting "externalPgConfig" and "pgSecretName". However, there are few more issues are seen when configure an external postgres.  As result, storagecluster doesn't deploy correctly and noobaa is not get created.

1. In the secret, "db_url" is incorrect.
   Provide link: postgres://postgres:postgres.99.1.namespace.svc:5432/Tiffany
   Correct link: postgresql://postgres:postgres.99.1:5432/tiffany

2. externalPgSSLRequired is set to "true" in noobaa.yaml when there is no SecureSSL database selected. This is causing the database connection error.

Comment 13 Vineet 2024-02-06 11:30:24 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=2262974 - Created clone for this bz on OCS operator as well. May need to move them both on QA

Comment 15 Vineet 2024-02-08 11:53:22 UTC
No doc changes required.

Comment 17 Tiffany Nguyen 2024-02-12 23:45:25 UTC
Test with ODF 4.15 build 139:
$ oc get csv -A
NAMESPACE                              NAME                                         DISPLAY                       VERSION             REPLACES   PHASE
openshift-operator-lifecycle-manager   packageserver                                Package Server                0.0.1-snapshot                 Succeeded
openshift-storage                      mcg-operator.v4.15.0-139.stable              NooBaa Operator               4.15.0-139.stable              Succeeded
openshift-storage                      ocs-operator.v4.15.0-139.stable              OpenShift Container Storage   4.15.0-139.stable              Succeeded
openshift-storage                      odf-csi-addons-operator.v4.15.0-139.stable   CSI Addons                    4.15.0-139.stable              Succeeded
openshift-storage                      odf-operator.v4.15.0-139.stable              OpenShift Data Foundation     4.15.0-139.stable              Succeeded

$ oc get pod
NAME                                                              READY   STATUS    RESTARTS        AGE
csi-addons-controller-manager-555bcf9c9d-n5v8g                    2/2     Running   0               5m57s
csi-cephfsplugin-dszbm                                            2/2     Running   0               4m11s
csi-cephfsplugin-provisioner-5b5575d8d5-2j4kx                     6/6     Running   0               4m11s
csi-cephfsplugin-provisioner-5b5575d8d5-pjdtn                     6/6     Running   0               4m11s
csi-cephfsplugin-sfmss                                            2/2     Running   0               4m11s
csi-cephfsplugin-v2q6l                                            2/2     Running   1 (3m22s ago)   4m11s
csi-rbdplugin-64k5k                                               3/3     Running   0               4m11s
csi-rbdplugin-fscs6                                               3/3     Running   0               4m11s
csi-rbdplugin-k7w7n                                               3/3     Running   1 (3m22s ago)   4m11s
csi-rbdplugin-provisioner-df8895f7b-qxmcb                         6/6     Running   0               4m11s
csi-rbdplugin-provisioner-df8895f7b-sc6f5                         6/6     Running   4 (2m36s ago)   4m11s
noobaa-operator-7cccc64c59-mf6nf                                  2/2     Running   0               6m17s
ocs-operator-5bc895b594-p6mgh                                     1/1     Running   0               6m19s
odf-console-7c7d845fb-qwc66                                       1/1     Running   0               6m19s
odf-operator-controller-manager-5ccc94dd7b-skswv                  2/2     Running   0               6m19s
rook-ceph-crashcollector-compute-0-755b9c4cf4-dqhgf               1/1     Running   0               88s
rook-ceph-crashcollector-compute-1-5698884fdc-f8z7s               1/1     Running   0               89s
rook-ceph-crashcollector-compute-2-6dc4dd7b4-xqhhw                1/1     Running   0               57s
rook-ceph-exporter-compute-0-5f6768cb7b-dk2zz                     1/1     Running   0               88s
rook-ceph-exporter-compute-1-85676cbdd5-dm5xf                     1/1     Running   0               89s
rook-ceph-exporter-compute-2-7cd9d965d-rjhwp                      1/1     Running   0               54s
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-657ff949lnpns   2/2     Running   0               57s
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-54f46f58nsl76   2/2     Running   0               55s
rook-ceph-mgr-a-7777dc7c77-7z4tb                                  3/3     Running   0               89s
rook-ceph-mgr-b-5d4685d7f8-p46nt                                  3/3     Running   0               89s
rook-ceph-mon-a-8584b5768-mfw55                                   2/2     Running   0               2m18s
rook-ceph-mon-b-5b5f99bcbd-r6vrc                                  2/2     Running   0               113s
rook-ceph-mon-c-6d8bfdc8b4-wk8jc                                  2/2     Running   0               102s
rook-ceph-operator-94b6546d-72hrq                                 1/1     Running   0               4m10s
ux-backend-server-687cddc8b7-ldf72                                2/2     Running   0               6m18s


$ oc get secret noobaa-external-pg -o yaml
apiVersion: v1
data:
  db_url: cG9zdGdyZXNxbDovL3Bvc3RncmVzOnBvc3RncmVzQDEwLjAuOTkuMTo1NDMyL3RpZmZhbnk=
kind: Secret
metadata:
  creationTimestamp: "2024-02-12T23:12:13Z"
  name: noobaa-external-pg
  namespace: openshift-storage
  resourceVersion: "80145"
  uid: 79c27999-b4a1-4152-acdc-187b97f42503
type: Opaque

Issue #1 in https://bugzilla.redhat.com/show_bug.cgi?id=2257982#c12 is now fixed.  Secret now has correct db_url. 
However, noobaa still doesn't deploy due to issue #2 which is tracking in this bz https://bugzilla.redhat.com/show_bug.cgi?id=2262974

Comment 18 Tiffany Nguyen 2024-02-13 05:20:18 UTC
Marked this issue as verified since db_url is correct with latest build.

Comment 21 errata-xmlrpc 2024-03-19 15:31:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.15.0 security, enhancement, & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:1383