Bug 2223780

Description of problem (please be detailed as possible and provide log
snippests):
1.Installed a cluster with Multus
2.After reset all the pods in openshift-storage namespace, I found a connection issue to  Noobaa DB
3.Tested the same procedure on the cluster without Multus, and everything worked as expected [Storagecluster moved to Ready state]
4.noobaa-core-0 pod in Running state althoght we got this error:

Jul-18 14:51:49.698 [Upgrade/20] [ERROR] core.util.postgres_client:: _connect: initial connect failed, will retry connect EHOSTUNREACH 10.128.2.30:5432
Jul-18 14:51:52.698 [Upgrade/20]    [L0] core.util.postgres_client:: _connect: called with { max: 10, host: 'noobaa-db-pg-0.noobaa-db-pg', user: 'noobaa', password: 'arrvkPEp/3MbXA==', database: 'nbcore', port: 5432 }
Jul-18 14:51:55.778 [Upgrade/20] [ERROR] core.util.postgres_client:: apply_sql_functions execute error Error: connect EHOSTUNREACH 10.128.2.30:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
  errno: -113,
  code: 'EHOSTUNREACH',
  syscall: 'connect',
  address: '10.128.2.30',
  port: 5432
}

Version of all relevant components (if applicable):
ODF Version: 4.13.0-218
OCP Version: 4.13.0-0.nightly-2023-07-18-041822
Plaform: BM

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Install LSO4.13
2.Install ODF4.13 with multus
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: public-net
 namespace: openshift-storage
 labels: {}
 annotations: {}
spec:
 config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "enp1s0f1", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.20.0/24" } }'
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: cluster-net
 namespace: openshift-storage
 labels: {}
 annotations: {}
spec:
  config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "enp1s0f1", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } }'

3.Verify storagecluster on Ready state

4.Verify ceph statsu is OK

5.Restart all pods in openshift-storage
$ oc delete pods --all -n openshift-storage
pod "csi-addons-controller-manager-7998b997-d6d8m" deleted
pod "csi-cephfsplugin-fl749" deleted

6.Check storagecluster status -> [stuck on Progressing state]
$ oc get storagecluster
NAME                 AGE   PHASE         EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   19m   Progressing              2023-07-18T14:28:43Z   4.13.0


    Status:                True
    Type:                  Available
    Last Heartbeat Time:   2023-07-18T14:48:17Z
    Last Transition Time:  2023-07-18T14:39:32Z
    Message:               Waiting on Nooba instance to finish initialization
    Reason:                NoobaaInitializing
    Status:                True

7. Check noobaa pods on openshift-storage namespace
$ oc get pods -l app=noobaa
NAME                               READY   STATUS    RESTARTS   AGE
noobaa-core-0                      1/1     Running   0          9m15s
noobaa-db-pg-0                     1/1     Running   0          8m44s
noobaa-endpoint-69c754f649-hgjmv   1/1     Running   0          9m45s
noobaa-operator-897469f66-6ghkl    1/1     Running   0          9m45s

8. Although the noobaa-core-0 pod is in Running state, there is a connection issue to Noobaa DB.
$ oc logs noobaa-core-0 
Jul-18 14:51:49.698 [Upgrade/20] [ERROR] core.util.postgres_client:: _connect: initial connect failed, will retry connect EHOSTUNREACH 10.128.2.30:5432
Jul-18 14:51:52.698 [Upgrade/20]    [L0] core.util.postgres_client:: _connect: called with { max: 10, host: 'noobaa-db-pg-0.noobaa-db-pg', user: 'noobaa', password: 'arrvkPEp/3MbXA==', database: 'nbcore', port: 5432 }
Jul-18 14:51:55.778 [Upgrade/20] [ERROR] core.util.postgres_client:: apply_sql_functions execute error Error: connect EHOSTUNREACH 10.128.2.30:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
  errno: -113,
  code: 'EHOSTUNREACH',
  syscall: 'connect',
  address: '10.128.2.30',
  port: 5432
}
Jul-18 14:51:55.778 [Upgrade/20] [ERROR] core.util.postgres_client:: _connect: initial connect failed, will retry connect EHOSTUNREACH 10.128.2.30:5432
Jul-18 14:51:58.779 [Upgrade/20]    [L0] core.util.postgres_client:: _connect: called with { max: 10, host: 'noobaa-db-pg-0.noobaa-db-pg', user: 'noobaa', password: 'arrvkPEp/3MbXA==', database: 'nbcore', port: 5432 }
Jul-18 14:51:58.850 [Upgrade/20] [ERROR] core.util.postgres_client:: apply_sql_functions execute error Error: connect EHOSTUNREACH 10.128.2.30:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
  errno: -113,
  code: 'EHOSTUNREACH',
  syscall: 'connect',
  address: '10.128.2.30',
  port: 5432
}



Actual results:
Connection issue to noobaa DB

Expected results:
Tested the same procedure on the cluster without Multus, and everything worked as expected [Storagecluster moved to Ready state]

Additional info:
OCS MG:
http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-2223780.tar.gz