Live cluster is available for debugging

c1 - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/29300/

c2 - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/29301/

hub - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/29299/

This bz and the other CephFS based bz https://bugzilla.redhat.com/show_bug.cgi?id=2241131 were hit around same time

Rook has incorrectly assigned `242.1.255.254` exported IP as the `public_addr` for rook-ceph-osd-0 pod in `c2` cluster. 

The service created by globalnet uses for osd-0 has `242.1.255.248` as the external (exported) IP.

```
❯ oc get service submariner-zoh2rvm7zupu5kldmqt6mduidrg6wek2 -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-09-23T00:41:35Z"
  finalizers:
  - submariner.io/globalnet-internal-service
  labels:
    submariner.io/exportedServiceRef: rook-ceph-osd-0
  name: submariner-zoh2rvm7zupu5kldmqt6mduidrg6wek2
  namespace: openshift-storage
  resourceVersion: "13827635"
  uid: 8b85c64e-c60c-4db6-aa1c-e9f7417890ef
spec:
  clusterIP: 172.30.60.17
  clusterIPs:
  - 172.30.60.17
  externalIPs:
  - 242.1.255.248
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: osd-port-v2
    port: 6800
    protocol: TCP
    targetPort: 6800
  selector:
    app: rook-ceph-osd
    ceph-osd-id: "0"
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
```

Running the nslookup on `kmanohar-long-c2.rook-ceph-osd-0.openshift-storage.svc.clusterset.local` returned `242.1.255.254` as the IP for `rook-ceph-osd-0` clusterset. So rook assigned `242.1.255.254` as the public-addr for rook-ceph-osd-0 pod. 

Rook logs:

```
2023-09-23 20:56:53.699249 I | op-osd: OSD will have its main bluestore block on "ocs-deviceset-thin-csi-odf-2-data-0vvp2c"
2023-09-23 20:56:53.846600 I | op-osd: osd.0 cluster IP is "172.30.241.158"
2023-09-23 20:56:53.889365 I | op-osd: osd.0 exported IP is "242.1.255.254"    ### Need to check by nslookup is returning `242.1.255.254` instead of `242.1.255.248`
2023-09-23 20:56:53.889588 I | op-osd: assigning osd 0 topology affinity to "topology.rook.io/rack=rack0"
2023-09-23 20:56:54.368601 I | op-mgr: successful modules: mgr module(s) from the spec```
```


Rook ceph osd pod:
```  oc get pods rook-ceph-osd-0-5747994569-4n7f9 -o yaml | grep public-addr
    - --public-addr=242.1.255.254

```

Sridhar, any idea why nslookup is not returning the expected external IP here. 

```
❯ kubectl -n openshift-storage run tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.0# nslookup kmanohar-long-c2.rook-ceph-osd-0.openshift-storage.svc.clusterset.local
Server:		172.30.0.10
Address:	172.30.0.10#53

Name:	kmanohar-long-c2.rook-ceph-osd-0.openshift-storage.svc.clusterset.local
Address: 242.1.255.254

bash-5.0# 

```

nsLookup returns `242.1.255.254` for the exported service `rook-ceph-osd-0`, but the exported service has `242.1.255.248` as the external service
```
❯ oc get service submariner-zoh2rvm7zupu5kldmqt6mduidrg6wek2 -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-09-23T00:41:35Z"
  finalizers:
  - submariner.io/globalnet-internal-service
  labels:
    submariner.io/exportedServiceRef: rook-ceph-osd-0
  name: submariner-zoh2rvm7zupu5kldmqt6mduidrg6wek2
  namespace: openshift-storage
  resourceVersion: "13827635"
  uid: 8b85c64e-c60c-4db6-aa1c-e9f7417890ef
spec:
  clusterIP: 172.30.60.17
  clusterIPs:
  - 172.30.60.17
  externalIPs:
  - 242.1.255.248
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: osd-port-v2
    port: 6800
    protocol: TCP
    targetPort: 6800
  selector:
    app: rook-ceph-osd
    ceph-osd-id: "0"
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
```

Globalnet logs:

```
[90m2023-09-23T20:34:02.228Z[0m [32mINF[0m ..essip_controller.go:129 Globalnet            Processing created ClusterGlobalEgressIP "cluster-egress.submariner.io", Spec.NumberOfIPs: 8, Status: v1.GlobalEgressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2023, time.September, 12, 13, 21, 46, 0, time.Local), Reason:"Success", Message:"Allocated 8 global IP(s)"}}, AllocatedIPs:[]string{"242.1.0.1", "242.1.0.2", "242.1.0.3", "242.1.0.4", "242.1.0.5", "242.1.0.6", "242.1.0.7", "242.1.0.8"}}
[90m2023-09-23T20:34:02.228Z[0m [33mDBG[0m ..essip_controller.go:194 Globalnet            Update called for "cluster-egress.submariner.io", but numberOfIPs 8 are already allocated
[90m2023-09-23T20:34:02.300Z[0m [32mINF[0m ../node_controller.go:114 Globalnet            Processing created Node "compute-2"
[90m2023-09-23T20:34:02.300Z[0m [32mINF[0m ../node_controller.go:144 Globalnet            Allocated global IP 242.1.255.254 for node "compute-2"
[90m2023-09-23T20:34:02.300Z[0m [32mINF[0m ../node_controller.go:147 Globalnet            Adding ingress rules for node "compute-2" with global IP 242.1.255.254, CNI IP 10.129.2.1
[90m2023-09-23T20:34:02.300Z[0m [33mDBG[0m ..s/iptables/iface.go:184 IPTables             Installing iptable ingress rules for Node: -p icmp -d 242.1.255.254 -j DNAT --to 10.129.2.1
[90m2023-09-23T20:34:02.528Z[0m [32mINF[0m ..essip_controller.go:138 Globalnet            Processing created openshift-storage/rook-ceph-mon-d, TargetRef: "ClusterIPService", "rook-ceph-mon-d", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2023, time.September, 13, 4, 20, 5, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.1.255.251"}
[90m2023-09-23T20:34:02.529Z[0m [32mINF[0m ..essip_controller.go:138 Globalnet            Processing created openshift-storage/rook-ceph-mon-f, TargetRef: "ClusterIPService", "rook-ceph-mon-f", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2023, time.September, 13, 4, 25, 19, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.1.255.249"}
[90m2023-09-23T20:34:02.529Z[0m [32mINF[0m ..essip_controller.go:138 Globalnet            Processing created openshift-storage/rook-ceph-mon-g, TargetRef: "ClusterIPService", "rook-ceph-mon-g", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2023, time.September, 23, 13, 46, 30, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.1.255.254"}
[90m2023-09-23T20:34:02.529Z[0m [32mINF[0m ..essip_controller.go:138 Globalnet            Processing created openshift-storage/rook-ceph-osd-0, TargetRef: "ClusterIPService", "rook-ceph-osd-0", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2023, time.September, 23, 0, 41, 35, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.1.255.248"}
[90m2023-09-23T20:34:02.529Z[0m [32mINF[0m ..essip_controller.go:138 Globalnet            Processing created openshift-storage/rook-ceph-osd-1, TargetRef: "ClusterIPService", "rook-ceph-osd-1", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2023, time.September, 13, 4, 18, 3, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.1.255.253"}
[90m2023-09-23T20:34:02.529Z[0m [32mINF[0m ..essip_controller.go:138 Globalnet            Processing created openshift-storage/rook-ceph-osd-2, TargetRef: "ClusterIPService", "rook-ceph-osd-2", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2023, time.September, 13, 4, 18, 32, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.1.255.252"}
[90m2023-09-23T20:34:02.933Z[0m [32mINF[0m ../gateway_monitor.go:351 Globalnet            Successfully started the controllers
```

(In reply to Santosh Pillai from comment #10)
> Sridhar, any idea why nslookup is not returning the expected external IP
> here. 

It looks like the actual IP assigned to the exported service is 242.1.255.248, but Lighthouse seems to be resolving it to 242.1.255.254.
We would need the logs of Lighthouse component to understand if there is any error. Can you please attach the output of "subctl gather..." from the cluster?

(In reply to Sridhar Gaddam from comment #11)
> (In reply to Santosh Pillai from comment #10)
> > Sridhar, any idea why nslookup is not returning the expected external IP
> > here. 
> 
> It looks like the actual IP assigned to the exported service is
> 242.1.255.248, but Lighthouse seems to be resolving it to 242.1.255.254.
> We would need the logs of Lighthouse component to understand if there is any
> error. Can you please attach the output of "subctl gather..." from the
> cluster?

attached submariner logs from c2 cluster in comment #12

Okay, I had a look at the logs and identified the root cause of the issue.

The ODF clusters are using multiple Gateway nodes and the clusters have been running for more than 20+ days. During this period, there were couple of times where the Active Gateway migrated from one node to the other. From the logs, we noticed that during GW migration, the submariner-globalnet controller on the passive GW nodes was running for a few seconds before it was stopped. During this interval, the submariner-globalnet controller on the newly elected ACTIVE gateway node has already started running. So we have a brief interval where two instances of submariner-globalnet controllers were active and trying to process the globalIPs for various objects like Nodes, exported services etc. Because of this race condition, some of the globalIPs allocated to the exported services were updated to new ones. More specifically, the globalIP assigned to rook-ceph-osd-0 changed from 242.1.255.254 to 242.1.255.248. 

Now when an application is trying to resolve the globalIP associated to the service rook-ceph-osd-0, its processed by Submariner Lighthouse component.
The Lighthouse component was assuming that globalIPs once assigned to an exported service will not be updated until the lifetime of the service. 
Because of this, Lighthouse was returning the old globalIP which is no longer valid and hence mirroring was not happening (as they were trying to connect to wrong global IP). 

I pushed the following fix which will partially fix this issue - https://github.com/submariner-io/submariner/pull/2739
But in parallel, we are also working on a proper fix to avoid this race condition all-together.

PS:
Apart from the issue I mentioned above, there is no issue with the connections between the Gateway nodes. You will notice that if you run "subctl verify ..." all the e2e tests would pass.
The issue is seen mainly because you have some existing exported services in the cluster and Gateway migration triggered a race condition that updated the globalIPs of the exported service.

Clearing need info on me as an ACM bug https://issues.redhat.com/browse/ACM-7844 has been raised, and a PR too https://github.com/submariner-io/submariner/pull/2743 is under review.

ACM bug is already ON QA

Verification was done for ACM bug. Moving the bug to verified.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days