Description of problem: After reinstall Submariner 0.8, without cleaning previous installation - Cluster A (AWS) showed connection established, while Cluster B (OSP) showed no endpoints found. Version-Release number of selected component (if applicable): ### OCP Cluster nmanos-cluster-a ### Client Version: 4.7.8 Server Version: 4.7.8 Kubernetes Version: v1.20.0+7d0a2b2 ### Submariner components ### subctl version: v0.8 Showing information for cluster "nmanos-cluster-a": COMPONENT REPOSITORY VERSION submariner registry.redhat.io/rhacm2-tech-preview v0.8 submariner-operator registry.redhat.io/rhacm2-tech-preview v0.8 service-discovery registry.redhat.io/rhacm2-tech-preview v0.8 ### OCP Cluster nmanos-cluster-b ### Client Version: 4.7.8 Server Version: 4.5.0 Kubernetes Version: v1.18.3+3415b61 ### Submariner components ### subctl version: v0.8 Showing information for cluster "default-cl1": COMPONENT REPOSITORY VERSION submariner registry.redhat.io/rhacm2-tech-preview v0.8 submariner-operator registry.redhat.io/rhacm2-tech-preview v0.8 service-discovery registry.redhat.io/rhacm2-tech-preview v0.8 ### Images of Pods (in namespace submariner-operator) ### ### lighthouse-agent-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/lighthouse-agent-rhel8@sha256:3890517e06ba92d29ebedfaedcfdbc1c461471076830762f2da74856add83368 name=rhacm2-tech-preview/lighthouse-agent-rhel8 release=26 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-agent-rhel8/images/v0.8-26 version=v0.8 ### lighthouse-coredns-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/lighthouse-coredns-rhel8@sha256:43ee4b341a210ec3ad96b99f548ce28d657facb8fd9150b4c617f7ca944c24eb name=rhacm2-tech-preview/lighthouse-coredns-rhel8 release=26 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-coredns-rhel8/images/v0.8-26 version=v0.8 ### submariner-gateway-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/submariner-gateway-rhel8@sha256:c9c991ec8f0c5df746a70d66e0e11fff05746e035982ec349cd37508f4fd0013 name=rhacm2-tech-preview/submariner-gateway-rhel8 release=44 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-gateway-rhel8/images/v0.8-44 version=v0.8 ### submariner-globalnet-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/submariner-globalnet-rhel8@sha256:9c9c2b3160e86d7177a08560f6a1d9974b1470dd9e4e575c6ce6eae0ff3a00ae name=rhacm2-tech-preview/submariner-globalnet-rhel8 release=39 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-globalnet-rhel8/images/v0.8-39 version=v0.8 ### submariner-rhel8-operator Image ### id=registry.redhat.io/rhacm2-tech-preview/submariner-rhel8-operator@sha256:4224be7ce0bf3b00884c14780dd8e97c7ef7162731b7ffc53e26b5e49a7c0632 name=rhacm2-tech-preview/submariner-rhel8-operator release=56 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-rhel8-operator/images/v0.8-56 version=v0.8 ### submariner-route-agent-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/submariner-route-agent-rhel8@sha256:b6f16f1a518442142a3af6732066722dd17d045e33558d93dba70b29b057cc2b name=rhacm2-tech-preview/submariner-route-agent-rhel8 release=39 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-route-agent-rhel8/images/v0.8-39 version=v0.8 How reproducible: Sometimes Steps to Reproduce: https://qe-jenkins-csb-skynet.cloud.paas.psi.redhat.com/job/Submariner-0.8-OSP-AWS-Globalnet/188/Test-Report/ Actual results: subctl show all Showing information for cluster "nmanos-cluster-a": Showing Network details Discovered network details: Network plugin: OpenShiftSDN Service CIDRs: [100.96.0.0/16] Cluster CIDRs: [10.252.0.0/14] Global CIDR: 169.254.0.0/19 Showing Endpoint details CLUSTER ID ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE nmanos-cluster-a 10.166.59.149 54.241.75.163 libreswan local default-cl1 10.166.3.205 66.187.232.129 libreswan remote Showing Connection details GATEWAY CLUSTER REMOTE IP CABLE DRIVER SUBNETS STATUS default-cl1-2nbr6-worker-8x4qr default-cl1 10.166.3.205 libreswan 169.254.32.0/19 connected Showing Gateway details NODE HA STATUS SUMMARY ip-10-166-59-149 active All connections (1) are established Showing version details COMPONENT REPOSITORY VERSION submariner registry.redhat.io/rhacm2-tech-preview v0.8 submariner-operator registry.redhat.io/rhacm2-tech-preview v0.8 service-discovery registry.redhat.io/rhacm2-tech-preview v0.8 Showing information for cluster "default-cl1": Showing Network details Discovered network details: Network plugin: OpenShiftSDN Service CIDRs: [100.96.0.0/16] Cluster CIDRs: [10.252.0.0/14] Global CIDR: 169.254.32.0/19 Showing Endpoint details No endpoints found subctl version: v0.8 Expected results: Cluster should be connected + Endpoints on both clusters should be found. Additional info: In submariner-operator pod on Cluster B there's an error: Operation cannot be fulfilled on submariners.submariner.io \"submariner\": the object has been modified; please apply your changes to the latest version and try again 2021-04-28T07:43:49.638692084Z 2021-04-28T07:43:49.638Z INFO controller_submariner Reconciling Submariner {"Request.Namespace": "submariner-operator", "Request.Name": "submariner"} 2021-04-28T07:43:49.639233830Z 2021-04-28T07:43:49.639Z ERROR controller_submariner failed to update the Submariner status {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "error": "Operation cannot be fulfilled on submariners.submariner.io \"submariner\": the object has been modified; please apply your changes to the latest version and try again"} 2021-04-28T07:43:49.639233830Z github.com/go-logr/zapr.(*zapLogger).Error 2021-04-28T07:43:49.639233830Z /remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr.1/zapr.go:128 2021-04-28T07:43:49.639233830Z github.com/submariner-io/submariner-operator/controllers/submariner.(*SubmarinerReconciler).Reconcile 2021-04-28T07:43:49.639233830Z /remote-source/app/controllers/submariner/submariner_controller.go:211 2021-04-28T07:43:49.639233830Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler 2021-04-28T07:43:49.639233830Z /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.9/pkg/internal/controller/controller.go:246 2021-04-28T07:43:49.639233830Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem 2021-04-28T07:43:49.639233830Z /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.9/pkg/internal/controller/controller.go:222 2021-04-28T07:43:49.639233830Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker 2021-04-28T07:43:49.639233830Z /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.9/pkg/internal/controller/controller.go:201 2021-04-28T07:43:49.639233830Z k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 2021-04-28T07:43:49.639233830Z /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.0/pkg/util/wait/wait.go:152 2021-04-28T07:43:49.639233830Z k8s.io/apimachinery/pkg/util/wait.JitterUntil 2021-04-28T07:43:49.639233830Z /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.0/pkg/util/wait/wait.go:153 2021-04-28T07:43:49.639233830Z k8s.io/apimachinery/pkg/util/wait.Until 2021-04-28T07:43:49.639233830Z /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.0/pkg/util/wait/wait.go:88 2021-04-28T07:43:49.639275101Z 2021-04-28T07:43:49.639Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "submariner-controller", "name": "submariner", "namespace": "submariner-operator"} 2021-04-28T07:43:49.639275101Z 2021-04-28T07:43:49.639Z INFO controller_submariner Reconciling Submariner {"Request.Namespace": "submariner-operator", "Request.Name": "submariner"} 2021-04-28T07:43:49.649158970Z 2021-04-28T07:43:49.648Z INFO controller_submariner Using detected CIDR {"type": "Cluster", "CIDR": "10.252.0.0/14"} 2021-04-28T07:43:49.649158970Z 2021-04-28T07:43:49.648Z INFO controller_submariner Using detected CIDR {"type": "Service", "CIDR": "100.96.0.0/16"} 2021-04-28T07:43:49.649158970Z 2021-04-28T07:43:49.648Z INFO controller_submariner Using detected CIDR {"type": "Cluster", "CIDR": "10.252.0.0/14"} 2021-04-28T07:43:49.649158970Z 2021-04-28T07:43:49.648Z INFO controller_submariner Using detected CIDR {"type": "Service", "CIDR": "100.96.0.0/16"} 2021-04-28T07:43:49.657187559Z 2021-04-28T07:43:49.657Z INFO controller_submariner Updated existing DaemonSet {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-gateway"} 2021-04-28T07:43:49.657723972Z 2021-04-28T07:43:49.657Z INFO controller_submariner Updated existing DaemonSet {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-gateway"} 2021-04-28T07:43:49.662466643Z 2021-04-28T07:43:49.662Z INFO controller_submariner Updated existing Service {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "Service.Namespace": "submariner-operator", "Service.Name": "submariner-engine-metrics"} 2021-04-28T07:43:49.666145003Z 2021-04-28T07:43:49.666Z INFO controller_submariner Updated existing Service {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "Service.Namespace": "submariner-operator", "Service.Name": "submariner-engine-metrics"} 2021-04-28T07:43:52.524617428Z 2021-04-28T07:43:52.524Z INFO controller_submariner Could not create ServiceMonitor object {"error": "servicemonitors.monitoring.coreos.com \"submariner-engine-metrics\" already exists"} 2021-04-28T07:43:52.526584173Z 2021-04-28T07:43:52.526Z INFO controller_submariner Could not create ServiceMonitor object {"error": "servicemonitors.monitoring.coreos.com \"submariner-engine-metrics\" already exists"} 2021-04-28T07:43:52.532511698Z 2021-04-28T07:43:52.532Z INFO controller_submariner Updated existing DaemonSet {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-routeagent"} 2021-04-28T07:43:52.533324899Z 2021-04-28T07:43:52.533Z INFO controller_submariner Updated existing DaemonSet {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-routeagent"} 2021-04-28T07:43:52.538102340Z 2021-04-28T07:43:52.538Z INFO controller_submariner Updated existing DaemonSet {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-globalnet"} 2021-04-28T07:43:52.538779806Z 2021-04-28T07:43:52.538Z INFO controller_submariner Updated existing DaemonSet {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-globalnet"} 2021-04-28T07:43:52.548490948Z 2021-04-28T07:43:52.548Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "submariner", "name": "submariner", "namespace": "submariner-operator"} 2021-04-28T07:43:52.549802394Z 2021-04-28T07:43:52.549Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "submariner-controller", "name": "submariner", "namespace": "submariner-operator"} 2021-04-28T07:43:53.852547325Z 2021-04-28T07:43:53.852Z INFO controller_submariner Reconciling Submariner {"Request.Namespace": "submariner-operator", "Request.Name": "submariner"} 2021-04-28T07:43:53.862444668Z 2021-04-28T07:43:53.862Z INFO controller_submariner Using detected CIDR {"type": "Cluster", "CIDR": "10.252.0.0/14"} 2021-04-28T07:43:53.862444668Z 2021-04-28T07:43:53.862Z INFO controller_submariner Using detected CIDR {"type": "Service", "CIDR": "100.96.0.0/16"} 2021-04-28T07:43:53.871374229Z 2021-04-28T07:43:53.871Z INFO controller_submariner Updated existing DaemonSet {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-gateway"} 2021-04-28T07:43:53.878492146Z 2021-04-28T07:43:53.878Z INFO controller_submariner Updated existing Service {"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "Service.Namespace": "submariner-operator", "Service.Name": "submariner-engine-metrics"}
Hi Noam, Can you give it a try with 0.9? If this is something specific to 0.8, I don't see the urgency to fix this - given that this is a tech preview release. Thanks, Nir
@nmanos I tested on 2 AWS cluster and couldn't reproduce this issue. Do we still have this problem on v0.8 ? Can you tell if the same scenario happens on v0.9 ?
Installing 0.9 on an existing 0.9 deployment did not encounter the described problem: https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/Submariner-0.9-AWSx2-OVN/96/Test-Report/
Also tested upgrade from Submariner 0.9 (d/s) to Submariner 0.10 (subctl-devel), and the issue was not reproduced: https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/Submariner-0.9-OSP-AWS-SDN/89/Test-Report/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.2.4 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2461