Bug 1954535

Summary: Reinstall Submariner - No endpoints found on one cluster
Product: Red Hat Advanced Cluster Management for Kubernetes Reporter: Noam Manos <nmanos>
Component: SubmarinerAssignee: Nir Yechiel <nyechiel>
Status: CLOSED ERRATA QA Contact: Noam Manos <nmanos>
Severity: medium Docs Contact: Christopher Dawson <cdawson>
Priority: low    
Version: rhacm-2.2CC: smattar, tfreger
Target Milestone: ---Flags: ming: rhacm-2.2.z+
Target Release: rhacm-2.2.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-16 19:28:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Noam Manos 2021-04-28 10:54:26 UTC 
Description of problem:
After reinstall Submariner 0.8, without cleaning previous installation - Cluster A (AWS) showed connection established, while Cluster B (OSP) showed no endpoints found.

Version-Release number of selected component (if applicable):

### OCP Cluster nmanos-cluster-a ###
Client Version: 4.7.8
Server Version: 4.7.8
Kubernetes Version: v1.20.0+7d0a2b2

### Submariner components ###

subctl version: v0.8

Showing information for cluster "nmanos-cluster-a":
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.8            
submariner-operator             registry.redhat.io/rhacm2-tech-preview                v0.8            
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.8            

### OCP Cluster nmanos-cluster-b ###
Client Version: 4.7.8
Server Version: 4.5.0
Kubernetes Version: v1.18.3+3415b61

### Submariner components ###

subctl version: v0.8

Showing information for cluster "default-cl1":
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.8            
submariner-operator             registry.redhat.io/rhacm2-tech-preview                v0.8            
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.8            

### Images of Pods (in namespace submariner-operator) ###

### lighthouse-agent-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/lighthouse-agent-rhel8@sha256:3890517e06ba92d29ebedfaedcfdbc1c461471076830762f2da74856add83368
name=rhacm2-tech-preview/lighthouse-agent-rhel8
release=26
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-agent-rhel8/images/v0.8-26
version=v0.8

### lighthouse-coredns-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/lighthouse-coredns-rhel8@sha256:43ee4b341a210ec3ad96b99f548ce28d657facb8fd9150b4c617f7ca944c24eb
name=rhacm2-tech-preview/lighthouse-coredns-rhel8
release=26
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-coredns-rhel8/images/v0.8-26
version=v0.8

### submariner-gateway-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-gateway-rhel8@sha256:c9c991ec8f0c5df746a70d66e0e11fff05746e035982ec349cd37508f4fd0013
name=rhacm2-tech-preview/submariner-gateway-rhel8
release=44
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-gateway-rhel8/images/v0.8-44
version=v0.8

### submariner-globalnet-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-globalnet-rhel8@sha256:9c9c2b3160e86d7177a08560f6a1d9974b1470dd9e4e575c6ce6eae0ff3a00ae
name=rhacm2-tech-preview/submariner-globalnet-rhel8
release=39
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-globalnet-rhel8/images/v0.8-39
version=v0.8

### submariner-rhel8-operator Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-rhel8-operator@sha256:4224be7ce0bf3b00884c14780dd8e97c7ef7162731b7ffc53e26b5e49a7c0632
name=rhacm2-tech-preview/submariner-rhel8-operator
release=56
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-rhel8-operator/images/v0.8-56
version=v0.8

### submariner-route-agent-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-route-agent-rhel8@sha256:b6f16f1a518442142a3af6732066722dd17d045e33558d93dba70b29b057cc2b
name=rhacm2-tech-preview/submariner-route-agent-rhel8
release=39
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-route-agent-rhel8/images/v0.8-39
version=v0.8


How reproducible:
Sometimes

Steps to Reproduce:
https://qe-jenkins-csb-skynet.cloud.paas.psi.redhat.com/job/Submariner-0.8-OSP-AWS-Globalnet/188/Test-Report/

Actual results:

subctl show all

Showing information for cluster "nmanos-cluster-a":
Showing Network details
    Discovered network details:
        Network plugin:  OpenShiftSDN
        Service CIDRs:   [100.96.0.0/16]
        Cluster CIDRs:   [10.252.0.0/14]
        Global CIDR:     169.254.0.0/19


Showing Endpoint details
CLUSTER ID                    ENDPOINT IP     PUBLIC IP       CABLE DRIVER        TYPE            
nmanos-cluster-a              10.166.59.149   54.241.75.163   libreswan           local           
default-cl1                   10.166.3.205    66.187.232.129  libreswan           remote          

Showing Connection details
GATEWAY                         CLUSTER                 REMOTE IP       CABLE DRIVER        SUBNETS                                 STATUS          
default-cl1-2nbr6-worker-8x4qr  default-cl1             10.166.3.205    libreswan           169.254.32.0/19                         connected       

Showing Gateway details
NODE                            HA STATUS       SUMMARY                         
ip-10-166-59-149                active          All connections (1) are established

Showing version details
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.8            
submariner-operator             registry.redhat.io/rhacm2-tech-preview                v0.8            
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.8            

Showing information for cluster "default-cl1":
Showing Network details
    Discovered network details:
        Network plugin:  OpenShiftSDN
        Service CIDRs:   [100.96.0.0/16]
        Cluster CIDRs:   [10.252.0.0/14]
        Global CIDR:     169.254.32.0/19


Showing Endpoint details
No endpoints found

subctl version: v0.8

Expected results:
Cluster should be connected + Endpoints on both clusters should be found.

Additional info:

In submariner-operator pod on Cluster B there's an error:
Operation cannot be fulfilled on submariners.submariner.io \"submariner\": the object has been modified; please apply your changes to the latest version and try again



2021-04-28T07:43:49.638692084Z 2021-04-28T07:43:49.638Z	INFO	controller_submariner	Reconciling Submariner	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner"}
2021-04-28T07:43:49.639233830Z 2021-04-28T07:43:49.639Z	ERROR	controller_submariner	failed to update the Submariner status	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "error": "Operation cannot be fulfilled on submariners.submariner.io \"submariner\": the object has been modified; please apply your changes to the latest version and try again"}
2021-04-28T07:43:49.639233830Z github.com/go-logr/zapr.(*zapLogger).Error
2021-04-28T07:43:49.639233830Z 	/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr.1/zapr.go:128
2021-04-28T07:43:49.639233830Z github.com/submariner-io/submariner-operator/controllers/submariner.(*SubmarinerReconciler).Reconcile
2021-04-28T07:43:49.639233830Z 	/remote-source/app/controllers/submariner/submariner_controller.go:211
2021-04-28T07:43:49.639233830Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2021-04-28T07:43:49.639233830Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.9/pkg/internal/controller/controller.go:246
2021-04-28T07:43:49.639233830Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2021-04-28T07:43:49.639233830Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.9/pkg/internal/controller/controller.go:222
2021-04-28T07:43:49.639233830Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
2021-04-28T07:43:49.639233830Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.9/pkg/internal/controller/controller.go:201
2021-04-28T07:43:49.639233830Z k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
2021-04-28T07:43:49.639233830Z 	/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.0/pkg/util/wait/wait.go:152
2021-04-28T07:43:49.639233830Z k8s.io/apimachinery/pkg/util/wait.JitterUntil
2021-04-28T07:43:49.639233830Z 	/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.0/pkg/util/wait/wait.go:153
2021-04-28T07:43:49.639233830Z k8s.io/apimachinery/pkg/util/wait.Until
2021-04-28T07:43:49.639233830Z 	/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.0/pkg/util/wait/wait.go:88
2021-04-28T07:43:49.639275101Z 2021-04-28T07:43:49.639Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "submariner-controller", "name": "submariner", "namespace": "submariner-operator"}
2021-04-28T07:43:49.639275101Z 2021-04-28T07:43:49.639Z	INFO	controller_submariner	Reconciling Submariner	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner"}
2021-04-28T07:43:49.649158970Z 2021-04-28T07:43:49.648Z	INFO	controller_submariner	Using detected CIDR	{"type": "Cluster", "CIDR": "10.252.0.0/14"}
2021-04-28T07:43:49.649158970Z 2021-04-28T07:43:49.648Z	INFO	controller_submariner	Using detected CIDR	{"type": "Service", "CIDR": "100.96.0.0/16"}
2021-04-28T07:43:49.649158970Z 2021-04-28T07:43:49.648Z	INFO	controller_submariner	Using detected CIDR	{"type": "Cluster", "CIDR": "10.252.0.0/14"}
2021-04-28T07:43:49.649158970Z 2021-04-28T07:43:49.648Z	INFO	controller_submariner	Using detected CIDR	{"type": "Service", "CIDR": "100.96.0.0/16"}
2021-04-28T07:43:49.657187559Z 2021-04-28T07:43:49.657Z	INFO	controller_submariner	Updated existing DaemonSet	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-gateway"}
2021-04-28T07:43:49.657723972Z 2021-04-28T07:43:49.657Z	INFO	controller_submariner	Updated existing DaemonSet	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-gateway"}
2021-04-28T07:43:49.662466643Z 2021-04-28T07:43:49.662Z	INFO	controller_submariner	Updated existing Service	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "Service.Namespace": "submariner-operator", "Service.Name": "submariner-engine-metrics"}
2021-04-28T07:43:49.666145003Z 2021-04-28T07:43:49.666Z	INFO	controller_submariner	Updated existing Service	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "Service.Namespace": "submariner-operator", "Service.Name": "submariner-engine-metrics"}
2021-04-28T07:43:52.524617428Z 2021-04-28T07:43:52.524Z	INFO	controller_submariner	Could not create ServiceMonitor object	{"error": "servicemonitors.monitoring.coreos.com \"submariner-engine-metrics\" already exists"}
2021-04-28T07:43:52.526584173Z 2021-04-28T07:43:52.526Z	INFO	controller_submariner	Could not create ServiceMonitor object	{"error": "servicemonitors.monitoring.coreos.com \"submariner-engine-metrics\" already exists"}
2021-04-28T07:43:52.532511698Z 2021-04-28T07:43:52.532Z	INFO	controller_submariner	Updated existing DaemonSet	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-routeagent"}
2021-04-28T07:43:52.533324899Z 2021-04-28T07:43:52.533Z	INFO	controller_submariner	Updated existing DaemonSet	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-routeagent"}
2021-04-28T07:43:52.538102340Z 2021-04-28T07:43:52.538Z	INFO	controller_submariner	Updated existing DaemonSet	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-globalnet"}
2021-04-28T07:43:52.538779806Z 2021-04-28T07:43:52.538Z	INFO	controller_submariner	Updated existing DaemonSet	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-globalnet"}
2021-04-28T07:43:52.548490948Z 2021-04-28T07:43:52.548Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "submariner", "name": "submariner", "namespace": "submariner-operator"}
2021-04-28T07:43:52.549802394Z 2021-04-28T07:43:52.549Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "submariner-controller", "name": "submariner", "namespace": "submariner-operator"}
2021-04-28T07:43:53.852547325Z 2021-04-28T07:43:53.852Z	INFO	controller_submariner	Reconciling Submariner	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner"}
2021-04-28T07:43:53.862444668Z 2021-04-28T07:43:53.862Z	INFO	controller_submariner	Using detected CIDR	{"type": "Cluster", "CIDR": "10.252.0.0/14"}
2021-04-28T07:43:53.862444668Z 2021-04-28T07:43:53.862Z	INFO	controller_submariner	Using detected CIDR	{"type": "Service", "CIDR": "100.96.0.0/16"}
2021-04-28T07:43:53.871374229Z 2021-04-28T07:43:53.871Z	INFO	controller_submariner	Updated existing DaemonSet	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "DaemonSet.Namespace": "submariner-operator", "DaemonSet.Name": "submariner-gateway"}
2021-04-28T07:43:53.878492146Z 2021-04-28T07:43:53.878Z	INFO	controller_submariner	Updated existing Service	{"Request.Namespace": "submariner-operator", "Request.Name": "submariner", "Service.Namespace": "submariner-operator", "Service.Name": "submariner-engine-metrics"}
Comment 1 Nir Yechiel 2021-05-03 08:45:20 UTC 
Hi Noam,

Can you give it a try with 0.9? If this is something specific to 0.8, I don't see the urgency to fix this - given that this is a tech preview release.


Thanks,
Nir
Comment 2 Steve Mattar 2021-05-19 16:04:47 UTC 
@nmanos I tested on 2 AWS cluster and couldn't reproduce this issue.
Do we still have this problem on v0.8 ?
Can you tell if the same scenario happens on v0.9 ?
Comment 4 Noam Manos 2021-06-13 10:23:15 UTC 
Installing 0.9 on an existing 0.9 deployment did not encounter the described problem:
https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/Submariner-0.9-AWSx2-OVN/96/Test-Report/
Comment 5 Noam Manos 2021-06-14 13:48:08 UTC 
Also tested upgrade from Submariner 0.9 (d/s) to Submariner 0.10 (subctl-devel), and the issue was not reproduced:
https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/Submariner-0.9-OSP-AWS-SDN/89/Test-Report/
Comment 10 errata-xmlrpc 2021-06-16 19:28:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.2.4 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2461