2013157 – subctl diagnose firewall intra-cluster - failed VXLAN checks

Bug 2013157 - subctl diagnose firewall intra-cluster - failed VXLAN checks

Summary: subctl diagnose firewall intra-cluster - failed VXLAN checks

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Advanced Cluster Management for Kubernetes
Classification:	Red Hat
Component:	Submariner
Sub Component:
Version:	rhacm-2.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	low
Target Milestone:	---
Target Release:	rhacm-2.4
Assignee:	Sridhar Gaddam
QA Contact:	Noam Manos
Docs Contact:	Christopher Dawson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-10-12 09:12 UTC by Noam Manos
Modified:	2021-11-11 18:34 UTC (History)
CC List:	1 user (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-11-11 18:33:54 UTC
Target Upstream Version:
Embargoed:
Flags:	ming: rhacm-2.4+

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	open-cluster-management backlog issues 17124	None	None	None	2021-10-12 17:06:33 UTC
Github	submariner-io submariner-operator issues 1614	None	open	subctl diagnose firewall intra-cluster - failed VXLAN checks	2021-10-12 14:59:00 UTC
Red Hat Product Errata	RHSA-2021:4618	None	None	None	2021-11-11 18:34:36 UTC

Description Noam Manos 2021-10-12 09:12:18 UTC

**What happened**:

Running "subctl diagnose firewall intra-cluster kubeconfig1 kubeconfig2" returns that all VXLAN checks have failed, while * all E2E did pass * on this environment, so there's seems to be wrong diagnose:

$ subctl diagnose firewall intra-cluster  /mnt/skynet-data/skynet-env-1/nmanos-aws-devcluster-a/auth/kubeconfig  /mnt/skynet-data/skynet-env-1/nmanos-aws-devcluster-c/auth/kubeconfig

Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ Could not find the local Endpoint in cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"

Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ Could not find the local Endpoint in cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"

Cluster "nmanos-aws-devcluster-c"
 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ Could not find the local Endpoint in cluster "nmanos-aws-devcluster-c"

Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ Could not find the local Endpoint in cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"

Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ Could not find the local Endpoint in cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"

Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ Could not find the local Endpoint in cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"

Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ Could not find the local Endpoint in cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"

Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ Could not find the local Endpoint in cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"


**What you expected to happen**:
All subctl diagnose firewall intra-cluster checks should pass.

**How to reproduce it (as minimally and precisely as possible)**:
Install Submariner 0.11 on ACM 2.4 with two managed clusters, and run subctl.

https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/ACM-2.4-Submariner-0.11-AWSx2-SDN/157/Test-Report/


**Anything else we need to know?**:

**Environment**:

# Current Kubeconfig contexts:
CURRENT   NAME                                                                                   CLUSTER                                                     AUTHINFO                                                           NAMESPACE
          admin                                                                                  nmanos-aws-devcluster-a                                     admin                                                              default
          default-api-nmanos-aws-devcluster-a-devcluster-openshift-com-6443-master               api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   test-submariner
          ocm/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master                   api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   ocm
*         submariner-operator/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master   api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   submariner-operator
          test-submariner/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master       api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   test-submariner

### OCP Cluster api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443 ###
Client Version: 4.8.13
Server Version: 4.8.13
Kubernetes Version: v1.21.1+a620f50

NAMESPACE                             NAME                       HOST/PORT                                                                                                            PATH   SERVICES                   PORT    TERMINATION            WILDCARD
ocm                                   multicloud-console         multicloud-console.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                                    management-ingress         https   reencrypt/Redirect     None
open-cluster-management-agent-addon   klusterlet-addon-workmgr   klusterlet-addon-workmgr-open-cluster-management-agent-addon.apps.nmanos-aws-devcluster-a.devcluster.openshift.com          klusterlet-addon-workmgr   <all>   passthrough            None
openshift-authentication              oauth-openshift            oauth-openshift.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                                       oauth-openshift            6443    passthrough/Redirect   None
openshift-console                     console                    console-openshift-console.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                             console                    https   reencrypt/Redirect     None
openshift-console                     downloads                  downloads-openshift-console.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                           downloads                  http    edge/Redirect          None
openshift-ingress-canary              canary                     canary-openshift-ingress-canary.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                       ingress-canary             8080    edge/Redirect          None
openshift-monitoring                  alertmanager-main          alertmanager-main-openshift-monitoring.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                alertmanager-main          web     reencrypt/Redirect     None
openshift-monitoring                  grafana                    grafana-openshift-monitoring.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                          grafana                    https   reencrypt/Redirect     None
openshift-monitoring                  prometheus-k8s             prometheus-k8s-openshift-monitoring.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                   prometheus-k8s             web     reencrypt/Redirect     None
openshift-monitoring                  thanos-querier             thanos-querier-openshift-monitoring.apps.nmanos-aws-devcluster-a.devcluster.openshift.com                                   thanos-querier             web     reencrypt/Redirect     None


# Current Kubeconfig contexts:
CURRENT   NAME                                                                                   CLUSTER                                                     AUTHINFO                                                           NAMESPACE
          admin                                                                                  nmanos-aws-devcluster-c                                     admin                                                              default
          default-api-nmanos-aws-devcluster-c-devcluster-openshift-com-6443-master               api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   test-submariner
*         submariner-operator/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443/master   api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   test-submariner
          test-submariner/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443/master       api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   test-submariner

### OCP Cluster api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443 ###
Client Version: 4.8.13
Server Version: 4.8.13
Kubernetes Version: v1.21.1+a620f50

NAMESPACE                             NAME                       HOST/PORT                                                                                                            PATH   SERVICES                   PORT    TERMINATION            WILDCARD
open-cluster-management-agent-addon   klusterlet-addon-workmgr   klusterlet-addon-workmgr-open-cluster-management-agent-addon.apps.nmanos-aws-devcluster-c.devcluster.openshift.com          klusterlet-addon-workmgr   <all>   passthrough            None
openshift-authentication              oauth-openshift            oauth-openshift.apps.nmanos-aws-devcluster-c.devcluster.openshift.com                                                       oauth-openshift            6443    passthrough/Redirect   None
openshift-console                     console                    console-openshift-console.apps.nmanos-aws-devcluster-c.devcluster.openshift.com                                             console                    https   reencrypt/Redirect     None
openshift-console                     downloads                  downloads-openshift-console.apps.nmanos-aws-devcluster-c.devcluster.openshift.com                                           downloads                  http    edge/Redirect          None
openshift-ingress-canary              canary                     canary-openshift-ingress-canary.apps.nmanos-aws-devcluster-c.devcluster.openshift.com                                       ingress-canary             8080    edge/Redirect          None
openshift-monitoring                  alertmanager-main          alertmanager-main-openshift-monitoring.apps.nmanos-aws-devcluster-c.devcluster.openshift.com                                alertmanager-main          web     reencrypt/Redirect     None
openshift-monitoring                  grafana                    grafana-openshift-monitoring.apps.nmanos-aws-devcluster-c.devcluster.openshift.com                                          grafana                    https   reencrypt/Redirect     None
openshift-monitoring                  prometheus-k8s             prometheus-k8s-openshift-monitoring.apps.nmanos-aws-devcluster-c.devcluster.openshift.com                                   prometheus-k8s             web     reencrypt/Redirect     None
openshift-monitoring                  thanos-querier             thanos-querier-openshift-monitoring.apps.nmanos-aws-devcluster-c.devcluster.openshift.com                                   thanos-querier             web     reencrypt/Redirect     None

### Submariner components ###

subctl version: v0.11.0
Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 • Showing versions  ...
 ✓ Showing versions
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.11.0         
submariner-operator             registry.redhat.io/rhacm2-tech-preview                5f615e0763abca9 
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.11.0         

Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 • Showing versions  ...
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.11.0         
submariner-operator             registry.redhat.io/rhacm2-tech-preview                5f615e0763abca9 
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.11.0         

Cluster "nmanos-aws-devcluster-c"
 ✓ Showing versions
 • Showing versions  ...
 ✓ Showing versions
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.11.0         
submariner-operator             registry.redhat.io/rhacm2-tech-preview                5f615e0763abca9 
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.11.0         

Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 • Showing versions  ...
COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.11.0         
submariner-operator             registry.redhat.io/rhacm2-tech-preview                5f615e0763abca9 
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.11.0         

 ✓ Showing versions

### Images of Pods (in namespace submariner-operator) ###

### submariner-operator-bundle-index Image ###
id=image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-operator-bundle-index@sha256:2e14e7edd34469815c131b80f99690a640961538882054f3805eed5ba4e5eb5d
name=openshift/ose-operator-registry
release=202110080132.p0.git.5649248.assembly.stream
url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.8.0-202110080132.p0.git.5649248.assembly.stream
version=v4.8.0

### ocp-v4.0-art-dev Image ###
id=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7989faef8c94da91813f4aafc7b5d529745500023395eb086a5c258994695002

### ocp-v4.0-art-dev Image ###
id=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88be8e3159448b0dfecdc19dc5c849d0f56424f238fccb214484c3daff97891a

### rhacm2-tech-preview-submariner-operator-bundle Image ###
id=registry-proxy.engineering.redhat.com/rh-osbs/rhacm2-tech-preview-submariner-operator-bundle@sha256:1c8b86ff8d627315d20e562b17d326310680a4bcc6c892854b44d5203d12d475
name=rhacm2-tech-preview/submariner-operator-bundle
release=30
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-operator-bundle/images/v0.11.0-30
version=v0.11.0

### lighthouse-agent-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/lighthouse-agent-rhel8@sha256:3ecd6af1e108c8858c49630e17f01e500b91f4747ed01d767377bcfd52245052
name=rhacm2-tech-preview/lighthouse-agent-rhel8
release=14
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-agent-rhel8/images/v0.11.0-14
version=v0.11.0

### lighthouse-coredns-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/lighthouse-coredns-rhel8@sha256:a7078bbba19caa8e7d9eb2bd67f568aaf5a507a28151b8de622ad5965f8454e3
name=rhacm2-tech-preview/lighthouse-coredns-rhel8
release=14
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-coredns-rhel8/images/v0.11.0-14
version=v0.11.0

### submariner-gateway-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-gateway-rhel8@sha256:4efa7cbaaf294552498fcc8c21e83333fff0b4c85f52f95556b4063520ac0d83
name=rhacm2-tech-preview/submariner-gateway-rhel8
release=15
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-gateway-rhel8/images/v0.11.0-15
version=v0.11.0

### submariner-rhel8-operator Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-rhel8-operator@sha256:465f8c6c8c22e5531bd22e1642e2b0e4c264193f8b74e18cf357cdb739ff99e2
name=rhacm2-tech-preview/submariner-rhel8-operator
release=36
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-rhel8-operator/images/v0.11.0-36
version=v0.11.0

### submariner-route-agent-rhel8 Image ###
id=registry.redhat.io/rhacm2-tech-preview/submariner-route-agent-rhel8@sha256:304e8b5bd7e1cb89b44c2b7f8c143e76cebbd13002446ea9ad36f657c5c1d0ff
name=rhacm2-tech-preview/submariner-route-agent-rhel8
release=15
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-route-agent-rhel8/images/v0.11.0-15
version=v0.11.0

### submariner-addon-rhel8 Image ###
id=registry.redhat.io/rhacm2/submariner-addon-rhel8@sha256:21633c4d0eeaa29f90c5052a7b868729c9cc00f6aa8bd4da4baee6d40db7d3d7

Comment 1 Mike Ng 2021-10-14 13:23:49 UTC

G2Bsync 941083357 comment 
 nyechiel Tue, 12 Oct 2021 14:47:56 UTC 
 G2Bsync This issue has nothing to do with the addon nor ACM. It's being tracked here: https://github.com/submariner-io/submariner-operator/issues/1614

@qiujian16 can you please close this issue (or let me know how can I get the required permissions to do so myself)? Thanks!

Comment 2 Sridhar Gaddam 2021-10-21 12:15:07 UTC

@nmanos fix is merged both in upstream and downstream. Please verify and close this BZ.

Comment 3 Noam Manos 2021-10-26 07:13:51 UTC

Thanks Sridhar, it works good on d/s now:
https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/ACM-2.4-Submariner-0.11-AWSx2-SDN/246/Test-Report/

subctl version: v0.11.0

COMPONENT                       REPOSITORY                                            VERSION         
submariner                      registry.redhat.io/rhacm2-tech-preview                v0.11.0         
submariner-operator             registry.redhat.io/rhacm2-tech-preview                08d185eea8ee48a 
service-discovery               registry.redhat.io/rhacm2-tech-preview                v0.11.0   


$ oc  config get-contexts
CURRENT   NAME                                                                                   CLUSTER                                                     AUTHINFO                                                           NAMESPACE
          admin                                                                                  nmanos-aws-devcluster-c                                     admin                                                              default
          default-api-nmanos-aws-devcluster-a-devcluster-openshift-com-6443-master               api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   test-submariner
          default-api-nmanos-aws-devcluster-c-devcluster-openshift-com-6443-master               api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   test-submariner
          ocm/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master                   api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   ocm
*         submariner-operator/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master   api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   submariner-operator
          submariner-operator/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443/master   api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   test-submariner
          test-submariner/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master       api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443   test-submariner
          test-submariner/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443/master       api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   master/api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443   test-submariner


$ subctl diagnose firewall intra-cluster --validation-timeout 120

Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 • Checking the firewall configuration to determine if VXLAN traffic is allowed  ...
 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ The firewall configuration allows VXLAN traffic

Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 • Checking the firewall configuration to determine if VXLAN traffic is allowed  ...
 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ The firewall configuration allows VXLAN traffic

Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 • Checking the firewall configuration to determine if VXLAN traffic is allowed  ...
 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ The firewall configuration allows VXLAN traffic

Cluster "nmanos-aws-devcluster-c"
 • Checking the firewall configuration to determine if VXLAN traffic is allowed  ...
 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ The firewall configuration allows VXLAN traffic

Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 • Checking the firewall configuration to determine if VXLAN traffic is allowed  ...
 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ The firewall configuration allows VXLAN traffic

Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 • Checking the firewall configuration to determine if VXLAN traffic is allowed  ...

Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ The firewall configuration allows VXLAN traffic
 • Checking the firewall configuration to determine if VXLAN traffic is allowed  ...

Cluster "api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443"
 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ The firewall configuration allows VXLAN traffic
 • Checking the firewall configuration to determine if VXLAN traffic is allowed  ...

 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ The firewall configuration allows VXLAN traffic

Comment 6 errata-xmlrpc 2021-11-11 18:33:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Advanced Cluster Management 2.4 images and security updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4618

Note You need to log in before you can comment on or make changes to this bug.