Bug 2013711

Basically subctl diagnose does not support kubecontext so its trying to run the commands on every context from the kubeconfig. This is more like a small enhancement on the subctl diagnose tool.

It also happens when using kubeconfig of a single cluster:

$ export KUBECONFIG=nmanos-aws-devcluster-a/auth/kubeconfig

$ oc config view

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://api.nmanos-aws-devcluster-a.devcluster.openshift.com:6443
  name: api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://api.nmanos-aws-devcluster-a.devcluster.openshift.com:6443
  name: nmanos-aws-devcluster-a
contexts:
- context:
    cluster: nmanos-aws-devcluster-a
    namespace: default
    user: admin
  name: admin
- context:
    cluster: api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
    namespace: test-submariner
    user: master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
  name: default-api-nmanos-aws-devcluster-a-devcluster-openshift-com-6443-master
- context:
    cluster: api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
    namespace: ocm
    user: master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
  name: ocm/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master
- context:
    cluster: api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
    namespace: submariner-operator
    user: master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
  name: submariner-operator/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master
- context:
    cluster: api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
    namespace: test-submariner
    user: master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
  name: test-submariner/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master
current-context: submariner-operator/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443/master
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
- name: master/api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443
  user:
    token: REDACTED


$ subctl diagnose firewall metrics --validation-timeout 120 --verbose
Cluster "api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443"
 • Checking the firewall configuration to determine if the metrics port (8080) is allowed  ...
 ✓ Checking the firewall configuration to determine if the metrics port (8080) is allowed
 ✓ tcpdump output from sniffer pod on Gateway node
 ✓ tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:07:32.768030 ens3  In  IP 10.8.148.225.9898 > 10.8.85.98.8080: Flags [S], seq 2064033732, win 26733, options [mss 8911,sackOK,TS val 640497194 ecr 0,nop,wscale 7], length 0
07:07:32.771761 ens3  In  IP 10.8.148.225.9898 > 10.8.85.98.8080: Flags [S], seq 2064099271, win 26697, options [mss 8911,sackOK,TS val 640497198 ecr 2364456545,nop,wscale 7], length 0
07:07:32.776588 ens3  In  IP 10.8.148.225.9898 > 10.8.85.98.8080: Flags [S], seq 2064164810, win 26697, options [mss 8911,sackOK,TS val 640497203 ecr 2364456548,nop,wscale 7], length 0
07:07:32.781412 ens3  In  IP 10.8.148.225.9898 > 10.8.85.98.8080: Flags [S], seq 2064230349, win 26697, options [mss 8911,sackOK,TS val 640497208 ecr 2364456555,nop,wscale 7], length 0
07:07:32.784761 ens3  In  IP 10.8.148.225.9898 > 10.8.85.98.8080: Flags [S], seq 2064295888, win 26697, options [mss 8911,sackOK,TS val 640497211 ecr 2364456558,nop,wscale 7], length 0
5 packets captured
10 packets received by filter
0 packets dropped by kernel

 ✓ The firewall configuration allows metrics to be retrieved from Gateway nodes

Cluster "nmanos-aws-devcluster-a"
 • Checking the firewall configuration to determine if the metrics port (8080) is allowed  ...
 ✗ Checking the firewall configuration to determine if the metrics port (8080) is allowed
 ✓ tcpdump output from sniffer pod on Gateway node
 ✓ tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

0 packets captured
0 packets received by filter
0 packets dropped by kernel

 ✗ The tcpdump output from the sniffer pod does not contain the client pod HostIP. Please check that your firewall configuration allows TCP/8080 traffic on the "ip-10-8-85-98.us-east-2.compute.internal" node.

*** Bug 2042300 has been marked as a duplicate of this bug. ***

This bug is still relevant in Submariner v0.12.0:

$ subctl gather

Cluster "api-default-cl1-devcluster-openshift-com:6443"
Gathering information from cluster "api-default-cl1-devcluster-openshift-com:6443"
 • Gathering connectivity logs  ...
 ⚠ Gathering connectivity logs
 ✓ Found 1 pods matching label selector "app=submariner-gateway"
 ⚠ Found logs for previous instances of pod submariner-gateway-ztxnv
 ✓ Found 5 pods matching label selector "app=submariner-routeagent"
 ✓ Found 1 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-networkplugin-syncer"
 • Gathering connectivity resources  ...
 ✓ Gathering connectivity resources
 ✓ Gathering CNI data from 5 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"
 ✓ Gathering globalnet data from 1 pods matching label selector "app=submariner-globalnet"
 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 1 gateways in namespace "submariner-operator"
 ✓ Found 1 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 • Gathering service-discovery logs  ...
 ✓ Gathering service-discovery logs
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 5 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 • Gathering service-discovery resources  ...
 ✓ Gathering service-discovery resources
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
 • Gathering broker logs  ...
 ✓ Gathering broker logs
 • Gathering broker resources  ...
 ✓ Gathering broker resources
 ✓ Found 2 endpoints in namespace "submariner-broker"
 ✓ Found 2 clusters in namespace "submariner-broker"
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-broker"
 ✓ Found 0 serviceimports in namespace "submariner-broker"
 • Gathering operator logs  ...
 ✓ Gathering operator logs
 ✓ Found 0 pods matching label selector "name=submariner-operator"
 • Gathering operator resources  ...
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 0 deployments by label selector "app=submariner-networkplugin-syncer" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
Files are stored under directory "submariner-20220315085950"

Encountered following Kubernetes warnings while running:
Warning: discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice

Cluster "nmanos-devcluster-a2-aws"
Error retrieving Submariner resource: error retrieving Submariner object submariner: Unauthorized

The error in that last message suggests that one of the user configurations is incorrect. See https://github.com/submariner-io/submariner-website/issues/707 and https://github.com/submariner-io/submariner-operator/issues/1946#issuecomment-1066932112

The original issue, which is that subctl diagnose specifically does not support merged kubeconfigs, is still present.

Subctl command with merged kubeconfig, is looking good with Submariner 0.14.0 build:

$ oc  config get-contexts

CURRENT   NAME                                                                          CLUSTER                                                AUTHINFO                                                      NAMESPACE
          aws-nmanos-a1                                                                 aws-nmanos-a1                                          admin                                                         default
          default/api-aws-nmanos-a1-devcluster-openshift-com:6443/master                api-aws-nmanos-a1-devcluster-openshift-com:6443        master/api-aws-nmanos-a1-devcluster-openshift-com:6443        test-submariner
          default/api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443/master           api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443   master/api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443   test-submariner
          gcp-nmanos-c1                                                                 gcp-nmanos-c1                                          admin                                                         default
*         test-submariner/api-aws-nmanos-a1-devcluster-openshift-com:6443/master        api-aws-nmanos-a1-devcluster-openshift-com:6443        master/api-aws-nmanos-a1-devcluster-openshift-com:6443        test-submariner
          test-submariner/api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443/master   api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443   master/api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443   test-submariner

# Current OC user: master


$ subctl diagnose deployment

Cluster "aws-nmanos-a1"
 ✗ Error building the cluster.Info for the default configuration: error creating client producer: error creating controller client: Unauthorized
Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Globalnet deployment detected - checking if globalnet CIDRs overlap  ...
 ✓ Globalnet deployment detected - checking if globalnet CIDRs overlap
 ✓ Clusters do not have overlapping globalnet CIDRs
 • Checking Submariner pods  ...
 ✗ Checking Submariner pods
 ✗ Pod "c80f5b5478dc10761d241d89418ece2ed10d31319e704923eb36b18b64f4pjm" is not running. (current state is Succeeded)
 ✗ Pod "submariner-metrics-proxy-w69xq" is not running. (current state is Pending)
Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Globalnet deployment detected - checking if globalnet CIDRs overlap  ...
 ✓ Globalnet deployment detected - checking if globalnet CIDRs overlap
 ✓ Clusters do not have overlapping globalnet CIDRs
 • Checking Submariner pods  ...
 ✗ Checking Submariner pods
Cluster "gcp-nmanos-c1"
 ✗ Pod "c80f5b5478dc10761d241d89418ece2ed10d31319e704923eb36b18b64f4pjm" is not running. (current state is Succeeded)
 ✗ Pod "submariner-metrics-proxy-w69xq" is not running. (current state is Pending)
 • Globalnet deployment detected - checking if globalnet CIDRs overlap  ...
 ✓ Globalnet deployment detected - checking if globalnet CIDRs overlap
 ✓ Clusters do not have overlapping globalnet CIDRs
 • Checking Submariner pods  ...
 ✗ Checking Submariner pods
 ✗ Pod "c80f5b5478dc10761d241d89418ece2ed10d31319e704923eb36b18b649rg65" is not running. (current state is Succeeded)
Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 ✗ Pod "submariner-metrics-proxy-dgdz4" is not running. (current state is Pending)
 • Globalnet deployment detected - checking if globalnet CIDRs overlap  ...
 ✓ Globalnet deployment detected - checking if globalnet CIDRs overlap
 ✓ Clusters do not have overlapping globalnet CIDRs
 • Checking Submariner pods  ...
Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 ✗ Checking Submariner pods
 ✗ Pod "c80f5b5478dc10761d241d89418ece2ed10d31319e704923eb36b18b649rg65" is not running. (current state is Succeeded)
 ✗ Pod "submariner-metrics-proxy-dgdz4" is not running. (current state is Pending)
 • Globalnet deployment detected - checking if globalnet CIDRs overlap  ...
 ✓ Globalnet deployment detected - checking if globalnet CIDRs overlap
 ✓ Clusters do not have overlapping globalnet CIDRs
 • Checking Submariner pods  ...
 ✗ Checking Submariner pods
 ✗ Pod "c80f5b5478dc10761d241d89418ece2ed10d31319e704923eb36b18b649rg65" is not running. (current state is Succeeded)
 ✗ Pod "submariner-metrics-proxy-dgdz4" is not running. (current state is Pending)


$ subctl diagnose connections

Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Checking gateway connections  ...

Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 ✓ Checking gateway connections
 ✓ All connections are established
 • Checking gateway connections  ...
 ✓ Checking gateway connections

Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 ✓ All connections are established
 • Checking gateway connections  ...
 ✓ Checking gateway connections
 ✓ All connections are established

Cluster "gcp-nmanos-c1"
 • Checking gateway connections  ...
 ✓ Checking gateway connections
 ✓ All connections are established

Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 • Checking gateway connections  ...
 ✓ Checking gateway connections
 ✓ All connections are established

Cluster "aws-nmanos-a1"
 ✗ Error building the cluster.Info for the default configuration: error creating client producer: error creating controller client: Unauthorized


$ subctl diagnose k8s-version

Cluster "aws-nmanos-a1"
 ✗ Error building the cluster.Info for the default configuration: error creating client producer: error creating controller client: Unauthorized
Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Checking Submariner support for the Kubernetes version  ...
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.24.6+5157800" is supported

Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Checking Submariner support for the Kubernetes version  ...
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.24.6+5157800" is supported

Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 • Checking Submariner support for the Kubernetes version  ...
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.3+e419edf" is supported

Cluster "gcp-nmanos-c1"
 • Checking Submariner support for the Kubernetes version  ...
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.3+e419edf" is supported

Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 • Checking Submariner support for the Kubernetes version  ...

 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.3+e419edf" is supported


$ subctl diagnose kube-proxy-mode test-submariner

Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 • Checking Submariner support for the kube-proxy mode  ...
 ⚠ Starting with Kubernetes 1.23, the Pod Security admission controller expects namespaces to have security labels. Without these, you will see warnings in subctl's output. subctl should work fine, but you can avoid the warnings and ensure correct behavior by adding at least one of these labels to the namespace "test-submariner":
  pod-security.kubernetes.io/warn=privileged
  pod-security.kubernetes.io/enforce=privileged
  pod-security.kubernetes.io/audit=privileged

 ✗ Checking Submariner support for the kube-proxy mode
Cluster "gcp-nmanos-c1"
 ✗ Error spawning the network pod: timed out waiting for the condition
 • Checking Submariner support for the kube-proxy mode  ...
 ⚠ Starting with Kubernetes 1.23, the Pod Security admission controller expects namespaces to have security labels. Without these, you will see warnings in subctl's output. subctl should work fine, but you can avoid the warnings and ensure correct behavior by adding at least one of these labels to the namespace "test-submariner":
  pod-security.kubernetes.io/enforce=privileged
  pod-security.kubernetes.io/audit=privileged
  pod-security.kubernetes.io/warn=privileged

 ✗ Checking Submariner support for the kube-proxy mode
 ✗ Error spawning the network pod: timed out waiting for the condition
Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 • Checking Submariner support for the kube-proxy mode  ...
 ⚠ Starting with Kubernetes 1.23, the Pod Security admission controller expects namespaces to have security labels. Without these, you will see warnings in subctl's output. subctl should work fine, but you can avoid the warnings and ensure correct behavior by adding at least one of these labels to the namespace "test-submariner":
  pod-security.kubernetes.io/enforce=privileged
  pod-security.kubernetes.io/audit=privileged
  pod-security.kubernetes.io/warn=privileged

 ✗ Checking Submariner support for the kube-proxy mode
Cluster "aws-nmanos-a1"
 ✗ Error spawning the network pod: timed out waiting for the condition
 ✗ Error building the cluster.Info for the default configuration: error creating client producer: error creating controller client: Unauthorized
Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Checking Submariner support for the kube-proxy mode  ...
 ⚠ Starting with Kubernetes 1.23, the Pod Security admission controller expects namespaces to have security labels. Without these, you will see warnings in subctl's output. subctl should work fine, but you can avoid the warnings and ensure correct behavior by adding at least one of these labels to the namespace "test-submariner":
  pod-security.kubernetes.io/enforce=privileged
  pod-security.kubernetes.io/audit=privileged
  pod-security.kubernetes.io/warn=privileged

 ✗ Checking Submariner support for the kube-proxy mode
Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 ✗ Error spawning the network pod: timed out waiting for the condition
 • Checking Submariner support for the kube-proxy mode  ...
 ⚠ Starting with Kubernetes 1.23, the Pod Security admission controller expects namespaces to have security labels. Without these, you will see warnings in subctl's output. subctl should work fine, but you can avoid the warnings and ensure correct behavior by adding at least one of these labels to the namespace "test-submariner":
  pod-security.kubernetes.io/enforce=privileged
  pod-security.kubernetes.io/audit=privileged
  pod-security.kubernetes.io/warn=privileged

 ✗ Checking Submariner support for the kube-proxy mode
 ✗ Error spawning the network pod: timed out waiting for the condition


$ subctl diagnose cni
Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 • Checking Submariner support for the CNI network plugin  ...
 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OpenShiftSDN") is supported
 • Trying to detect the Calico ConfigMap  ...
 ✓ Trying to detect the Calico ConfigMap

Cluster "gcp-nmanos-c1"
 • Checking Submariner support for the CNI network plugin  ...
 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OpenShiftSDN") is supported
 • Trying to detect the Calico ConfigMap  ...
 ✓ Trying to detect the Calico ConfigMap

Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 • Checking Submariner support for the CNI network plugin  ...
 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OpenShiftSDN") is supported
 • Trying to detect the Calico ConfigMap  ...
 ✓ Trying to detect the Calico ConfigMap

Cluster "aws-nmanos-a1"
 ✗ Error building the cluster.Info for the default configuration: error creating client producer: error creating controller client: Unauthorized
Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Checking Submariner support for the CNI network plugin  ...
 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OpenShiftSDN") is supported
 • Trying to detect the Calico ConfigMap  ...
 ✓ Trying to detect the Calico ConfigMap

Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Checking Submariner support for the CNI network plugin  ...
 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OpenShiftSDN") is supported
 • Trying to detect the Calico ConfigMap  ...
 ✓ Trying to detect the Calico ConfigMap


$ subctl diagnose firewall intra-cluster --validation-timeout 120

Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 • Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed  ...
 ⚠ Starting with Kubernetes 1.23, the Pod Security admission controller expects namespaces to have security labels. Without these, you will see warnings in subctl's output. subctl should work fine, but you can avoid the warnings and ensure correct behavior by adding at least one of these labels to the namespace "submariner-operator":
  pod-security.kubernetes.io/enforce=privileged
  pod-security.kubernetes.io/audit=privileged
  pod-security.kubernetes.io/warn=privileged

 ✗ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
Cluster "aws-nmanos-a1"
 ✗ Error spawning the sniffer pod on the Gateway node: error scheduling pod: timed out waiting for the condition
Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 ✗ Error building the cluster.Info for the default configuration: error creating client producer: error creating controller client: Unauthorized
 • Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed  ...
 ✗ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
 ✗ Error spawning the sniffer pod on the node "ip-10-16-81-237.us-west-1.compute.internal": %!!(MISSING)v(MISSING): error scheduling pod: timed out waiting for the condition
 ✗ Unable to obtain a gateway node: Error spawning the sniffer pod on the node "ip-10-16-81-237.us-west-1.compute.internal": %!v(MISSING): error scheduling pod: timed out waiting for the condition
Cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 • Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed  ...
 ✗ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
Cluster "api-gcp-nmanos-c1-gcp-subm-red-chesterfield-com:6443"
 ✗ Error spawning the sniffer pod on the node "ip-10-16-81-237.us-west-1.compute.internal": %!!(MISSING)v(MISSING): error scheduling pod: timed out waiting for the condition
 ✗ Unable to obtain a gateway node: Error spawning the sniffer pod on the node "ip-10-16-81-237.us-west-1.compute.internal": %!v(MISSING): error scheduling pod: timed out waiting for the condition
 • Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed  ...
 ⚠ Starting with Kubernetes 1.23, the Pod Security admission controller expects namespaces to have security labels. Without these, you will see warnings in subctl's output. subctl should work fine, but you can avoid the warnings and ensure correct behavior by adding at least one of these labels to the namespace "submariner-operator":
  pod-security.kubernetes.io/warn=privileged
  pod-security.kubernetes.io/enforce=privileged
  pod-security.kubernetes.io/audit=privileged

 ✗ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
 ✗ Error spawning the sniffer pod on the Gateway node: error scheduling pod: timed out waiting for the condition
Cluster "gcp-nmanos-c1"
 • Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed  ...
 ⚠ Starting with Kubernetes 1.23, the Pod Security admission controller expects namespaces to have security labels. Without these, you will see warnings in subctl's output. subctl should work fine, but you can avoid the warnings and ensure correct behavior by adding at least one of these labels to the namespace "submariner-operator":
  pod-security.kubernetes.io/enforce=privileged
  pod-security.kubernetes.io/audit=privileged
  pod-security.kubernetes.io/warn=privileged

 ✗ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
 ✗ Error spawning the sniffer pod on the Gateway node: error scheduling pod: timed out waiting for the condition


$ subctl diagnose firewall inter-cluster "/mnt/skynet-data/skynet-env-1/aws-nmanos-a1/auth/kubeconfig" "/mnt/skynet-data/skynet-env-1/gcp-nmanos-c1/auth/kubeconfig" --validation-timeout 120 --verbose

 ⚠ The two-argument form of inter-cluster is deprecated, see the documentation for details
 • Checking if tunnels can be setup on the gateway node of cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"  ...
 ✗ Checking if tunnels can be setup on the gateway node of cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"
 ✗ Error spawning the sniffer pod on the node "ip-10-16-81-237.us-west-1.compute.internal": %!!(MISSING)v(MISSING): error scheduling pod: timed out waiting for the condition
 ✗ Could not determine if Tunnels can be established on the gateway node of cluster "api-aws-nmanos-a1-devcluster-openshift-com:6443"