**What happened**: In ACM 2.5.0, when submariner broker is missing, then Submariner config does not get all conditions ready. In ACM console UI under "Submariner Add-ons", the "Connection status" and "Agent status" icon keeps rolling with "Progressing" mesage, instead of displaying proper error message (e.g. "Broker is missing") Looking at the spec of the created Submariner config object - it does not show any error. However not all expected conditions (for example "SubmarinerConfigApplied") were listed: Spec: IP Sec IKE Port: 502 IP Sec NATT Port: 4502 NATT Discovery Port: 4900 NATT Enable: true Cable Driver: libreswan Credentials Secret: Name: acm-nmanos-devcluster-a2-aws-aws-creds Gateway Config: Aws: Instance Type: c5d.large Gateways: 1 Image Pull Specs: Load Balancer Enable: false Subscription Config: Channel: alpha-0.12 Source: my-submariner-catalog Source Namespace: submariner-operator Starting CSV: submariner.v0.12.0 Status: Conditions: Last Transition Time: 2022-03-10T08:15:06Z Message: Submariner cluster environment was prepared Reason: SubmarinerClusterEnvPrepared Status: True Type: SubmarinerClusterEnvironmentPrepared Last Transition Time: 2022-03-10T08:17:53Z Message: 1 node(s) ("ip-10-0-72-198.us-west-1.compute.internal") are labeled as gateways Reason: Success Status: True Type: SubmarinerGatewaysLabeled Managed Cluster Info: Cluster Name: acm-nmanos-devcluster-a2-aws Infra Id: nmanos-devcluster-a2-hqwtp Platform: AWS Region: us-west-1 Vendor: OpenShift Events: <none> **What you expected to happen**: If Submariner Broker is missing, ACM UI should indicate it, and stop showing false "Progressing" status. **How to reproduce it (as minimally and precisely as possible)**: 1) Install ACM 2.5.0 with Submariner 0.12.0 operator. 2) Do NOT create Submariner Broker. 3) Create this submariner config: apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: acm-nmanos-devcluster-a2-aws spec: IPSecIKEPort: 502 IPSecNATTPort: 4502 cableDriver: libreswan credentialsSecret: name: acm-nmanos-devcluster-a2-aws-aws-creds gatewayConfig: aws: instanceType: c5d.large gateways: 1 imagePullSpecs: lighthouseAgentImagePullSpec: '' lighthouseCoreDNSImagePullSpec: '' submarinerImagePullSpec: '' submarinerRouteAgentImagePullSpec: '' subscriptionConfig: channel: alpha-0.12 source: my-submariner-catalog sourceNamespace: submariner-operator startingCSV: submariner.v0.12.0 **Anything else we need to know?**: See full scenario here: https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/ACM-2.5.0-Submariner-0.12.0-AWS-OSP/55/Test-Report/ **Environment**: # Cloud platform: Amazon # OCP version: 4.9.0 # ACM version: 2.5.0 ### Submariner components ### subctl version: v0.12.0 Cluster "api-nmanos-devcluster-a2-aws-devcluster-openshift-com:6443" • Submariner is not installed ### submariner-operator-bundle-index Image ### id=image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-operator-bundle-index@sha256:e42781acae3cd6eca70f5928a87a0cd106afe9cbf0b29b89dfb47f7a91218536 name=openshift/ose-operator-registry release=202202120107.p0.g3cd421d.assembly.stream url =https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.9.0-202202120107.p0.g3cd421d.assembly.stream version=v4.9.0 ### submariner-addon-rhel8 Image ### id=registry.redhat.io/rhacm2/submariner-addon-rhel8@sha256:7fabdd2b39a029c9ddffe46299c4246bc5b434998d49a4ba13c64e3b05dfc052 ################################################################### # ImageStream Tags (in namespace submariner-operator) # ################################################################### ### lighthouse-agent-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/lighthouse-agent-rhel8 release=9 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-agent-rhel8/images/v0.12.0-9 version=v0.12.0 ### lighthouse-coredns-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/lighthouse-coredns-rhel8 release=10 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-coredns-rhel8/images/v0.12.0-10 version=v0.12.0 ### submariner-gateway-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-gateway-rhel8 release=14 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-gateway-rhel8/images/v0.12.0-14 version=v0.12.0 ### submariner-globalnet-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-globalnet-rhel8 release=14 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-globalnet-rhel8/images/v0.12.0-14 version=v0.12.0 ### submariner-networkplugin-syncer-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-networkplugin-syncer-rhel8 release=14 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-networkplugin-syncer-rhel8/images/v0.12.0-14 version=v0.12.0 ### submariner-operator-bundle:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-operator-bundle release=12 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-operator-bundle/images/v0.12.0-12 version=v0.12.0 ### submariner-operator-bundle-index:v0.12.0 Image-Stream tag ### name=openshift/ose-operator-registry release=202202120107.p0.g3cd421d.assembly.stream url =https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.9.0-202202120107.p0.g3cd421d.assembly.stream version=v4.9.0 ### submariner-rhel8-operator:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-rhel8-operator release=24 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-rhel8-operator/images/v0.12.0-24 version=v0.12.0 ### submariner-route-agent-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-route-agent-rhel8 release=13 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-route-agent-rhel8/images/v0.12.0-13 version=v0.12.0
Can you share the following form the cluster that is not working: 1. oc get managedclusteraddons -n <clustername> submariner -o yaml 2. Log output of the submariner-addon-agent 3. Status of all pods in submariner-operator namespace. The relevant status messages show up as deployment comes up. Missing status messages likely means something went wrong during deployment.
Small correction, pod running submariner-addon-agent is just called submariner-addon. Labels: app=submariner-addon
Here are the required logs (it's a new test run, but with the same bug): $ oc describe managedclusteraddon -n acm-nmanos-devcluster-a2-aws submariner Name: submariner Namespace: acm-nmanos-devcluster-a2-aws Labels: <none> Annotations: <none> API Version: addon.open-cluster-management.io/v1alpha1 Kind: ManagedClusterAddOn Metadata: Creation Timestamp: 2022-03-11T08:18:43Z Finalizers: submarineraddon.open-cluster-management.io/submariner-addon-cleanup Generation: 1 Managed Fields: API Version: addon.open-cluster-management.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:installNamespace: Manager: kubectl-client-side-apply Operation: Update Time: 2022-03-11T08:18:43Z API Version: addon.open-cluster-management.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"submarineraddon.open-cluster-management.io/submariner-addon-cleanup": Manager: submariner Operation: Update Time: 2022-03-11T08:18:46Z API Version: addon.open-cluster-management.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:addOnConfiguration: .: f:crName: f:crdName: f:addOnMeta: .: f:description: f:displayName: f:conditions: f:healthCheck: .: f:mode: f:registrations: Manager: submariner Operation: Update Subresource: status Time: 2022-03-11T08:21:23Z Resource Version: 80275 UID: 98917d45-35f1-4684-829d-679ff240ee48 Spec: Install Namespace: submariner-operator Status: Add On Configuration: Cr Name: Crd Name: Add On Meta: Description: Display Name: Conditions: Last Transition Time: 2022-03-11T08:18:43Z Message: Registration of the addon agent is configured Reason: RegistrationConfigured Status: True Type: RegistrationApplied Last Transition Time: 2022-03-11T08:18:44Z Message: manifest of addon applied successfully Reason: AddonManifestApplied Status: True Type: ManifestApplied Last Transition Time: 2022-03-11T08:18:49Z Message: Reason: BrokerConfigMissing Status: False Type: SubmarinerBrokerConfigApplied Last Transition Time: 2022-03-11T08:21:23Z Message: The nodes "ip-10-0-5-147.us-west-1.compute.internal" are labeled with "submariner.io/gateway" Reason: SubmarinerGatewayNodesLabeled Status: True Type: SubmarinerGatewayNodesLabeled Last Transition Time: 2022-03-11T08:19:11Z Message: submariner add-on is available. Reason: ManagedClusterAddOnLeaseUpdated Status: True Type: Available Health Check: Mode: Lease Registrations: Signer Name: kubernetes.io/kube-apiserver-client Subject: Groups: system:open-cluster-management:cluster:acm-nmanos-devcluster-a2-aws:addon:submariner system:open-cluster-management:addon:submariner system:authenticated User: system:open-cluster-management:cluster:acm-nmanos-devcluster-a2-aws:addon:submariner:agent:submariner-addon-agent Events: <none> ####################################################################### # ACM and Submariner resources in nmanos-devcluster-a2-aws # ####################################################################### $ /usr/bin/oc get all -n ocm NAME READY STATUS RESTARTS AGE pod/application-chart-160df-applicationui-5cccc47668-mjqdq 1/1 Running 0 16m pod/application-chart-160df-applicationui-5cccc47668-wcxls 1/1 Running 0 16m pod/application-chart-160df-consoleapi-7b8945459c-kmj25 1/1 Running 0 16m pod/application-chart-160df-consoleapi-7b8945459c-n5wj6 1/1 Running 0 16m pod/c5d9f22731b6141f655d08d37b3829f09b7ad1cd360f753adc93de--1-zj55g 0/1 Completed 0 18m pod/console-chart-4c510-console-v2-77569d4b94-2qfs4 1/1 Running 0 15m pod/console-chart-4c510-console-v2-77569d4b94-bs264 1/1 Running 0 15m pod/grc-06e40-grcui-7c7f475649-b6lfg 1/1 Running 0 15m pod/grc-06e40-grcui-7c7f475649-gfwb4 1/1 Running 0 15m pod/grc-06e40-grcuiapi-69bc87cb4c-kr9jx 1/1 Running 0 15m pod/grc-06e40-grcuiapi-69bc87cb4c-wx8cj 1/1 Running 0 15m pod/grc-06e40-policy-addon-controller-5c567f4856-6n76g 1/1 Running 0 15m pod/grc-06e40-policy-addon-controller-5c567f4856-dmv8j 1/1 Running 0 15m pod/grc-06e40-policy-propagator-597f459b9d-6j8rt 2/2 Running 0 15m pod/grc-06e40-policy-propagator-597f459b9d-mt5v8 2/2 Running 0 15m pod/klusterlet-addon-controller-v2-6655d49f5c-tcnz4 1/1 Running 0 15m pod/klusterlet-addon-controller-v2-6655d49f5c-v89gc 1/1 Running 0 15m pod/management-ingress-46dfe-6fd69489b4-fc5t4 2/2 Running 0 15m pod/management-ingress-46dfe-6fd69489b4-zbsj9 2/2 Running 0 15m pod/multicluster-observability-operator-67f47c5556-xnqjz 1/1 Running 0 17m pod/multicluster-operators-application-6dfc894b88-qdwdg 3/3 Running 1 (17m ago) 17m pod/multicluster-operators-channel-66f45746b7-566ll 1/1 Running 1 (16m ago) 17m pod/multicluster-operators-hub-subscription-6c8d8b845b-grf5j 1/1 Running 1 (17m ago) 17m pod/multicluster-operators-standalone-subscription-858cc89b66-k4zhr 1/1 Running 0 17m pod/multicluster-operators-subscription-report-5f999975f5-nhvpc 1/1 Running 0 17m pod/multiclusterhub-operator-56cc7588b8-kt9rv 1/1 Running 0 17m pod/multiclusterhub-repo-d54db874c-fhp7c 1/1 Running 0 16m pod/my-acm-catalog-qk6kb 1/1 Running 0 19m pod/policyreport-3dc9b-insights-client-74f799b75b-t8hr7 1/1 Running 0 15m pod/policyreport-3dc9b-metrics-68b6c8cc88-226gw 2/2 Running 0 15m pod/search-operator-f4bcc7c8c-h72cf 1/1 Running 0 15m pod/search-prod-dc192-search-aggregator-85b78649d5-7b5f2 1/1 Running 0 15m pod/search-prod-dc192-search-api-567555db89-6sn6t 1/1 Running 0 15m pod/search-prod-dc192-search-api-567555db89-wspcq 1/1 Running 0 15m pod/search-prod-dc192-search-collector-74494857fd-8bhj9 1/1 Running 0 15m pod/search-redisgraph-0 1/1 Running 0 15m pod/submariner-addon-665b595ff5-s5xkr 1/1 Running 0 17m pod/volsync-addon-controller-2f678-deploy-64585c89c-gfkz8 1/1 Running 0 15m pod/volsync-addon-controller-2f678-deploy-64585c89c-p98lx 1/1 Running 0 15m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/application-chart-160df-applicationui ClusterIP 172.30.227.206 <none> 3001/TCP 16m service/channels-apps-open-cluster-management-webhook-svc ClusterIP 172.30.204.10 <none> 443/TCP 16m service/console-api ClusterIP 172.30.6.235 <none> 4000/TCP 16m service/console-chart-4c510-console-v2 ClusterIP 172.30.209.158 <none> 3000/TCP 15m service/grc-06e40-grcui ClusterIP 172.30.237.233 <none> 3000/TCP 15m service/grc-06e40-grcuiapi ClusterIP 172.30.52.1 <none> 4000/TCP 15m service/grc-06e40-policy-propagator-metrics ClusterIP 172.30.244.152 <none> 8443/TCP 15m service/insights-client ClusterIP 172.30.151.236 <none> 3030/TCP 15m service/insights-metrics ClusterIP 172.30.94.22 <none> 8443/TCP 15m service/management-ingress ClusterIP 172.30.132.92 <none> 443/TCP,8443/TCP 15m service/multicluster-observability-webhook-service ClusterIP 172.30.255.234 <none> 443/TCP 17m service/multicluster-operators-application-svc ClusterIP 172.30.213.49 <none> 443/TCP 16m service/multicluster-operators-subscription ClusterIP 172.30.138.79 <none> 8443/TCP 17m service/multiclusterhub-operator-webhook ClusterIP 172.30.156.8 <none> 443/TCP 17m service/multiclusterhub-repo ClusterIP 172.30.222.240 <none> 3000/TCP 16m service/my-acm-catalog ClusterIP 172.30.178.76 <none> 50051/TCP 19m service/search-aggregator ClusterIP 172.30.108.216 <none> 3010/TCP 15m service/search-prod-dc192-search-redisgraph ClusterIP 172.30.87.213 <none> 6380/TCP 15m service/search-search-api ClusterIP 172.30.183.104 <none> 4010/TCP 15m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/application-chart-160df-applicationui 2/2 2 2 16m deployment.apps/application-chart-160df-consoleapi 2/2 2 2 16m deployment.apps/console-chart-4c510-console-v2 2/2 2 2 15m deployment.apps/grc-06e40-grcui 2/2 2 2 15m deployment.apps/grc-06e40-grcuiapi 2/2 2 2 15m deployment.apps/grc-06e40-policy-addon-controller 2/2 2 2 15m deployment.apps/grc-06e40-policy-propagator 2/2 2 2 15m deployment.apps/klusterlet-addon-controller-v2 2/2 2 2 15m deployment.apps/management-ingress-46dfe 2/2 2 2 15m deployment.apps/multicluster-observability-operator 1/1 1 1 17m deployment.apps/multicluster-operators-application 1/1 1 1 17m deployment.apps/multicluster-operators-channel 1/1 1 1 17m deployment.apps/multicluster-operators-hub-subscription 1/1 1 1 17m deployment.apps/multicluster-operators-standalone-subscription 1/1 1 1 17m deployment.apps/multicluster-operators-subscription-report 1/1 1 1 17m deployment.apps/multiclusterhub-operator 1/1 1 1 17m deployment.apps/multiclusterhub-repo 1/1 1 1 16m deployment.apps/policyreport-3dc9b-insights-client 1/1 1 1 15m deployment.apps/policyreport-3dc9b-metrics 1/1 1 1 15m deployment.apps/search-operator 1/1 1 1 15m deployment.apps/search-prod-dc192-search-aggregator 1/1 1 1 15m deployment.apps/search-prod-dc192-search-api 2/2 2 2 15m deployment.apps/search-prod-dc192-search-collector 1/1 1 1 15m deployment.apps/submariner-addon 1/1 1 1 17m deployment.apps/volsync-addon-controller-2f678-deploy 2/2 2 2 15m NAME DESIRED CURRENT READY AGE replicaset.apps/application-chart-160df-applicationui-5cccc47668 2 2 2 16m replicaset.apps/application-chart-160df-consoleapi-7b8945459c 2 2 2 16m replicaset.apps/console-chart-4c510-console-v2-77569d4b94 2 2 2 15m replicaset.apps/grc-06e40-grcui-7c7f475649 2 2 2 15m replicaset.apps/grc-06e40-grcuiapi-69bc87cb4c 2 2 2 15m replicaset.apps/grc-06e40-policy-addon-controller-5c567f4856 2 2 2 15m replicaset.apps/grc-06e40-policy-propagator-597f459b9d 2 2 2 15m replicaset.apps/klusterlet-addon-controller-v2-6655d49f5c 2 2 2 15m replicaset.apps/management-ingress-46dfe-6fd69489b4 2 2 2 15m replicaset.apps/multicluster-observability-operator-67f47c5556 1 1 1 17m replicaset.apps/multicluster-operators-application-6dfc894b88 1 1 1 17m replicaset.apps/multicluster-operators-channel-66f45746b7 1 1 1 17m replicaset.apps/multicluster-operators-hub-subscription-6c8d8b845b 1 1 1 17m replicaset.apps/multicluster-operators-standalone-subscription-858cc89b66 1 1 1 17m replicaset.apps/multicluster-operators-subscription-report-5f999975f5 1 1 1 17m replicaset.apps/multiclusterhub-operator-56cc7588b8 1 1 1 17m replicaset.apps/multiclusterhub-repo-d54db874c 1 1 1 16m replicaset.apps/policyreport-3dc9b-insights-client-74f799b75b 1 1 1 15m replicaset.apps/policyreport-3dc9b-metrics-68b6c8cc88 1 1 1 15m replicaset.apps/search-operator-f4bcc7c8c 1 1 1 15m replicaset.apps/search-prod-dc192-search-aggregator-85b78649d5 1 1 1 15m replicaset.apps/search-prod-dc192-search-api-567555db89 2 2 2 15m replicaset.apps/search-prod-dc192-search-collector-74494857fd 1 1 1 15m replicaset.apps/submariner-addon-665b595ff5 1 1 1 17m replicaset.apps/volsync-addon-controller-2f678-deploy-64585c89c 2 2 2 15m NAME READY AGE statefulset.apps/search-redisgraph 1/1 15m NAME COMPLETIONS DURATION AGE job.batch/c5d9f22731b6141f655d08d37b3829f09b7ad1cd360f753adc93defdea8ae9d 1/1 7s 18m NAME IMAGE REPOSITORY TAGS UPDATED imagestream.image.openshift.io/acm-operator-bundle-index image-registry.openshift-image-registry.svc:5000/ocm/acm-operator-bundle-index v2.5.0 19 minutes ago NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD route.route.openshift.io/multicloud-console multicloud-console.apps.nmanos-devcluster-a2-aws.devcluster.openshift.com management-ingress https reencrypt/Redirect None $ /usr/bin/oc get all -n submariner-operator NAME READY STATUS RESTARTS AGE pod/my-submariner-catalog-stvk2 1/1 Running 0 11m pod/submariner-addon-5cfc5fbc94-b2zbf 1/1 Running 0 5m39s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/my-submariner-catalog ClusterIP 172.30.84.63 <none> 50051/TCP 11m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/submariner-addon 1/1 1 1 5m39s NAME DESIRED CURRENT READY AGE replicaset.apps/submariner-addon-5cfc5fbc94 1 1 1 5m39s NAME IMAGE REPOSITORY TAGS UPDATED imagestream.image.openshift.io/lighthouse-agent-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/lighthouse-agent-rhel8 v0.12.0 32 minutes ago imagestream.image.openshift.io/lighthouse-coredns-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/lighthouse-coredns-rhel8 v0.12.0 32 minutes ago imagestream.image.openshift.io/submariner-gateway-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-gateway-rhel8 v0.12.0 32 minutes ago imagestream.image.openshift.io/submariner-globalnet-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-globalnet-rhel8 v0.12.0 32 minutes ago imagestream.image.openshift.io/submariner-networkplugin-syncer-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-networkplugin-syncer-rhel8 v0.12.0 32 minutes ago imagestream.image.openshift.io/submariner-operator-bundle image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-operator-bundle v0.12.0 32 minutes ago imagestream.image.openshift.io/submariner-operator-bundle-index image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-operator-bundle-index v0.12.0 11 minutes ago imagestream.image.openshift.io/submariner-rhel8-operator image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-rhel8-operator v0.12.0 32 minutes ago imagestream.image.openshift.io/submariner-route-agent-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-route-agent-rhel8 v0.12.0 32 minutes ago
And also seeing in submariner-addon deployment logs - "Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource": E0311 11:56:49.244240 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws/submariner", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker" E0311 11:57:09.833219 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker" I0311 11:57:10.001819 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"ocm", Name:"submariner-addon", UID:"743f2970-4ea4-4fd3-850c-98f831819877", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManifestWorkUpdate' Updated acm-nmanos-devcluster-a2-aws/addon-submariner-deploy because it was changing E0311 11:57:24.076409 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker" I0311 11:57:24.755329 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"ocm", Name:"submariner-addon", UID:"743f2970-4ea4-4fd3-850c-98f831819877", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManifestWorkUpdate' Updated acm-nmanos-devcluster-a2-aws/addon-submariner-deploy because it was changing E0311 11:57:27.227106 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker" E0311 11:57:27.238085 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker" nmanos ▶ ~ ▶ temp ▶ $ ▶ $▶ oc logs deploy/submariner-addon --all-containers --limit-bytes=10000 --since=10m I0311 11:54:23.653281 1 leaderelection.go:258] successfully acquired lease submariner-operator/submariner-agent-lock I0311 11:54:23.653474 1 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-agent-lock", UID:"cbdd5549-685a-454b-95bf-58c3094142db", APIVersion:"v1", ResourceVersion:"496634", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' submariner-addon-5cfc5fbc94-b2zbf_2adf1a8a-8634-4f95-b6c2-9a9441840221 became leader I0311 11:54:24.858773 1 request.go:665] Waited for 1.194502324s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/admission.cluster.open-cluster-management.io/v1?timeout=32s I0311 11:54:27.266623 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"5c4c9f61-2efd-4206-a619-197d40a7b39d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "SubmarinerAgentConfigController" resync interval is set to 0s which might lead to client request throttling I0311 11:54:27.266694 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"5c4c9f61-2efd-4206-a619-197d40a7b39d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "SubmarinerAgentStatusController" resync interval is set to 0s which might lead to client request throttling I0311 11:54:27.266837 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"5c4c9f61-2efd-4206-a619-197d40a7b39d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "SubmarinerAgentStatusController" resync interval is set to 0s which might lead to client request throttling I0311 11:54:27.266890 1 base_controller.go:67] Waiting for caches to sync for SubmarinerAgentConfigController I0311 11:54:27.267059 1 base_controller.go:67] Waiting for caches to sync for SubmarinerAgentStatusController I0311 11:54:27.267094 1 base_controller.go:67] Waiting for caches to sync for SubmarinerAgentStatusController I0311 11:54:27.267110 1 base_controller.go:67] Waiting for caches to sync for SubmarinerConnectionsStatusController I0311 11:54:27.267213 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"5c4c9f61-2efd-4206-a619-197d40a7b39d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "SubmarinerConnectionsStatusController" resync interval is set to 0s which might lead to client request throttling E0311 11:54:27.270061 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource I0311 11:54:27.367174 1 base_controller.go:73] Caches are synced for SubmarinerAgentStatusController I0311 11:54:27.367209 1 base_controller.go:110] Starting #1 worker of SubmarinerAgentStatusController controller ... I0311 11:54:27.367189 1 base_controller.go:73] Caches are synced for SubmarinerAgentStatusController I0311 11:54:27.367272 1 base_controller.go:110] Starting #1 worker of SubmarinerAgentStatusController controller ... I0311 11:54:27.367317 1 base_controller.go:73] Caches are synced for SubmarinerAgentConfigController I0311 11:54:27.367338 1 base_controller.go:110] Starting #1 worker of SubmarinerAgentConfigController controller ... E0311 11:54:28.781367 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:54:31.803717 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:54:37.645326 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:54:47.757978 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:55:10.347946 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:56:00.051943 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:56:51.997248 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:57:37.894125 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:58:18.112209 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:58:48.178520 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 11:59:27.856815 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 12:00:21.825960 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 12:01:12.920977 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource E0311 12:02:09.839896 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
The issue is seen because the brokers.submariner.io object is not created in the <managedclusterset>-broker namespace. Please create a broker object with name submariner-broker in the <managedclusterset>-broker namespace as shown below. cat << EOF | oc apply -f - apiVersion: submariner.io/v1alpha1 kind: Broker metadata: name: submariner-broker namespace: <managed-cluster-set-name>-broker spec: globalnetEnabled: false EOF In the above configuration globalnet is disabled. To enable Globalnet, the value of globalnetEnabled should be set to true. The following PR is pushed few back to update the ACM documents - https://github.com/stolostron/rhacm-docs/pull/3174/files
G2Bsync 1116440695 comment mark-nc Tue, 03 May 2022 18:43:40 UTC G2Bsync The fix is now available in 2.5.0-SNAPSHOT-2022-05-02-15-39-48 or later builds. In this case where the broker has not yet been created, the submariner addon table will show that in the agent status column.
G2Bsync 1117626569 comment mark-nc Wed, 04 May 2022 17:39:38 UTC G2Bsync As the submariner addon controller is not returning a message on the status about the name and namespace of the missing broker, our fix in the UI includes a generic message is the hover text for the agent status in this case. It is a translatable string saying, `The add-on installation is unable to continue until the broker is created.` The submariner addon controller is logging a more specific message (as shown in the log output in an earlier comment in this issue), but that is not a translatable message. This message could be returned as the status condition message (with a PR for the submariner addon controller), but that is probably not best if it is not a translated string and better left to the log. If you think differently, please add a comment when verifying the current fix.
Created attachment 1878702 [details] Managed cluster status in UI - with proper error message
Created attachment 1878703 [details] Submariner Addon status in UI - with proper error message
The UI now indicates a proper message with relevant icon, both in Submariner Addon and in the Managed cluster pages.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Advanced Cluster Management 2.5 security updates, images, and bug fixes), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:4956
This comment was flagged a spam, view the edit history to see the original text if required.