Bug 2062787
| Summary: | Submariner Add-on UI does not indicate on Broker error | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Advanced Cluster Management for Kubernetes | Reporter: | Noam Manos <nmanos> | ||||||
| Component: | Console | Assignee: | Kevin Cormier <kcormier> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Noam Manos <nmanos> | ||||||
| Severity: | medium | Docs Contact: | Christopher Dawson <cdawson> | ||||||
| Priority: | unspecified | ||||||||
| Version: | rhacm-2.5 | CC: | dho, dhuynh, huichen, nyechiel, sgaddam | ||||||
| Target Milestone: | --- | Flags: | bot-tracker-sync:
rhacm-2.5+
|
||||||
| Target Release: | rhacm-2.5 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2022-06-09 02:09:24 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Can you share the following form the cluster that is not working: 1. oc get managedclusteraddons -n <clustername> submariner -o yaml 2. Log output of the submariner-addon-agent 3. Status of all pods in submariner-operator namespace. The relevant status messages show up as deployment comes up. Missing status messages likely means something went wrong during deployment. Small correction, pod running submariner-addon-agent is just called submariner-addon. Labels: app=submariner-addon Here are the required logs (it's a new test run, but with the same bug):
$ oc describe managedclusteraddon -n acm-nmanos-devcluster-a2-aws submariner
Name: submariner
Namespace: acm-nmanos-devcluster-a2-aws
Labels: <none>
Annotations: <none>
API Version: addon.open-cluster-management.io/v1alpha1
Kind: ManagedClusterAddOn
Metadata:
Creation Timestamp: 2022-03-11T08:18:43Z
Finalizers:
submarineraddon.open-cluster-management.io/submariner-addon-cleanup
Generation: 1
Managed Fields:
API Version: addon.open-cluster-management.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:installNamespace:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-03-11T08:18:43Z
API Version: addon.open-cluster-management.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"submarineraddon.open-cluster-management.io/submariner-addon-cleanup":
Manager: submariner
Operation: Update
Time: 2022-03-11T08:18:46Z
API Version: addon.open-cluster-management.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:addOnConfiguration:
.:
f:crName:
f:crdName:
f:addOnMeta:
.:
f:description:
f:displayName:
f:conditions:
f:healthCheck:
.:
f:mode:
f:registrations:
Manager: submariner
Operation: Update
Subresource: status
Time: 2022-03-11T08:21:23Z
Resource Version: 80275
UID: 98917d45-35f1-4684-829d-679ff240ee48
Spec:
Install Namespace: submariner-operator
Status:
Add On Configuration:
Cr Name:
Crd Name:
Add On Meta:
Description:
Display Name:
Conditions:
Last Transition Time: 2022-03-11T08:18:43Z
Message: Registration of the addon agent is configured
Reason: RegistrationConfigured
Status: True
Type: RegistrationApplied
Last Transition Time: 2022-03-11T08:18:44Z
Message: manifest of addon applied successfully
Reason: AddonManifestApplied
Status: True
Type: ManifestApplied
Last Transition Time: 2022-03-11T08:18:49Z
Message:
Reason: BrokerConfigMissing
Status: False
Type: SubmarinerBrokerConfigApplied
Last Transition Time: 2022-03-11T08:21:23Z
Message: The nodes "ip-10-0-5-147.us-west-1.compute.internal" are labeled with "submariner.io/gateway"
Reason: SubmarinerGatewayNodesLabeled
Status: True
Type: SubmarinerGatewayNodesLabeled
Last Transition Time: 2022-03-11T08:19:11Z
Message: submariner add-on is available.
Reason: ManagedClusterAddOnLeaseUpdated
Status: True
Type: Available
Health Check:
Mode: Lease
Registrations:
Signer Name: kubernetes.io/kube-apiserver-client
Subject:
Groups:
system:open-cluster-management:cluster:acm-nmanos-devcluster-a2-aws:addon:submariner
system:open-cluster-management:addon:submariner
system:authenticated
User: system:open-cluster-management:cluster:acm-nmanos-devcluster-a2-aws:addon:submariner:agent:submariner-addon-agent
Events: <none>
#######################################################################
# ACM and Submariner resources in nmanos-devcluster-a2-aws #
#######################################################################
$ /usr/bin/oc get all -n ocm
NAME READY STATUS RESTARTS AGE
pod/application-chart-160df-applicationui-5cccc47668-mjqdq 1/1 Running 0 16m
pod/application-chart-160df-applicationui-5cccc47668-wcxls 1/1 Running 0 16m
pod/application-chart-160df-consoleapi-7b8945459c-kmj25 1/1 Running 0 16m
pod/application-chart-160df-consoleapi-7b8945459c-n5wj6 1/1 Running 0 16m
pod/c5d9f22731b6141f655d08d37b3829f09b7ad1cd360f753adc93de--1-zj55g 0/1 Completed 0 18m
pod/console-chart-4c510-console-v2-77569d4b94-2qfs4 1/1 Running 0 15m
pod/console-chart-4c510-console-v2-77569d4b94-bs264 1/1 Running 0 15m
pod/grc-06e40-grcui-7c7f475649-b6lfg 1/1 Running 0 15m
pod/grc-06e40-grcui-7c7f475649-gfwb4 1/1 Running 0 15m
pod/grc-06e40-grcuiapi-69bc87cb4c-kr9jx 1/1 Running 0 15m
pod/grc-06e40-grcuiapi-69bc87cb4c-wx8cj 1/1 Running 0 15m
pod/grc-06e40-policy-addon-controller-5c567f4856-6n76g 1/1 Running 0 15m
pod/grc-06e40-policy-addon-controller-5c567f4856-dmv8j 1/1 Running 0 15m
pod/grc-06e40-policy-propagator-597f459b9d-6j8rt 2/2 Running 0 15m
pod/grc-06e40-policy-propagator-597f459b9d-mt5v8 2/2 Running 0 15m
pod/klusterlet-addon-controller-v2-6655d49f5c-tcnz4 1/1 Running 0 15m
pod/klusterlet-addon-controller-v2-6655d49f5c-v89gc 1/1 Running 0 15m
pod/management-ingress-46dfe-6fd69489b4-fc5t4 2/2 Running 0 15m
pod/management-ingress-46dfe-6fd69489b4-zbsj9 2/2 Running 0 15m
pod/multicluster-observability-operator-67f47c5556-xnqjz 1/1 Running 0 17m
pod/multicluster-operators-application-6dfc894b88-qdwdg 3/3 Running 1 (17m ago) 17m
pod/multicluster-operators-channel-66f45746b7-566ll 1/1 Running 1 (16m ago) 17m
pod/multicluster-operators-hub-subscription-6c8d8b845b-grf5j 1/1 Running 1 (17m ago) 17m
pod/multicluster-operators-standalone-subscription-858cc89b66-k4zhr 1/1 Running 0 17m
pod/multicluster-operators-subscription-report-5f999975f5-nhvpc 1/1 Running 0 17m
pod/multiclusterhub-operator-56cc7588b8-kt9rv 1/1 Running 0 17m
pod/multiclusterhub-repo-d54db874c-fhp7c 1/1 Running 0 16m
pod/my-acm-catalog-qk6kb 1/1 Running 0 19m
pod/policyreport-3dc9b-insights-client-74f799b75b-t8hr7 1/1 Running 0 15m
pod/policyreport-3dc9b-metrics-68b6c8cc88-226gw 2/2 Running 0 15m
pod/search-operator-f4bcc7c8c-h72cf 1/1 Running 0 15m
pod/search-prod-dc192-search-aggregator-85b78649d5-7b5f2 1/1 Running 0 15m
pod/search-prod-dc192-search-api-567555db89-6sn6t 1/1 Running 0 15m
pod/search-prod-dc192-search-api-567555db89-wspcq 1/1 Running 0 15m
pod/search-prod-dc192-search-collector-74494857fd-8bhj9 1/1 Running 0 15m
pod/search-redisgraph-0 1/1 Running 0 15m
pod/submariner-addon-665b595ff5-s5xkr 1/1 Running 0 17m
pod/volsync-addon-controller-2f678-deploy-64585c89c-gfkz8 1/1 Running 0 15m
pod/volsync-addon-controller-2f678-deploy-64585c89c-p98lx 1/1 Running 0 15m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/application-chart-160df-applicationui ClusterIP 172.30.227.206 <none> 3001/TCP 16m
service/channels-apps-open-cluster-management-webhook-svc ClusterIP 172.30.204.10 <none> 443/TCP 16m
service/console-api ClusterIP 172.30.6.235 <none> 4000/TCP 16m
service/console-chart-4c510-console-v2 ClusterIP 172.30.209.158 <none> 3000/TCP 15m
service/grc-06e40-grcui ClusterIP 172.30.237.233 <none> 3000/TCP 15m
service/grc-06e40-grcuiapi ClusterIP 172.30.52.1 <none> 4000/TCP 15m
service/grc-06e40-policy-propagator-metrics ClusterIP 172.30.244.152 <none> 8443/TCP 15m
service/insights-client ClusterIP 172.30.151.236 <none> 3030/TCP 15m
service/insights-metrics ClusterIP 172.30.94.22 <none> 8443/TCP 15m
service/management-ingress ClusterIP 172.30.132.92 <none> 443/TCP,8443/TCP 15m
service/multicluster-observability-webhook-service ClusterIP 172.30.255.234 <none> 443/TCP 17m
service/multicluster-operators-application-svc ClusterIP 172.30.213.49 <none> 443/TCP 16m
service/multicluster-operators-subscription ClusterIP 172.30.138.79 <none> 8443/TCP 17m
service/multiclusterhub-operator-webhook ClusterIP 172.30.156.8 <none> 443/TCP 17m
service/multiclusterhub-repo ClusterIP 172.30.222.240 <none> 3000/TCP 16m
service/my-acm-catalog ClusterIP 172.30.178.76 <none> 50051/TCP 19m
service/search-aggregator ClusterIP 172.30.108.216 <none> 3010/TCP 15m
service/search-prod-dc192-search-redisgraph ClusterIP 172.30.87.213 <none> 6380/TCP 15m
service/search-search-api ClusterIP 172.30.183.104 <none> 4010/TCP 15m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/application-chart-160df-applicationui 2/2 2 2 16m
deployment.apps/application-chart-160df-consoleapi 2/2 2 2 16m
deployment.apps/console-chart-4c510-console-v2 2/2 2 2 15m
deployment.apps/grc-06e40-grcui 2/2 2 2 15m
deployment.apps/grc-06e40-grcuiapi 2/2 2 2 15m
deployment.apps/grc-06e40-policy-addon-controller 2/2 2 2 15m
deployment.apps/grc-06e40-policy-propagator 2/2 2 2 15m
deployment.apps/klusterlet-addon-controller-v2 2/2 2 2 15m
deployment.apps/management-ingress-46dfe 2/2 2 2 15m
deployment.apps/multicluster-observability-operator 1/1 1 1 17m
deployment.apps/multicluster-operators-application 1/1 1 1 17m
deployment.apps/multicluster-operators-channel 1/1 1 1 17m
deployment.apps/multicluster-operators-hub-subscription 1/1 1 1 17m
deployment.apps/multicluster-operators-standalone-subscription 1/1 1 1 17m
deployment.apps/multicluster-operators-subscription-report 1/1 1 1 17m
deployment.apps/multiclusterhub-operator 1/1 1 1 17m
deployment.apps/multiclusterhub-repo 1/1 1 1 16m
deployment.apps/policyreport-3dc9b-insights-client 1/1 1 1 15m
deployment.apps/policyreport-3dc9b-metrics 1/1 1 1 15m
deployment.apps/search-operator 1/1 1 1 15m
deployment.apps/search-prod-dc192-search-aggregator 1/1 1 1 15m
deployment.apps/search-prod-dc192-search-api 2/2 2 2 15m
deployment.apps/search-prod-dc192-search-collector 1/1 1 1 15m
deployment.apps/submariner-addon 1/1 1 1 17m
deployment.apps/volsync-addon-controller-2f678-deploy 2/2 2 2 15m
NAME DESIRED CURRENT READY AGE
replicaset.apps/application-chart-160df-applicationui-5cccc47668 2 2 2 16m
replicaset.apps/application-chart-160df-consoleapi-7b8945459c 2 2 2 16m
replicaset.apps/console-chart-4c510-console-v2-77569d4b94 2 2 2 15m
replicaset.apps/grc-06e40-grcui-7c7f475649 2 2 2 15m
replicaset.apps/grc-06e40-grcuiapi-69bc87cb4c 2 2 2 15m
replicaset.apps/grc-06e40-policy-addon-controller-5c567f4856 2 2 2 15m
replicaset.apps/grc-06e40-policy-propagator-597f459b9d 2 2 2 15m
replicaset.apps/klusterlet-addon-controller-v2-6655d49f5c 2 2 2 15m
replicaset.apps/management-ingress-46dfe-6fd69489b4 2 2 2 15m
replicaset.apps/multicluster-observability-operator-67f47c5556 1 1 1 17m
replicaset.apps/multicluster-operators-application-6dfc894b88 1 1 1 17m
replicaset.apps/multicluster-operators-channel-66f45746b7 1 1 1 17m
replicaset.apps/multicluster-operators-hub-subscription-6c8d8b845b 1 1 1 17m
replicaset.apps/multicluster-operators-standalone-subscription-858cc89b66 1 1 1 17m
replicaset.apps/multicluster-operators-subscription-report-5f999975f5 1 1 1 17m
replicaset.apps/multiclusterhub-operator-56cc7588b8 1 1 1 17m
replicaset.apps/multiclusterhub-repo-d54db874c 1 1 1 16m
replicaset.apps/policyreport-3dc9b-insights-client-74f799b75b 1 1 1 15m
replicaset.apps/policyreport-3dc9b-metrics-68b6c8cc88 1 1 1 15m
replicaset.apps/search-operator-f4bcc7c8c 1 1 1 15m
replicaset.apps/search-prod-dc192-search-aggregator-85b78649d5 1 1 1 15m
replicaset.apps/search-prod-dc192-search-api-567555db89 2 2 2 15m
replicaset.apps/search-prod-dc192-search-collector-74494857fd 1 1 1 15m
replicaset.apps/submariner-addon-665b595ff5 1 1 1 17m
replicaset.apps/volsync-addon-controller-2f678-deploy-64585c89c 2 2 2 15m
NAME READY AGE
statefulset.apps/search-redisgraph 1/1 15m
NAME COMPLETIONS DURATION AGE
job.batch/c5d9f22731b6141f655d08d37b3829f09b7ad1cd360f753adc93defdea8ae9d 1/1 7s 18m
NAME IMAGE REPOSITORY TAGS UPDATED
imagestream.image.openshift.io/acm-operator-bundle-index image-registry.openshift-image-registry.svc:5000/ocm/acm-operator-bundle-index v2.5.0 19 minutes ago
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/multicloud-console multicloud-console.apps.nmanos-devcluster-a2-aws.devcluster.openshift.com management-ingress https reencrypt/Redirect None
$ /usr/bin/oc get all -n submariner-operator
NAME READY STATUS RESTARTS AGE
pod/my-submariner-catalog-stvk2 1/1 Running 0 11m
pod/submariner-addon-5cfc5fbc94-b2zbf 1/1 Running 0 5m39s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-submariner-catalog ClusterIP 172.30.84.63 <none> 50051/TCP 11m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/submariner-addon 1/1 1 1 5m39s
NAME DESIRED CURRENT READY AGE
replicaset.apps/submariner-addon-5cfc5fbc94 1 1 1 5m39s
NAME IMAGE REPOSITORY TAGS UPDATED
imagestream.image.openshift.io/lighthouse-agent-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/lighthouse-agent-rhel8 v0.12.0 32 minutes ago
imagestream.image.openshift.io/lighthouse-coredns-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/lighthouse-coredns-rhel8 v0.12.0 32 minutes ago
imagestream.image.openshift.io/submariner-gateway-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-gateway-rhel8 v0.12.0 32 minutes ago
imagestream.image.openshift.io/submariner-globalnet-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-globalnet-rhel8 v0.12.0 32 minutes ago
imagestream.image.openshift.io/submariner-networkplugin-syncer-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-networkplugin-syncer-rhel8 v0.12.0 32 minutes ago
imagestream.image.openshift.io/submariner-operator-bundle image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-operator-bundle v0.12.0 32 minutes ago
imagestream.image.openshift.io/submariner-operator-bundle-index image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-operator-bundle-index v0.12.0 11 minutes ago
imagestream.image.openshift.io/submariner-rhel8-operator image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-rhel8-operator v0.12.0 32 minutes ago
imagestream.image.openshift.io/submariner-route-agent-rhel8 image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-route-agent-rhel8 v0.12.0 32 minutes ago
And also seeing in submariner-addon deployment logs - "Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource":
E0311 11:56:49.244240 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws/submariner", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker"
E0311 11:57:09.833219 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker"
I0311 11:57:10.001819 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"ocm", Name:"submariner-addon", UID:"743f2970-4ea4-4fd3-850c-98f831819877", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManifestWorkUpdate' Updated acm-nmanos-devcluster-a2-aws/addon-submariner-deploy because it was changing
E0311 11:57:24.076409 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker"
I0311 11:57:24.755329 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"ocm", Name:"submariner-addon", UID:"743f2970-4ea4-4fd3-850c-98f831819877", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManifestWorkUpdate' Updated acm-nmanos-devcluster-a2-aws/addon-submariner-deploy because it was changing
E0311 11:57:27.227106 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker"
E0311 11:57:27.238085 1 base_controller.go:270] "SubmarinerAgentController" controller failed to sync "acm-nmanos-devcluster-a2-aws", err: brokers.submariner.io object named "submariner-broker" missing in namespace "submariner-broker"
nmanos ▶ ~ ▶ temp ▶ $ ▶
$▶ oc logs deploy/submariner-addon --all-containers --limit-bytes=10000 --since=10m
I0311 11:54:23.653281 1 leaderelection.go:258] successfully acquired lease submariner-operator/submariner-agent-lock
I0311 11:54:23.653474 1 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-agent-lock", UID:"cbdd5549-685a-454b-95bf-58c3094142db", APIVersion:"v1", ResourceVersion:"496634", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' submariner-addon-5cfc5fbc94-b2zbf_2adf1a8a-8634-4f95-b6c2-9a9441840221 became leader
I0311 11:54:24.858773 1 request.go:665] Waited for 1.194502324s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/admission.cluster.open-cluster-management.io/v1?timeout=32s
I0311 11:54:27.266623 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"5c4c9f61-2efd-4206-a619-197d40a7b39d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "SubmarinerAgentConfigController" resync interval is set to 0s which might lead to client request throttling
I0311 11:54:27.266694 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"5c4c9f61-2efd-4206-a619-197d40a7b39d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "SubmarinerAgentStatusController" resync interval is set to 0s which might lead to client request throttling
I0311 11:54:27.266837 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"5c4c9f61-2efd-4206-a619-197d40a7b39d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "SubmarinerAgentStatusController" resync interval is set to 0s which might lead to client request throttling
I0311 11:54:27.266890 1 base_controller.go:67] Waiting for caches to sync for SubmarinerAgentConfigController
I0311 11:54:27.267059 1 base_controller.go:67] Waiting for caches to sync for SubmarinerAgentStatusController
I0311 11:54:27.267094 1 base_controller.go:67] Waiting for caches to sync for SubmarinerAgentStatusController
I0311 11:54:27.267110 1 base_controller.go:67] Waiting for caches to sync for SubmarinerConnectionsStatusController
I0311 11:54:27.267213 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"submariner-operator", Name:"submariner-addon", UID:"5c4c9f61-2efd-4206-a619-197d40a7b39d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "SubmarinerConnectionsStatusController" resync interval is set to 0s which might lead to client request throttling
E0311 11:54:27.270061 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
I0311 11:54:27.367174 1 base_controller.go:73] Caches are synced for SubmarinerAgentStatusController
I0311 11:54:27.367209 1 base_controller.go:110] Starting #1 worker of SubmarinerAgentStatusController controller ...
I0311 11:54:27.367189 1 base_controller.go:73] Caches are synced for SubmarinerAgentStatusController
I0311 11:54:27.367272 1 base_controller.go:110] Starting #1 worker of SubmarinerAgentStatusController controller ...
I0311 11:54:27.367317 1 base_controller.go:73] Caches are synced for SubmarinerAgentConfigController
I0311 11:54:27.367338 1 base_controller.go:110] Starting #1 worker of SubmarinerAgentConfigController controller ...
E0311 11:54:28.781367 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:54:31.803717 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:54:37.645326 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:54:47.757978 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:55:10.347946 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:56:00.051943 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:56:51.997248 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:57:37.894125 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:58:18.112209 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:58:48.178520 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 11:59:27.856815 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 12:00:21.825960 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 12:01:12.920977 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0311 12:02:09.839896 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
The issue is seen because the brokers.submariner.io object is not created in the <managedclusterset>-broker namespace.
Please create a broker object with name submariner-broker in the <managedclusterset>-broker namespace as shown below.
cat << EOF | oc apply -f -
apiVersion: submariner.io/v1alpha1
kind: Broker
metadata:
name: submariner-broker
namespace: <managed-cluster-set-name>-broker
spec:
globalnetEnabled: false
EOF
In the above configuration globalnet is disabled. To enable Globalnet, the value of globalnetEnabled should be set to true.
The following PR is pushed few back to update the ACM documents - https://github.com/stolostron/rhacm-docs/pull/3174/files
G2Bsync 1116440695 comment mark-nc Tue, 03 May 2022 18:43:40 UTC G2Bsync The fix is now available in 2.5.0-SNAPSHOT-2022-05-02-15-39-48 or later builds. In this case where the broker has not yet been created, the submariner addon table will show that in the agent status column. G2Bsync 1117626569 comment mark-nc Wed, 04 May 2022 17:39:38 UTC G2Bsync As the submariner addon controller is not returning a message on the status about the name and namespace of the missing broker, our fix in the UI includes a generic message is the hover text for the agent status in this case. It is a translatable string saying, `The add-on installation is unable to continue until the broker is created.` The submariner addon controller is logging a more specific message (as shown in the log output in an earlier comment in this issue), but that is not a translatable message. This message could be returned as the status condition message (with a PR for the submariner addon controller), but that is probably not best if it is not a translated string and better left to the log. If you think differently, please add a comment when verifying the current fix. Created attachment 1878702 [details]
Managed cluster status in UI - with proper error message
Created attachment 1878703 [details]
Submariner Addon status in UI - with proper error message
The UI now indicates a proper message with relevant icon, both in Submariner Addon and in the Managed cluster pages. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Advanced Cluster Management 2.5 security updates, images, and bug fixes), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:4956 This comment was flagged a spam, view the edit history to see the original text if required. |
**What happened**: In ACM 2.5.0, when submariner broker is missing, then Submariner config does not get all conditions ready. In ACM console UI under "Submariner Add-ons", the "Connection status" and "Agent status" icon keeps rolling with "Progressing" mesage, instead of displaying proper error message (e.g. "Broker is missing") Looking at the spec of the created Submariner config object - it does not show any error. However not all expected conditions (for example "SubmarinerConfigApplied") were listed: Spec: IP Sec IKE Port: 502 IP Sec NATT Port: 4502 NATT Discovery Port: 4900 NATT Enable: true Cable Driver: libreswan Credentials Secret: Name: acm-nmanos-devcluster-a2-aws-aws-creds Gateway Config: Aws: Instance Type: c5d.large Gateways: 1 Image Pull Specs: Load Balancer Enable: false Subscription Config: Channel: alpha-0.12 Source: my-submariner-catalog Source Namespace: submariner-operator Starting CSV: submariner.v0.12.0 Status: Conditions: Last Transition Time: 2022-03-10T08:15:06Z Message: Submariner cluster environment was prepared Reason: SubmarinerClusterEnvPrepared Status: True Type: SubmarinerClusterEnvironmentPrepared Last Transition Time: 2022-03-10T08:17:53Z Message: 1 node(s) ("ip-10-0-72-198.us-west-1.compute.internal") are labeled as gateways Reason: Success Status: True Type: SubmarinerGatewaysLabeled Managed Cluster Info: Cluster Name: acm-nmanos-devcluster-a2-aws Infra Id: nmanos-devcluster-a2-hqwtp Platform: AWS Region: us-west-1 Vendor: OpenShift Events: <none> **What you expected to happen**: If Submariner Broker is missing, ACM UI should indicate it, and stop showing false "Progressing" status. **How to reproduce it (as minimally and precisely as possible)**: 1) Install ACM 2.5.0 with Submariner 0.12.0 operator. 2) Do NOT create Submariner Broker. 3) Create this submariner config: apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: acm-nmanos-devcluster-a2-aws spec: IPSecIKEPort: 502 IPSecNATTPort: 4502 cableDriver: libreswan credentialsSecret: name: acm-nmanos-devcluster-a2-aws-aws-creds gatewayConfig: aws: instanceType: c5d.large gateways: 1 imagePullSpecs: lighthouseAgentImagePullSpec: '' lighthouseCoreDNSImagePullSpec: '' submarinerImagePullSpec: '' submarinerRouteAgentImagePullSpec: '' subscriptionConfig: channel: alpha-0.12 source: my-submariner-catalog sourceNamespace: submariner-operator startingCSV: submariner.v0.12.0 **Anything else we need to know?**: See full scenario here: https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/ACM-2.5.0-Submariner-0.12.0-AWS-OSP/55/Test-Report/ **Environment**: # Cloud platform: Amazon # OCP version: 4.9.0 # ACM version: 2.5.0 ### Submariner components ### subctl version: v0.12.0 Cluster "api-nmanos-devcluster-a2-aws-devcluster-openshift-com:6443" • Submariner is not installed ### submariner-operator-bundle-index Image ### id=image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-operator-bundle-index@sha256:e42781acae3cd6eca70f5928a87a0cd106afe9cbf0b29b89dfb47f7a91218536 name=openshift/ose-operator-registry release=202202120107.p0.g3cd421d.assembly.stream url =https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.9.0-202202120107.p0.g3cd421d.assembly.stream version=v4.9.0 ### submariner-addon-rhel8 Image ### id=registry.redhat.io/rhacm2/submariner-addon-rhel8@sha256:7fabdd2b39a029c9ddffe46299c4246bc5b434998d49a4ba13c64e3b05dfc052 ################################################################### # ImageStream Tags (in namespace submariner-operator) # ################################################################### ### lighthouse-agent-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/lighthouse-agent-rhel8 release=9 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-agent-rhel8/images/v0.12.0-9 version=v0.12.0 ### lighthouse-coredns-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/lighthouse-coredns-rhel8 release=10 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-coredns-rhel8/images/v0.12.0-10 version=v0.12.0 ### submariner-gateway-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-gateway-rhel8 release=14 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-gateway-rhel8/images/v0.12.0-14 version=v0.12.0 ### submariner-globalnet-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-globalnet-rhel8 release=14 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-globalnet-rhel8/images/v0.12.0-14 version=v0.12.0 ### submariner-networkplugin-syncer-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-networkplugin-syncer-rhel8 release=14 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-networkplugin-syncer-rhel8/images/v0.12.0-14 version=v0.12.0 ### submariner-operator-bundle:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-operator-bundle release=12 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-operator-bundle/images/v0.12.0-12 version=v0.12.0 ### submariner-operator-bundle-index:v0.12.0 Image-Stream tag ### name=openshift/ose-operator-registry release=202202120107.p0.g3cd421d.assembly.stream url =https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.9.0-202202120107.p0.g3cd421d.assembly.stream version=v4.9.0 ### submariner-rhel8-operator:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-rhel8-operator release=24 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-rhel8-operator/images/v0.12.0-24 version=v0.12.0 ### submariner-route-agent-rhel8:v0.12.0 Image-Stream tag ### name=rhacm2-tech-preview/submariner-route-agent-rhel8 release=13 url =https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-route-agent-rhel8/images/v0.12.0-13 version=v0.12.0