**What happened**: Installing Submariner 0.13.0 addon in ACM 2.6.0 has failed: https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-2.6.0-Submariner-0.13.0-AWS-GCP-Globalnet/8/Test-Report/ Due to: Warning ClusterInfrastructureStatus 9m24s submariner-agent unable to get cluster infrastructure status, using HA cluster values for leader election: infrastructures.config.openshift.io "cluster" is forbidden: User "system:serviceaccount:submariner-operator:submariner-addon-sa" cannot get resource "infrastructures" in API group "config.openshift.io" at the cluster scope And: Warning FailedCreate 19m (x2 over 19m) replicaset-controller Error creating: pods "submariner-addon-749f7f7469-" is forbidden: error looking up service account submariner-operator/submariner-addon-sa: serviceaccount "submariner-addon-sa" not found Normal SuccessfulCreate 19m replicaset-controller Created pod: submariner-addon-749f7f7469-pbszn Also, there is a missing condition on for managedclusteraddons/submariner: $ oc wait --timeout=15m managedclusteraddons "submariner" -n "acm-aws-nmanos-cluster-a-1" --for=condition=SubmarinerAgentDegraded=false error: timed out waiting for the condition on managedclusteraddons/submariner **What you expected to happen**: managedclusteraddons/submariner should include condition: SubmarinerAgentDegraded=false **How to reproduce it (as minimally and precisely as possible)**: # Apply SubmarinerConfig: $ cat <<-EOF > "SubmarinerConfig_acm-aws-nmanos-cluster-a-1.yaml" apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: acm-aws-nmanos-cluster-a-1 spec: IPSecIKEPort: 501 IPSecNATTPort: 4501 cableDriver: libreswan credentialsSecret: name: acm-aws-nmanos-cluster-a-1-aws-creds gatewayConfig: aws: instanceType: c5d.large gateways: 1 imagePullSpecs: lighthouseAgentImagePullSpec: '' lighthouseCoreDNSImagePullSpec: '' submarinerImagePullSpec: '' submarinerRouteAgentImagePullSpec: '' subscriptionConfig: channel: stable-0.13 source: submariner-stable-0-13-catalog sourceNamespace: submariner-operator startingCSV: submariner.v0.13.0 EOF $ oc apply -f "SubmarinerConfig_acm-aws-nmanos-cluster-a-1.yaml" submarinerconfig.submarineraddon.open-cluster-management.io/submariner created oc wait --timeout=15m managedclusteraddons "submariner" -n "acm-aws-nmanos-cluster-a-1" --for=condition=SubmarinerAgentDegraded=false # Should return: managedclusteraddon.addon.open-cluster-management.io/submariner condition met **Anything else we need to know?**: **Environment**: - Submariner version (use `subctl version`): 0.13.0 registry-proxy.engineering.redhat.com/rh-osbs/rhacm2-submariner-operator-bundle:v0.13.0-2 Source image path: brew.registry.redhat.io/rh-osbs/iib:264412 Amazon cluster (Hub + managed cluster): OCP 4.12.0-0.nightly-2022-07-07-144231 Google cluster (managed cluster) : OCP 4.12.0-0.nightly-2022-07-07-144231
Noam saw this on OCP 4.9 with Kubernetes 1.22: https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-2.5.2-Submariner-0.12.2-AWS-OSP-Globalnet/3/Test-Report/
The SA is supposed to be created on the hub, it’s defined in https://github.com/stolostron/submariner-addon/blob/main/pkg/hub/submarineraddonagent/manifests/serviceaccount.yaml.
The SA is supposed to be created via ManifestWorks, see https://github.com/stolostron/submariner-addon/blob/main/pkg/hub/submarineraddonagent/agent.go#L90
From the logs: 2022-07-24T13:23:16Z 2022-07-24T13:23:16Z 1 submariner-controller Normal RoleBindingCreated Created RoleBinding.rbac.authorization.k8s.io/open-cluster-management:submariner-addon:agent -n acm-aws-nmanos-cluster-a2 because it was missing 2022-07-24T13:23:16Z 2022-07-24T13:23:16Z 2 replicaset-controller Warning FailedCreate Error creating: pods "submariner-addon-6cbff668fc-" is forbidden: error looking up service account submariner-operator/submariner-addon-sa: serviceaccount "submariner-addon-sa" not found 2022-07-24T13:23:16Z 2022-07-24T13:23:16Z 1 addon-addon-deploy-controller Warning ManifestWorkUpdateFailed Failed to update ManifestWork acm-aws-nmanos-cluster-a2/addon-submariner-deploy: Operation cannot be fulfilled on manifestworks.work.open-cluster-management.io "addon-submariner-deploy": the object has been modified; please apply your changes to the latest version and try again 2022-07-24T13:23:16Z 2022-07-24T13:23:16Z 1 addon-csr-approving-controller Normal AddonCSRAutoApproved addon csr "addon-acm-aws-nmanos-cluster-a2-submariner-6ksfv" is auto approved by addon csr controller <nil> <nil> <none> <none> Normal Scheduled Successfully assigned submariner-operator/submariner-addon-6cbff668fc-n44tm to ip-10-166-239-215.us-west-1.compute.internal 2022-07-24T13:23:16Z 2022-07-24T13:23:16Z 1 submariner-controller Normal RoleCreated Created Role.rbac.authorization.k8s.io/open-cluster-management:submariner-addon:agent -n acm-aws-nmanos-cluster-a2 because it was missing 2022-07-24T13:23:16Z 2022-07-24T13:23:16Z 1 work-agent-manifestworkagent Normal NamespaceUpdated Updated Namespace/submariner-operator because it changed 2022-07-24T13:23:16Z 2022-07-24T13:23:16Z 1 work-agent-manifestworkagent Normal ServiceAccountCreated Created ServiceAccount/submariner-addon-sa -n submariner-operator because it was missing The submariner-addon-sa account is created after the replicaset which wants to use it.
Fixed by https://github.com/stolostron/submariner-addon/pull/460
Verified: https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/view/ACM%202.6/job/ACM-2.6.0-Submariner-0.13.0-AWS-GCP-Globalnet/51/Test-Report/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.6.0 security updates and bug fixes), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:6370