Bug 2106347

Summary: Submariner error looking up service account submariner-operator/submariner-addon-sa
Product: Red Hat Advanced Cluster Management for Kubernetes Reporter: Noam Manos <nmanos>
Component: SubmarinerAssignee: Stephen Kitt <skitt>
Status: CLOSED ERRATA QA Contact: Maxim Babushkin <mbabushk>
Severity: high Docs Contact: Christopher Dawson <cdawson>
Priority: high    
Version: rhacm-2.6CC: cbynum, maafried, nyechiel, skitt
Target Milestone: ---Flags: bot-tracker-sync: rhacm-2.6+
cbynum: rhacm-2.6.z+
Target Release: rhacm-2.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-06 22:33:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Noam Manos 2022-07-12 12:09:21 UTC
**What happened**:
Installing Submariner 0.13.0 addon in ACM 2.6.0 has failed:
https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-2.6.0-Submariner-0.13.0-AWS-GCP-Globalnet/8/Test-Report/

Due to:

Warning  ClusterInfrastructureStatus       9m24s  submariner-agent                                        unable to get cluster infrastructure status, using HA cluster values for leader election: infrastructures.config.openshift.io "cluster" is forbidden: User "system:serviceaccount:submariner-operator:submariner-addon-sa" cannot get resource "infrastructures" in API group "config.openshift.io" at the cluster scope

And:

Warning  FailedCreate      19m (x2 over 19m)  replicaset-controller  Error creating: pods "submariner-addon-749f7f7469-" is forbidden: error looking up service account submariner-operator/submariner-addon-sa: serviceaccount "submariner-addon-sa" not found
  Normal   SuccessfulCreate  19m                replicaset-controller  Created pod: submariner-addon-749f7f7469-pbszn


Also, there is a missing condition on for managedclusteraddons/submariner:

$ oc  wait --timeout=15m managedclusteraddons "submariner" -n "acm-aws-nmanos-cluster-a-1" --for=condition=SubmarinerAgentDegraded=false

error: timed out waiting for the condition on managedclusteraddons/submariner


**What you expected to happen**:

managedclusteraddons/submariner should include condition: SubmarinerAgentDegraded=false


**How to reproduce it (as minimally and precisely as possible)**:

# Apply SubmarinerConfig:

$ cat <<-EOF > "SubmarinerConfig_acm-aws-nmanos-cluster-a-1.yaml" apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: acm-aws-nmanos-cluster-a-1 spec: IPSecIKEPort: 501 IPSecNATTPort: 4501 cableDriver: libreswan credentialsSecret: name: acm-aws-nmanos-cluster-a-1-aws-creds gatewayConfig: aws: instanceType: c5d.large gateways: 1 imagePullSpecs: lighthouseAgentImagePullSpec: '' lighthouseCoreDNSImagePullSpec: '' submarinerImagePullSpec: '' submarinerRouteAgentImagePullSpec: '' subscriptionConfig: channel: stable-0.13 source: submariner-stable-0-13-catalog sourceNamespace: submariner-operator startingCSV: submariner.v0.13.0 EOF


$ oc  apply -f "SubmarinerConfig_acm-aws-nmanos-cluster-a-1.yaml"
submarinerconfig.submarineraddon.open-cluster-management.io/submariner created

oc  wait --timeout=15m managedclusteraddons "submariner" -n "acm-aws-nmanos-cluster-a-1" --for=condition=SubmarinerAgentDegraded=false

# Should return:
managedclusteraddon.addon.open-cluster-management.io/submariner condition met



**Anything else we need to know?**:

**Environment**:
- Submariner version (use `subctl version`): 0.13.0 
registry-proxy.engineering.redhat.com/rh-osbs/rhacm2-submariner-operator-bundle:v0.13.0-2
Source image path: brew.registry.redhat.io/rh-osbs/iib:264412

Amazon cluster (Hub + managed cluster): OCP 4.12.0-0.nightly-2022-07-07-144231
Google cluster (managed cluster) : OCP 4.12.0-0.nightly-2022-07-07-144231

Comment 2 Stephen Kitt 2022-07-25 08:15:11 UTC
The SA is supposed to be created on the hub, it’s defined in https://github.com/stolostron/submariner-addon/blob/main/pkg/hub/submarineraddonagent/manifests/serviceaccount.yaml.

Comment 3 Stephen Kitt 2022-07-25 09:20:22 UTC
The SA is supposed to be created via ManifestWorks, see https://github.com/stolostron/submariner-addon/blob/main/pkg/hub/submarineraddonagent/agent.go#L90

Comment 4 Stephen Kitt 2022-07-25 12:45:44 UTC
From the logs:

2022-07-24T13:23:16Z   2022-07-24T13:23:16Z   1        submariner-controller                                                                                                  Normal    RoleBindingCreated                        Created RoleBinding.rbac.authorization.k8s.io/open-cluster-management:submariner-addon:agent -n acm-aws-nmanos-cluster-a2 because it was missing
2022-07-24T13:23:16Z   2022-07-24T13:23:16Z   2        replicaset-controller                                                                                                  Warning   FailedCreate                              Error creating: pods "submariner-addon-6cbff668fc-" is forbidden: error looking up service account submariner-operator/submariner-addon-sa: serviceaccount "submariner-addon-sa" not found
2022-07-24T13:23:16Z   2022-07-24T13:23:16Z   1        addon-addon-deploy-controller                                                                                          Warning   ManifestWorkUpdateFailed                  Failed to update ManifestWork acm-aws-nmanos-cluster-a2/addon-submariner-deploy: Operation cannot be fulfilled on manifestworks.work.open-cluster-management.io "addon-submariner-deploy": the object has been modified; please apply your changes to the latest version and try again
2022-07-24T13:23:16Z   2022-07-24T13:23:16Z   1        addon-csr-approving-controller                                                                                         Normal    AddonCSRAutoApproved                      addon csr "addon-acm-aws-nmanos-cluster-a2-submariner-6ksfv" is auto approved by addon csr controller
<nil>                  <nil>                  <none>   <none>                                                                                                                 Normal    Scheduled                                 Successfully assigned submariner-operator/submariner-addon-6cbff668fc-n44tm to ip-10-166-239-215.us-west-1.compute.internal
2022-07-24T13:23:16Z   2022-07-24T13:23:16Z   1        submariner-controller                                                                                                  Normal    RoleCreated                               Created Role.rbac.authorization.k8s.io/open-cluster-management:submariner-addon:agent -n acm-aws-nmanos-cluster-a2 because it was missing
2022-07-24T13:23:16Z   2022-07-24T13:23:16Z   1        work-agent-manifestworkagent                                                                                           Normal    NamespaceUpdated                          Updated Namespace/submariner-operator because it changed
2022-07-24T13:23:16Z   2022-07-24T13:23:16Z   1        work-agent-manifestworkagent                                                                                           Normal    ServiceAccountCreated                     Created ServiceAccount/submariner-addon-sa -n submariner-operator because it was missing


The submariner-addon-sa account is created after the replicaset which wants to use it.

Comment 5 Stephen Kitt 2022-07-27 09:54:17 UTC
Fixed by https://github.com/stolostron/submariner-addon/pull/460

Comment 9 errata-xmlrpc 2022-09-06 22:33:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.6.0 security updates and bug fixes), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6370