Bug 2091170 - Unable to deploy Hypershift operator on MCE hub using ManagedClusterAddOn without including s3 secret
Summary: Unable to deploy Hypershift operator on MCE hub using ManagedClusterAddOn wit...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Advanced Cluster Management for Kubernetes
Classification: Red Hat
Component: Hypershift
Version: rhacm-2.5
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: rhacm-2.5.1
Assignee: Roke Jung
QA Contact: Chad Crum
URL:
Whiteboard:
Depends On:
Blocks: 2100116
TreeView+ depends on / blocked
 
Reported: 2022-05-27 17:30 UTC by Chad Crum
Modified: 2022-09-13 20:06 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2100116 (view as bug list)
Environment:
Last Closed: 2022-09-13 20:06:21 UTC
Target Upstream Version:
Embargoed:
bot-tracker-sync: rhacm-2.5.z+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github stolostron backlog issues 22814 0 None None None 2022-05-27 19:05:02 UTC
Red Hat Product Errata RHSA-2022:6507 0 None None None 2022-09-13 20:06:30 UTC

Description Chad Crum 2022-05-27 17:30:02 UTC
Description of problem:
In the case of using MCE/Hypershift on BM (As opposed to AWS provider) and using agent based hosted control planes, there is no need for s3 secret "hypershift-operator-oidc-provider-s3-credentials", yet the hypershift operator pod will not deploy without the s3 secret set.


Version-Release number of selected component (if applicable):
Latest MCE2.0 downstream snapshot
OCP 4.10.x hub cluster

How reproducible:
100%

Steps to Reproduce:
1. Ensure hub cluster has a managedcluster resource
2. Create managedclusteraddon, referencing hub managedcluster
oc apply -f - <<EOF
apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: hypershift-addon
  namespace: local-cluster   # name of managed cluster
spec:
  installNamespace: open-cluster-management-agent-addon
EOF

3. Check managedclusteraddon status in the local-cluster namespace

Actual results:
Hypershift addon remains unknown

oc get managedclusteraddon -n local-cluster
NAME               AVAILABLE   DEGRADED   PROGRESSING
hypershift-addon   Unknown                
work-manager       True           


Expected results:
AVAILABLE moves to true and hypershift operator deploys in hypershift operator name space

Additional info:

## If I do not create the S3 key: 


# hypershift-addon shows the following errors and stays unknown
 oc get managedclusteraddons.addon.open-cluster-management.io 
NAME               AVAILABLE   DEGRADED   PROGRESSING
hypershift-addon   Unknown                
work-manager       True       


    Last Transition Time:  2022-05-27T17:16:57Z
    Message:               The status of hypershift-addon add-on is unknown.
    Reason:                ManagedClusterAddOnLeaseNotFound
    Status:                Unknown
    Type:                  Available


# Hypershift addon gent pod tries to launch hypershift with the s3 secret, but the second is not created so hypershift never runs.

oc logs -f hypershift-addon-agent-5499c58967-cz8vk -n open-cluster-management-agent-addon                                                                                                                                                                   
I0527 17:24:17.124602       1 request.go:665] Waited for 1.028914334s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v1alpha2?timeout=32s                                                              
2022-05-27T17:24:23.698Z        INFO    agent.agent-reconciler  agent/hypershift.go:192 enter runHypershiftInstall                                                                                                                                                              
2022-05-27T17:24:23.715Z        ERROR   agent.agent-reconciler  agent/hypershift.go:206 failed to get bucket secret(local-cluster/hypershift-operator-oidc-provider-s3-credentials) from hub, will retry.       {"error": "secrets \"hypershift-operator-oidc-provider-s3-creden
tials\" not found"}                                                               [...]
2022-05-27T17:24:23.715Z        INFO    agent.agent-reconciler  agent/hypershift.go:207 exit runHypershiftInstall
2022-05-27T17:24:23.715Z        ERROR   agent.agent-reconciler  agent/hypershift.go:103 failed to run hypershift cmd    {"error": "secrets \"hypershift-operator-oidc-provider-s3-credentials\" not found"}

Comment 1 Chad Crum 2022-06-01 18:48:41 UTC
Work around to get hypershift operator deployed is to create a dummy s3 secret:

# Create file with dummy aws s3 credentials called s3-dummy-credentials and content below
[default]
aws_access_key_id     = ABCDEFGHIJKLMNOPQRST
aws_secret_access_key = ABCDEFGHIJKLMNOPQRSTUVWXYZabcd1234567890

# Create secret using previously created file (namespace should match the namespace used in add-on operator step)
oc create secret generic hypershift-operator-oidc-provider-s3-credentials --from-file=credentials=s3-dummy-credentials --from-literal=bucket=bucket --from-literal=region=us-east-1 -n local-cluster

Comment 3 Nelson Jean 2022-07-04 19:45:09 UTC
Hi @ccrum , can you please verify and close if no issues. Thanks!

Comment 4 phwu 2022-07-05 18:59:12 UTC
Same as this issue reported in 2.5.1: https://github.com/stolostron/backlog/issues/23868

Fix will be in 2.5.2

Comment 5 bot-tracker-sync 2022-07-05 20:26:05 UTC
G2Bsync 1175397613 comment 
 nelsonjean Tue, 05 Jul 2022 18:57:50 UTC 
 G2Bsync 

This issue was not fixed in MCE 2.0.1 / ACM 2.5.1. Moving it to 2.5.2.

Comment 7 Chad Crum 2022-07-13 21:40:11 UTC
Confirmed fixed with latest mce ds snapshot.

Comment 12 errata-xmlrpc 2022-09-13 20:06:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Critical: Red Hat Advanced Cluster Management 2.5.2 security fixes and bug fixes), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6507


Note You need to log in before you can comment on or make changes to this bug.