Bug 1948359

Summary: [aws] shared tag was not removed from user provided IAM role
Product: OpenShift Container Platform Reporter: Yunfei Jiang <yunjiang>
Component: InstallerAssignee: Matthew Staebler <mstaeble>
Installer sub component: openshift-installer QA Contact: Yunfei Jiang <yunjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: gpei, mstaeble, rteague
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Bug in new feature
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 22:58:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1945907    

Description Yunfei Jiang 2021-04-12 03:09:13 UTC
Description of problem:

If user provide an existing IAM role in install-config.yaml, the installer will add a new shared tag to this role.
e.g. 
  {
    "Key": "kubernetes.io/cluster/yunjiang-iamplat-8gtkm",
    "Value": "shared"
  }

after destroy cluster, the shared tag was not removed by installer

How reproducible:
Always.

Steps to Reproduce:
1. Create an install-config.yaml, and provide an existing IAM role,
e.g.
metadata:
  name: yunjiang-iamtpl
platform:
  aws:
    region: us-east-2
    defaultMachinePlatform:
      iamRole: existing_iam_role
2. create cluster, a shared tag will be added to role existing_iam_role
3. destroy cluster

Actual results:
check existing_iam_role tags, the shared was not removed

Expected results:
the shared tag should be removed in the destroy process.

Additional info:

Comment 2 Yunfei Jiang 2021-04-25 05:33:54 UTC
verified. FAILED.

the `shared` tag was not removed from IAM role.

destroy log:
time="2021-04-25T00:09:13-04:00" level=debug msg="search for IAM roles"
time="2021-04-25T00:09:14-04:00" level=debug msg="search for IAM users"
time="2021-04-25T00:09:15-04:00" level=debug msg="search for IAM instance profiles"
time="2021-04-25T00:09:15-04:00" level=debug msg="Search for and remove tags in us-east-2 matching kubernetes.io/cluster/yunjiang-359b-x78hf: shared"
time="2021-04-25T00:09:15-04:00" level=debug msg="No matches in us-east-2 for kubernetes.io/cluster/yunjiang-359b-x78hf: shared, removing client"
time="2021-04-25T00:09:15-04:00" level=debug msg="Search for and remove tags in us-east-1 matching kubernetes.io/cluster/yunjiang-359b-x78hf: shared"
time="2021-04-25T00:09:15-04:00" level=debug msg="No matches in us-east-1 for kubernetes.io/cluster/yunjiang-359b-x78hf: shared, removing client"
time="2021-04-25T00:09:15-04:00" level=info msg="Time elapsed: 8m52s"

Comment 4 Gaoyun Pei 2021-04-30 02:42:47 UTC
Verified this bug with nightly 4.8.0-0.nightly-2021-04-29-151418.

1. Set BYO AWS IAM role to master and worker in install-config.yaml as following, start the cluster installation.
```
  controlPlane:
    architecture: amd64
    hyperthreading: Enabled
    name: master
    platform:
      aws:
        iamRole: gpei-iam-4930746-pre-cr-role-master
    replicas: 3
  compute:
  - architecture: amd64
    hyperthreading: Enabled
    name: worker
    platform:
      aws:
        iamRole: gpei-iam-4930746-pre-cr-role-worker
    replicas: 3
```


2. The installation finished without error, check the two IAM role, "shared" tag was added.

# aws iam get-role --role-name gpei-iam-4930746-pre-cr-role-worker | jq -r .Role.Tags
[
  {
    "Key": "kubernetes.io/cluster/gpei-iam-lrcph",
    "Value": "shared"
  }
]

# aws iam get-role --role-name gpei-iam-4930746-pre-cr-role-master | jq -r .Role.Tags
[
  {
    "Key": "kubernetes.io/cluster/gpei-iam-lrcph",
    "Value": "shared"
  }
]


3. Destroy the cluster, shared tag was removed from the provided IAM role.

10:25:21  level=debug msg=search for IAM roles
10:25:21  level=debug msg=search for IAM users
10:25:24  level=debug msg=search for IAM instance profiles
10:25:24  level=debug msg=Search for and remove tags in us-east-2 matching kubernetes.io/cluster/gpei-iam-lrcph: shared
10:25:24  level=debug msg=Nothing to clean for shared ec2 resource arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-0c0176e6c346d3c30
10:25:24  level=debug msg=Nothing to clean for shared ec2 resource arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-0e68b1d34f95506a8
10:25:24  level=info msg=Removed tag kubernetes.io/cluster/gpei-iam-lrcph: shared arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-0c0176e6c346d3c30
10:25:24  level=info msg=Removed tag kubernetes.io/cluster/gpei-iam-lrcph: shared arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-0e68b1d34f95506a8
10:25:24  level=debug msg=Search for and remove tags in us-east-1 matching kubernetes.io/cluster/gpei-iam-lrcph: shared
10:25:24  level=debug msg=No matches in us-east-1 for kubernetes.io/cluster/gpei-iam-lrcph: shared, removing client
10:25:24  level=debug msg=Search for and remove tags in us-east-2 matching kubernetes.io/cluster/gpei-iam-lrcph: shared
10:25:24  level=debug msg=No matches in us-east-2 for kubernetes.io/cluster/gpei-iam-lrcph: shared, removing client
10:25:24  level=debug msg=Search for and remove shared tags for IAM roles matching kubernetes.io/cluster/gpei-iam-lrcph: shared
10:25:30  level=debug msg=Removing the shared tag from the "gpei-iam-4930746-pre-cr-role-master" IAM role
10:25:30  level=debug msg=Removing the shared tag from the "gpei-iam-4930746-pre-cr-role-worker" IAM role

Comment 7 errata-xmlrpc 2021-07-27 22:58:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438