Bug 1926547 - OpenShift installer not reporting IAM permission issue when removing the Shared Subnet Tag
Summary: OpenShift installer not reporting IAM permission issue when removing the Shar...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Russell Teague
QA Contact: Yunfei Jiang
Depends On:
TreeView+ depends on / blocked
Reported: 2021-02-09 02:41 UTC by aygarg
Modified: 2021-07-27 22:43 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Results from untagging resources were not being checked for errors. Consequence: Shared resources were not being untagged but the logging indicated they were. Fix: Checking the results for untagging errors and logging the error. Result: Logs correctly indicate the status of untagging shared resources.
Clone Of:
Last Closed: 2021-07-27 22:42:29 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4913 0 None open Bug 1926547: pkg/destroy/aws: Log errors untagging shared resources 2021-05-07 15:54:44 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:43:31 UTC

Description aygarg 2021-02-09 02:41:50 UTC

$ ./openshift-install version
./openshift-install 4.6.15
built from commit 26aab99447a65a1a3d46342318486bd1dd11b1e2
release image quay.io/openshift-release-dev/ocp-release@sha256:b70f550e3fa94af2f7d60a3437ec0275194db36f2dc49991da2336fe21e2824c


Please specify:

What happened?
OpenShift installer adds a shared tag such as "kubernetes.io/cluster/aygarg-lkcdg: shared" to the existing private subnets during private cluster installation. When the cluster is destroyed, the installer doesn't delete that tag from the subnets. This is happening because the "ec2:DeleteTags" permission is not specified in the documentation that is required by the user and for which I have already raised another Bugzilla BZ#1926543.

level=info msg="Removed tag kubernetes.io/cluster/awsad-w9fpb: shared" arn="arn:aws:ec2:us-east-1:037xxxxxxxxx:subnet/subnet-04xxxxxx"
level=info msg="Removed tag kubernetes.io/cluster/awsad-w9fpb: shared" arn="arn:aws:ec2:us-east-1:037xxxxxxxxx:subnet/subnet-3bxxxxxx"
--> The above logs are shown by the installer when cluster is destroyed even after not having the "ec2:DeleteTags" IAM permission. So this is a false report as the installer should complain about the missing permission instead of showing the above logs.

What did you expect to happen?
OpenShift installer must report about the missing "ec2:DeleteTags" IAM permission when deleting the tag during cluster destroy.

How to reproduce it (as minimally and precisely as possible)?
Deploy a private cluster over AWS using existing resources (VPC, Subnets, etc) by adding only those permissions which are mentioned in the below documentation as "ec2:DeleteTags" is missing from it. Then destroy the cluster and check the logs will contain the above specified false information while the tag will still be there on the subnets.
--> https://docs.openshift.com/container-platform/4.6/installing/installing_aws/installing-aws-account.html#installation-aws-permissions_installing-aws-account

Comment 1 Matthew Staebler 2021-02-09 03:51:08 UTC
The destroyer should be looking at the response from the call to UntagResources to see which resources could not be untagged.

In addition to having the tag:UnTagResource permission required to un-tag using the tagging API, we also need permission to delete tags in the service of the resource.

From https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_UntagResources.html, 
> To remove tags from a resource, you need the necessary permissions for the service that the resource belongs to as well as permissions for removing tags.

Comment 2 Matthew Staebler 2021-02-15 18:27:18 UTC
We will take a look at this during this sprint.

Comment 7 Yunfei Jiang 2021-05-24 10:41:56 UTC
verified. PASS.
OCP version: 4.8.0-0.nightly-2021-05-21-233425

INFO Removed tag kubernetes.io/cluster/yunjiang-bz547-p2n2g: shared  arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-0dd122200922d13e3
INFO Removed tag kubernetes.io/cluster/yunjiang-bz547-p2n2g: shared  arn=arn:aws:ec2:us-east-2:301721915996:subnet/subnet-05619e9c7817ba6f9

Comment 10 errata-xmlrpc 2021-07-27 22:42:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.