Bug 2028695 - destroy cluster does not prune bootstrap instance profile
Summary: destroy cluster does not prune bootstrap instance profile
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Aditya Narayanaswamy
QA Contact: Yunfei Jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-02 23:36 UTC by dlbewley
Modified: 2022-03-10 16:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The installer destroy code was not checking the bootstrap instance profiles and was only checking the control-plane and workers for destroy. Adding bootstrap to the list of instance profiles to check for destroy cluster.
Clone Of:
Environment:
Last Closed: 2022-03-10 16:31:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
destroy log (19.48 KB, text/plain)
2021-12-02 23:36 UTC, dlbewley
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5506 0 None open Bug 2028695: Delete bootstrap profiles during cluster destroy 2021-12-22 16:47:22 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:31:53 UTC

Description dlbewley 2021-12-02 23:36:04 UTC
Created attachment 1844531 [details]
destroy log

Version:

$ openshift-install version
openshift-install 4.9.8
built from commit 1c538b8949f3a0e5b993e1ae33b9cd799806fa93
release image quay.io/openshift-release-dev/ocp-release@sha256:c91c0faf7ae3c480724a935b3dab7e5f49aae19d195b12f3a4ae38f8440ea96b
release architecture amd64

Platform:

* AWS

Please specify:

* IPI

What happened?

* Created ec2 instance with an instance profile using custom role named 'openshift-provision-role'
* Assign 'openshift-provision-role' to compute[].platform.aws.iamRole and controlPlane.platform.aws.iamRole in install config
* Run installer from this ec2 instance
* Perform failed cluster install (fwiw failed at create bootstrap ignition s3 bucket so no bootstrap instance created)
* Destroy Cluster
* Observe orphaned <infraid>-bootstrap-profile instance profile resource attached to custom role 'openshift-provision-role'
 
What did you expect to happen?

* All created artifacts removed by cluster destroy operation. 

How to reproduce it (as minimally and precisely as possible)?

* Not sure if this is a side effect of a failed install or a consistent bug.

Anything else we need to know?


 [ssm-user@ip-100-127-128-160 poc1221]$ openshift-install destroy cluster --dir $CLUSTER_DIR
 INFO Credentials loaded from the AWS config using "EC2RoleProvider" provider
 INFO Disassociated                                 id=i-0dc17ba5892c862b7 name=poc1221-9q8l5-master-profile role=openshift-provision-role
 INFO Deleted                                       InstanceProfileName=poc1221-9q8l5-master-profile arn=arn:aws:iam::000000000000:instance-profile/poc1221-9q8l5-master-profile id=i-0dc17ba5892c862b7
 kINFO Deleted                                       id=eni-0d0ab112ae472560c
 INFO Disassociated                                 id=poc1221-9q8l5-worker-profile name=poc1221-9q8l5-worker-profile role=openshift-provision-role
 INFO Deleted                                       InstanceProfileName=poc1221-9q8l5-worker-profile arn=arn:aws:iam::000000000000:instance-profile/poc1221-9q8l5-worker-profile id=poc1221-9q8l5-worker-profile
 INFO Deleted                                       id=eni-0d9fa403ec1c107f7
 INFO Deleted                                       id=eni-01cf65e61a3fdfa96
 INFO Deleted                                       id=sg-0f67d7ba8dc549f7c
 INFO Deleted                                       id=net/poc1221-9q8l5-int/bd74514eb822205e
 INFO Deleted                                       id=sg-0836017850fa54ff8
 INFO Deleted                                       id=poc1221-9q8l5-aint/8a7f6d1459d92931
 INFO Deleted                                       id=sg-0d66b2c1e18789141
 INFO Deleted                                       id=poc1221-9q8l5-sint/7822ffcb371275ff
 INFO Removed tag kubernetes.io/cluster/poc1221-9q8l5: shared  arn=arn:aws:ec2:us-west-2:000000000000:subnet/subnet-0685cf0f9bb331346
 INFO Removed tag kubernetes.io/cluster/poc1221-9q8l5: shared  arn=arn:aws:ec2:us-west-2:000000000000:subnet/subnet-0f9dcb3f0c6a9de4a
 INFO Removed tag kubernetes.io/cluster/poc1221-9q8l5: shared  arn=arn:aws:ec2:us-west-2:000000000000:subnet/subnet-0414749f0554795a9
 INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z037588923GH9C9V5G1I5 id=Z037588923GH9C9V5G1I5 record set=A api-int.poc1221.ocp.corp.example.com.
 INFO Deleted                                       arn=arn:aws:route53:::hostedzone/Z037588923GH9C9V5G1I5 id=Z037588923GH9C9V5G1I5 record set=A api.poc1221.ocp.corp.example.com.
 INFO Cleaned record sets from hosted zone          arn=arn:aws:route53:::hostedzone/Z037588923GH9C9V5G1I5 id=Z037588923GH9C9V5G1I5
 INFO Removed tag kubernetes.io/cluster/poc1221-9q8l5: shared  arn=arn:aws:route53:::hostedzone/Z037588923GH9C9V5G1I5
 INFO Time elapsed: 7m11s

The manual cleanup required for this is:

 aws iam remove-role-from-instance-profile \
        --instance-profile-name poc1221-9q8l5-bootstrap-profile \
        --role-name openshift-provision-role
 aws iam delete-instance-profile \
        --instance-profile-name poc1221-9q8l5-bootstrap-profile

Comment 2 Matthew Staebler 2021-12-03 13:46:17 UTC
The bootstrap profile should be added to the list of instance profile that the destroyer is searching for to delete at https://github.com/openshift/installer/blob/4172316e7a3032f9b79115af2a9ba8b94889b86d/pkg/destroy/aws/aws.go#L455.

Comment 6 Yunfei Jiang 2021-12-27 07:30:22 UTC
verified. PASS.
OCP version: 4.10.0-0.nightly-2021-12-23-153012


install-config:
<--snip-->
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    aws:
      iamRole: yunjiang-iam1-034232622-pre-cr-role-master
  replicas: 3
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    aws:
      iamRole: yunjiang-iam1-034232622-pre-cr-role-worker
  replicas: 3
metadata:
  name: yunjiang-iam1
<--snip-->

destroy logs:
time="2021-12-27T02:18:31-05:00" level=debug msg="OpenShift Installer 4.10.0-0.nightly-2021-12-23-153012"
<--snip-->
time="2021-12-27T02:19:23-05:00" level=info msg=Disassociated id=i-08aa32edfc428cc91 name=yunjiang-iam1-c9j5n-master-profile role=yunjiang-iam1-034232622-pre-cr-role-master
time="2021-12-27T02:19:23-05:00" level=info msg=Deleted InstanceProfileName=yunjiang-iam1-c9j5n-master-profile arn="arn:aws:iam::301721915996:instance-profile/yunjiang-iam1-c9j5n-master-profile" id=i-08aa32edfc428cc91
time="2021-12-27T02:19:24-05:00" level=debug msg=Terminating id=i-08aa32edfc428cc91
time="2021-12-27T02:19:24-05:00" level=info msg=Disassociated id=i-0663def5371791363 name=yunjiang-iam1-c9j5n-bootstrap-profile role=yunjiang-iam1-034232622-pre-cr-role-master
time="2021-12-27T02:19:24-05:00" level=info msg=Deleted InstanceProfileName=yunjiang-iam1-c9j5n-bootstrap-profile arn="arn:aws:iam::301721915996:instance-profile/yunjiang-iam1-c9j5n-bootstrap-profile" id=i-0663def5371791363
time="2021-12-27T02:19:24-05:00" level=debug msg=Terminating id=i-0663def5371791363
time="2021-12-27T02:19:24-05:00" level=debug msg=Terminating id=i-0cf5d3ec5d5055f0c
time="2021-12-27T02:19:25-05:00" level=debug msg=Terminating id=i-0120b1948b92d793b
<--snip-->
time="2021-12-27T02:21:32-05:00" level=info msg="Time elapsed: 3m2s"

Comment 9 errata-xmlrpc 2022-03-10 16:31:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.