1839268 – GCP destroy is leaking cluster created service accounts and project iam bindings

Bug 1839268 - GCP destroy is leaking cluster created service accounts and project iam bindings [NEEDINFO]

Summary: GCP destroy is leaking cluster created service accounts and project iam bindings

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Installer
Sub Component:
Version:	4.5
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.5.0
Assignee:	Abhinav Dahiya
QA Contact:	Yang Yang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-05-22 22:47 UTC by Abhinav Dahiya
Modified:	2020-07-13 17:41 UTC (History)
CC List:	1 user (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-07-13 17:41:26 UTC
Target Upstream Version:
Flags:	yanyang: needinfo? (adahiya)

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift installer pull 3646	0	None	closed	Bug 1839268: pkg/destroy/gcp: ensure cluster service accounts and policy bindings are removed	2021-01-17 09:04:18 UTC
Red Hat Product Errata	RHBA-2020:2409	0	None	None	None	2020-07-13 17:41:39 UTC

Description Abhinav Dahiya 2020-05-22 22:47:59 UTC

GCP destroy leaks the service accounts created by the cluster. It also does not remove the bindings for these service accounts from the Project IAM policy.

see the destroy logs
```
time="2020-05-20T18:18:56Z" level=debug msg="Listing service accounts"
time="2020-05-20T18:19:02Z" level=debug msg="Found service account: projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:06Z" level=debug msg="Found service account: projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Deleting service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=info msg="Deleted service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Deleting service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=info msg="Deleted service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Fetching project IAM policy"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/compute.instanceAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/compute.networkAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/compute.securityAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/compute.viewer"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/storage.admin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/storage.admin"
time="2020-05-20T18:19:08Z" level=debug msg="Setting project IAM policy"
time="2020-05-20T18:19:09Z" level=debug msg="Policy bindings: 1 items pending"
```

it only deleted the installer created, and nothing else.

When it should have actually looked like
```
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Fetching project IAM policy
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.instanceAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.instanceAdmin.v1
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.loadBalancerAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.networkAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.securityAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.viewer
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bqr6x@openshift-dev-installer.iam.gserviceaccount.com from role roles/dns.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com from role roles/storage.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/storage.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com from role roles/storage.admin
DEBUG Setting project IAM policy
DEBUG Policy bindings: 1 items pending
D
```

there are total 5 service accounts, 2 created by installer 3 created by the cluster.

Comment 3 Yang Yang 2020-05-25 12:21:57 UTC

Hi Abhinav,

I'm trying to reproduce it with 4.5.0-0.nightly-2020-05-22-062554, but cluster destroy does not leak service accounts. What scenario did you find the issue in?

level=debug msg="Listing service accounts"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Fetching project IAM policy"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.instanceAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com from role roles/compute.instanceAdmin.v1"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com from role roles/compute.loadBalancerAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.networkAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.securityAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com from role roles/compute.viewer"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-tc59p@openshift-qe.iam.gserviceaccount.com from role roles/dns.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="Setting project IAM policy"
level=debug msg="Policy bindings: 1 items pending"

Trying to verify with 4.5.0-0.nightly-2020-05-24-223848, cluster destroy deletes all service accounts.

level=debug msg="Listing service accounts"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Fetching project IAM policy"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.instanceAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com from role roles/compute.instanceAdmin.v1"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com from role roles/compute.loadBalancerAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.networkAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.securityAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com from role roles/compute.viewer"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-2lpxp@openshift-qe.iam.gserviceaccount.com from role roles/dns.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="Setting project IAM policy"
level=debug msg="Policy bindings: 1 items pending"
level=debug msg="Fetching project IAM policy"
level=info msg="Deleted IAM project role bindings"

Comment 4 Yang Yang 2020-06-15 08:18:40 UTC

Moving it to verified state as I do not experience the issue recently.

Comment 5 errata-xmlrpc 2020-07-13 17:41:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409

Note You need to log in before you can comment on or make changes to this bug.