Bug 1839268

Summary:	GCP destroy is leaking cluster created service accounts and project iam bindings
Product:	OpenShift Container Platform	Reporter:	Abhinav Dahiya <adahiya>
Component:	Installer	Assignee:	Abhinav Dahiya <adahiya>
Installer sub component:	openshift-installer	QA Contact:	Yang Yang <yanyang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	gpei
Version:	4.5
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-13 17:41:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Abhinav Dahiya 2020-05-22 22:47:59 UTC

GCP destroy leaks the service accounts created by the cluster. It also does not remove the bindings for these service accounts from the Project IAM policy.

see the destroy logs
```
time="2020-05-20T18:18:56Z" level=debug msg="Listing service accounts"
time="2020-05-20T18:19:02Z" level=debug msg="Found service account: projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w.gserviceaccount.com"
time="2020-05-20T18:19:06Z" level=debug msg="Found service account: projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Deleting service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=info msg="Deleted service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Deleting service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=info msg="Deleted service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Fetching project IAM policy"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m.gserviceaccount.com from role roles/compute.instanceAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m.gserviceaccount.com from role roles/compute.networkAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m.gserviceaccount.com from role roles/compute.securityAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-w.gserviceaccount.com from role roles/compute.viewer"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m.gserviceaccount.com from role roles/iam.serviceAccountUser"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m.gserviceaccount.com from role roles/storage.admin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-w.gserviceaccount.com from role roles/storage.admin"
time="2020-05-20T18:19:08Z" level=debug msg="Setting project IAM policy"
time="2020-05-20T18:19:09Z" level=debug msg="Policy bindings: 1 items pending"
```

it only deleted the installer created, and nothing else.

When it should have actually looked like
```
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x.gserviceaccount.com
DEBUG Fetching project IAM policy
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m.gserviceaccount.com from role roles/compute.instanceAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm.gserviceaccount.com from role roles/compute.instanceAdmin.v1
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm.gserviceaccount.com from role roles/compute.loadBalancerAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m.gserviceaccount.com from role roles/compute.networkAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m.gserviceaccount.com from role roles/compute.securityAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-w.gserviceaccount.com from role roles/compute.viewer
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bqr6x.gserviceaccount.com from role roles/dns.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bj84x.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bj84x.gserviceaccount.com from role roles/storage.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m.gserviceaccount.com from role roles/storage.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-w.gserviceaccount.com from role roles/storage.admin
DEBUG Setting project IAM policy
DEBUG Policy bindings: 1 items pending
D
```

there are total 5 service accounts, 2 created by installer 3 created by the cluster.

Comment 3 Yang Yang 2020-05-25 12:21:57 UTC

Hi Abhinav,

I'm trying to reproduce it with 4.5.0-0.nightly-2020-05-22-062554, but cluster destroy does not leak service accounts. What scenario did you find the issue in?

level=debug msg="Listing service accounts"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-m.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-w.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-m.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-m.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-w.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-w.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p.gserviceaccount.com"
level=debug msg="Fetching project IAM policy"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m.gserviceaccount.com from role roles/compute.instanceAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n.gserviceaccount.com from role roles/compute.instanceAdmin.v1"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n.gserviceaccount.com from role roles/compute.loadBalancerAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m.gserviceaccount.com from role roles/compute.networkAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m.gserviceaccount.com from role roles/compute.securityAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-w.gserviceaccount.com from role roles/compute.viewer"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-tc59p.gserviceaccount.com from role roles/dns.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-q4p9v.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-q4p9v.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-w.gserviceaccount.com from role roles/storage.admin"
level=debug msg="Setting project IAM policy"
level=debug msg="Policy bindings: 1 items pending"

Trying to verify with 4.5.0-0.nightly-2020-05-24-223848, cluster destroy deletes all service accounts.

level=debug msg="Listing service accounts"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf.gserviceaccount.com"
level=debug msg="Fetching project IAM policy"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m.gserviceaccount.com from role roles/compute.instanceAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf.gserviceaccount.com from role roles/compute.instanceAdmin.v1"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf.gserviceaccount.com from role roles/compute.loadBalancerAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m.gserviceaccount.com from role roles/compute.networkAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m.gserviceaccount.com from role roles/compute.securityAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-w.gserviceaccount.com from role roles/compute.viewer"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-2lpxp.gserviceaccount.com from role roles/dns.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-vvq8b.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-vvq8b.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-w.gserviceaccount.com from role roles/storage.admin"
level=debug msg="Setting project IAM policy"
level=debug msg="Policy bindings: 1 items pending"
level=debug msg="Fetching project IAM policy"
level=info msg="Deleted IAM project role bindings"

Comment 4 Yang Yang 2020-06-15 08:18:40 UTC

Moving it to verified state as I do not experience the issue recently.

Comment 5 errata-xmlrpc 2020-07-13 17:41:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409

Comment 6 Red Hat Bugzilla 2023-09-14 06:01:05 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days