Bug 1839268 - GCP destroy is leaking cluster created service accounts and project iam bindings [NEEDINFO]
Summary: GCP destroy is leaking cluster created service accounts and project iam bindings
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.0
Assignee: Abhinav Dahiya
QA Contact: Yang Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-22 22:47 UTC by Abhinav Dahiya
Modified: 2020-07-13 17:41 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:41:26 UTC
Target Upstream Version:
yanyang: needinfo? (adahiya)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 3646 0 None closed Bug 1839268: pkg/destroy/gcp: ensure cluster service accounts and policy bindings are removed 2021-01-17 09:04:18 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:41:39 UTC

Description Abhinav Dahiya 2020-05-22 22:47:59 UTC
GCP destroy leaks the service accounts created by the cluster. It also does not remove the bindings for these service accounts from the Project IAM policy.

see the destroy logs
```
time="2020-05-20T18:18:56Z" level=debug msg="Listing service accounts"
time="2020-05-20T18:19:02Z" level=debug msg="Found service account: projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:06Z" level=debug msg="Found service account: projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Deleting service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=info msg="Deleted service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Deleting service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=info msg="Deleted service account projects/openshift-gce-devel-ci/serviceAccounts/ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com"
time="2020-05-20T18:19:07Z" level=debug msg="Fetching project IAM policy"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/compute.instanceAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/compute.networkAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/compute.securityAdmin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/compute.viewer"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-m@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/storage.admin"
time="2020-05-20T18:19:08Z" level=debug msg="IAM: removing serviceAccount:ci-op-hmqstbi8-15937-9zz8h-w@openshift-gce-devel-ci.iam.gserviceaccount.com from role roles/storage.admin"
time="2020-05-20T18:19:08Z" level=debug msg="Setting project IAM policy"
time="2020-05-20T18:19:09Z" level=debug msg="Policy bindings: 1 items pending"
```

it only deleted the installer created, and nothing else.

When it should have actually looked like
```
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Found service account: projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Deleting service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x@openshift-dev-installer.iam.gserviceaccount.com
INFO Deleted service account projects/openshift-dev-installer/serviceAccounts/adahiya-2-zp-openshift-i-bqr6x@openshift-dev-installer.iam.gserviceaccount.com
DEBUG Fetching project IAM policy
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.instanceAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.instanceAdmin.v1
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.loadBalancerAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.networkAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.securityAdmin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com from role roles/compute.viewer
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bqr6x@openshift-dev-installer.iam.gserviceaccount.com from role roles/dns.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-m-m7hjm@openshift-dev-installer.iam.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/iam.serviceAccountUser
DEBUG IAM: removing serviceAccount:adahiya-2-zp-openshift-i-bj84x@openshift-dev-installer.iam.gserviceaccount.com from role roles/storage.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-m@openshift-dev-installer.iam.gserviceaccount.com from role roles/storage.admin
DEBUG IAM: removing serviceAccount:adahiya-2-zpjc6-w@openshift-dev-installer.iam.gserviceaccount.com from role roles/storage.admin
DEBUG Setting project IAM policy
DEBUG Policy bindings: 1 items pending
D
```

there are total 5 service accounts, 2 created by installer 3 created by the cluster.

Comment 3 Yang Yang 2020-05-25 12:21:57 UTC
Hi Abhinav,

I'm trying to reproduce it with 4.5.0-0.nightly-2020-05-22-062554, but cluster destroy does not leak service accounts. What scenario did you find the issue in?

level=debug msg="Listing service accounts"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangya-bk2dh-openshift-i-tc59p@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Fetching project IAM policy"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.instanceAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com from role roles/compute.instanceAdmin.v1"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com from role roles/compute.loadBalancerAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.networkAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.securityAdmin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com from role roles/compute.viewer"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-tc59p@openshift-qe.iam.gserviceaccount.com from role roles/dns.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-m-tpq7n@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-m@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-openshift-i-q4p9v@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangya-bk2dh-w@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="Setting project IAM policy"
level=debug msg="Policy bindings: 1 items pending"

Trying to verify with 4.5.0-0.nightly-2020-05-24-223848, cluster destroy deletes all service accounts.

level=debug msg="Listing service accounts"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Found service account: projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-i-2lpxp@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Deleting service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com"
level=info msg="Deleted service account projects/openshift-qe/serviceAccounts/yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com"
level=debug msg="Fetching project IAM policy"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.instanceAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com from role roles/compute.instanceAdmin.v1"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com from role roles/compute.loadBalancerAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.networkAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/compute.securityAdmin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com from role roles/compute.viewer"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-2lpxp@openshift-qe.iam.gserviceaccount.com from role roles/dns.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-m-tlhrf@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/iam.serviceAccountUser"
level=debug msg="IAM: removing serviceAccount:yangyang1837-openshift-i-vvq8b@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-m@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="IAM: removing serviceAccount:yangyang1837642-dxzpm-w@openshift-qe.iam.gserviceaccount.com from role roles/storage.admin"
level=debug msg="Setting project IAM policy"
level=debug msg="Policy bindings: 1 items pending"
level=debug msg="Fetching project IAM policy"
level=info msg="Deleted IAM project role bindings"

Comment 4 Yang Yang 2020-06-15 08:18:40 UTC
Moving it to verified state as I do not experience the issue recently.

Comment 5 errata-xmlrpc 2020-07-13 17:41:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.