Bug 1953077
| Summary: | Handling GCP's: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | W. Trevor King <wking> |
| Component: | Cloud Credential Operator | Assignee: | Joel Diaz <jdiaz> |
| Status: | CLOSED ERRATA | QA Contact: | wang lin <lwan> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.8 | CC: | arane, gshereme, jdiaz, lwan |
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: Permissions checking when cloud-credential-operator is in Passthrough mode on GCP could cause errors when reconciling certain CredentialsRequest CRs.
Consequence: Otherwise valid CredentialsRequest CRs would end up not being reconciled.
Fix: Change logic to check against a list of permissions that are valid to TestIamPermissions() against at the GCP Project resource level.
Result: CredentialsRequests which include permissions that apply outside of the GCP Project resource level will no longer error during reconcilation.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 23:03:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1958983 | ||
PR to fix this is open and under review https://github.com/openshift/cloud-credential-operator/pull/330 Verified on 4.8.0-0.nightly-2021-05-09-105430 1. Create install-config ./openshift-install create install-config 2. Set cco to Passthrough mode echo "credentialsMode: Passthrough" >> install-config.yaml 3. Create cluster ./openshift-install create cluster --log-level debug Install nightly build 4.8.0-0.nightly-2021-05-06-190249 with above steps, after installation, check cco logs, will show the below error: time="2021-05-10T03:16:03Z" level=error msg="error syncing credentials: error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:16:03Z" level=error msg="unexpected error while syncing credentialsrequest: error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds Install nightly build 4.8.0-0.nightly-2021-05-09-105430 with this fix, after installation, check cco logs,cco will ignore project scoped permission check, and won't hit the error. time="2021-05-10T03:18:49Z" level=info msg="clusteroperator status updated" controller=status time="2021-05-10T03:19:08Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-gcp-pd-csi-driver-operator time="2021-05-10T03:19:08Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-gcp-pd-csi-driver-operator time="2021-05-10T03:19:08Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-gcp-pd-csi-driver-operator time="2021-05-10T03:19:08Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/openshift-gcp-pd-csi-driver-operator secret=openshift-cluster-csi-drivers/gcp-pd-cloud-credentials time="2021-05-10T03:19:10Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp time="2021-05-10T03:19:10Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp time="2021-05-10T03:19:10Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp time="2021-05-10T03:19:11Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-gcp secret=openshift-machine-api/gcp-cloud-credentials time="2021-05-10T03:19:11Z" level=info msg="reconciling clusteroperator status" time="2021-05-10T03:19:11Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=info msg="clusteroperator status updated" controller=status time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.accessLevels.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.accessPolicies.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.accessPolicies.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.accessZones.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.gcpUserAccessBindings.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.policies.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.policies.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.servicePerimeters.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of assuredworkloads.operations.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of assuredworkloads.workload.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.accounts.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.accounts.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.budgets.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.credits.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.resourceAssociations.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.subscriptions.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudprivatecatalogproducer.associations.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudprivatecatalogproducer.catalogs.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudprivatecatalogproducer.catalogs.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudsupport.accounts.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudsupport.accounts.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of consumerprocurement.accounts.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of consumerprocurement.orders.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of earlyaccesscenter.campaigns.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of earlyaccesscenter.customerAllowlists.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.folders.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.folders.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.organizations.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.tagKeys.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.tagKeys.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.tagValues.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.tagValues.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of securitycenter.notificationconfig.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of servicemanagement.consumerSettings.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of servicemanagement.consumerSettings.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:12Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-05-10T03:19:12Z" level=info msg="reconciling clusteroperator status" time="2021-05-10T03:19:12Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress-gcp time="2021-05-10T03:19:12Z" level=info msg="clusteroperator status updated" controller=status time="2021-05-10T03:19:13Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-ingress-gcp time="2021-05-10T03:19:13Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress-gcp secret=openshift-ingress-operator/cloud-credentials time="2021-05-10T03:19:13Z" level=info msg="reconciling clusteroperator status" Hi Joel, I found the 4.7 has the same issue, do we need to backport this fix to 4.7? Yes, I will start a backport to 4.7. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Seen in GCP CI [1], the job fails because: : operator conditions cloud-credential 0s Operator progressing (Reconciling): 4 of 5 credentials requests provisioned, 0 reporting errors. Checking all the operator's conditions: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp/1385311485223243776/artifacts/e2e-gcp/gather-extra/artifacts/clusteroperators.json | jq -r '.items[] | select(.metadata.name == "cloud-credential") | .status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + (.reason // "-") + ": " + (.message // "-")' 2021-04-22T19:25:58Z Available=True -: - 2021-04-22T19:25:58Z Degraded=False -: - 2021-04-22T19:34:25Z Progressing=True Reconciling: 4 of 5 credentials requests provisioned, 0 reporting errors. 2021-04-22T19:25:58Z Upgradeable=True -: - The cloud-credential-operator-gcp-ro-creds CredentialsRequest contains no status: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp/1385311485223243776/artifacts/e2e-gcp/gather-extra/artifacts/credentialsrequests.json | jq -r '.items[] | select(.metadata.name == "cloud-credential-operator-gcp-ro-creds")' { "apiVersion": "cloudcredential.openshift.io/v1", "kind": "CredentialsRequest", "metadata": { "annotations": { "exclude.release.openshift.io/internal-openshift-hosted": "true", "include.release.openshift.io/self-managed-high-availability": "true" }, "creationTimestamp": "2021-04-22T19:25:59Z", "finalizers": [ "cloudcredential.openshift.io/deprovision" ], "generation": 1, "name": "cloud-credential-operator-gcp-ro-creds", "namespace": "openshift-cloud-credential-operator", "resourceVersion": "2399", "uid": "5c8d5d51-80e1-4de8-96c6-b2e8cc71a173" }, "spec": { "providerSpec": { "apiVersion": "cloudcredential.openshift.io/v1", "kind": "GCPProviderSpec", "predefinedRoles": [ "roles/iam.securityReviewer", "roles/iam.roleViewer" ], "skipServiceCheck": true }, "secretRef": { "name": "cloud-credential-operator-gcp-ro-creds", "namespace": "openshift-cloud-credential-operator" } } } And from the operator's log [2]: time="2021-04-22T19:51:42Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-04-22T19:51:42Z" level=warning msg="read-only creds not found, using root creds client" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-04-22T19:51:44Z" level=error msg="error syncing credentials: error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2021-04-22T19:51:44Z" level=error msg="unexpected error while syncing credentialsrequest: error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds Talking with Joel, the root cause appears to be that there are some permissions which cannot be checked at the project level. Possible pivots: a. When we do try to check permissions and fail in that check, start setting status.conditions[] mentioning the failure on the request. This will allow admins to understand the problem, or at least get some good, searchable error messages, without having to drop into operator logs. b. Add accesscontextmanager.accessLevels.list to invalidPermsForProjectScopedPermissionsTesting [3]. c. Introduce a new flag in the CredentialsRequest CRD to say "skip permissions checking for this request". [1]: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp/1385311485223243776 [2]: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp/1385311485223243776/artifacts/e2e-gcp/gather-extra/artifacts/pods/openshift-cloud-credential-operator_cloud-credential-operator-7868d5bf9-jdt86_cloud-credential-operator.log [3]: https://github.com/openshift/cloud-credential-operator/blob/517f71efb6c5babddc7207b5e6331f51316f5880/pkg/operator/utils/gcp/utils.go#L69-L74