Bug 1953077 - Handling GCP's: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource
Summary: Handling GCP's: Error 400: Permission accesscontextmanager.accessLevels.list ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.8.0
Assignee: Joel Diaz
QA Contact: wang lin
URL:
Whiteboard:
Depends On:
Blocks: 1958983
TreeView+ depends on / blocked
 
Reported: 2021-04-23 19:57 UTC by W. Trevor King
Modified: 2021-07-27 23:03 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Permissions checking when cloud-credential-operator is in Passthrough mode on GCP could cause errors when reconciling certain CredentialsRequest CRs. Consequence: Otherwise valid CredentialsRequest CRs would end up not being reconciled. Fix: Change logic to check against a list of permissions that are valid to TestIamPermissions() against at the GCP Project resource level. Result: CredentialsRequests which include permissions that apply outside of the GCP Project resource level will no longer error during reconcilation.
Clone Of:
Environment:
Last Closed: 2021-07-27 23:03:30 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 330 0 None open Bug 1953077: rework GCP passthrough permissions checking 2021-05-05 18:55:00 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:03:47 UTC

Description W. Trevor King 2021-04-23 19:57:14 UTC
Seen in GCP CI [1], the job fails because:

  : operator conditions cloud-credential	0s
    Operator progressing (Reconciling): 4 of 5 credentials requests provisioned, 0 reporting errors.

Checking all the operator's conditions:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp/1385311485223243776/artifacts/e2e-gcp/gather-extra/artifacts/clusteroperators.json | jq -r '.items[] | select(.metadata.name == "cloud-credential") | .status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + (.reason // "-") + ": " + (.message // "-")'
  2021-04-22T19:25:58Z Available=True -: -
  2021-04-22T19:25:58Z Degraded=False -: -
  2021-04-22T19:34:25Z Progressing=True Reconciling: 4 of 5 credentials requests provisioned, 0 reporting errors.
  2021-04-22T19:25:58Z Upgradeable=True -: -

The cloud-credential-operator-gcp-ro-creds CredentialsRequest contains no status:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp/1385311485223243776/artifacts/e2e-gcp/gather-extra/artifacts/credentialsrequests.json | jq -r '.items[] | select(.metadata.name == "cloud-credential-operator-gcp-ro-creds")'
  {
    "apiVersion": "cloudcredential.openshift.io/v1",
    "kind": "CredentialsRequest",
    "metadata": {
      "annotations": {
        "exclude.release.openshift.io/internal-openshift-hosted": "true",
        "include.release.openshift.io/self-managed-high-availability": "true"
      },
      "creationTimestamp": "2021-04-22T19:25:59Z",
      "finalizers": [
        "cloudcredential.openshift.io/deprovision"
      ],
      "generation": 1,
      "name": "cloud-credential-operator-gcp-ro-creds",
      "namespace": "openshift-cloud-credential-operator",
      "resourceVersion": "2399",
      "uid": "5c8d5d51-80e1-4de8-96c6-b2e8cc71a173"
    },
    "spec": {
      "providerSpec": {
        "apiVersion": "cloudcredential.openshift.io/v1",
        "kind": "GCPProviderSpec",
        "predefinedRoles": [
          "roles/iam.securityReviewer",
          "roles/iam.roleViewer"
        ],
        "skipServiceCheck": true
      },
      "secretRef": {
        "name": "cloud-credential-operator-gcp-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }
    }
  }

And from the operator's log [2]:

  time="2021-04-22T19:51:42Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
  time="2021-04-22T19:51:42Z" level=warning msg="read-only creds not found, using root creds client" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
  time="2021-04-22T19:51:44Z" level=error msg="error syncing credentials: error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
  time="2021-04-22T19:51:44Z" level=error msg="unexpected error while syncing credentialsrequest: error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds

Talking with Joel, the root cause appears to be that there are some permissions which cannot be checked at the project level.  Possible pivots:

a. When we do try to check permissions and fail in that check, start setting status.conditions[] mentioning the failure on the request.  This will allow admins to understand the problem, or at least get some good, searchable error messages, without having to drop into operator logs.
b. Add accesscontextmanager.accessLevels.list to invalidPermsForProjectScopedPermissionsTesting [3].
c. Introduce a new flag in the CredentialsRequest CRD to say "skip permissions checking for this request".

[1]: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp/1385311485223243776
[2]: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp/1385311485223243776/artifacts/e2e-gcp/gather-extra/artifacts/pods/openshift-cloud-credential-operator_cloud-credential-operator-7868d5bf9-jdt86_cloud-credential-operator.log
[3]: https://github.com/openshift/cloud-credential-operator/blob/517f71efb6c5babddc7207b5e6331f51316f5880/pkg/operator/utils/gcp/utils.go#L69-L74

Comment 2 Akhil Rane 2021-04-30 04:00:06 UTC
PR to fix this is open and under review https://github.com/openshift/cloud-credential-operator/pull/330

Comment 4 wang lin 2021-05-10 04:08:33 UTC
Verified on 4.8.0-0.nightly-2021-05-09-105430

1. Create install-config
./openshift-install create install-config 
2. Set cco to Passthrough mode
echo "credentialsMode: Passthrough" >> install-config.yaml
3. Create cluster
./openshift-install create cluster --log-level debug

Install nightly build 4.8.0-0.nightly-2021-05-06-190249 with above steps, after installation, check cco logs, will show the below error:
time="2021-05-10T03:16:03Z" level=error msg="error syncing credentials: error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:16:03Z" level=error msg="unexpected error while syncing credentialsrequest: error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission accesscontextmanager.accessLevels.list is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds

Install nightly build 4.8.0-0.nightly-2021-05-09-105430 with this fix, after installation, check cco logs,cco will ignore project scoped permission check, and won't hit the error.

time="2021-05-10T03:18:49Z" level=info msg="clusteroperator status updated" controller=status
time="2021-05-10T03:19:08Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-gcp-pd-csi-driver-operator
time="2021-05-10T03:19:08Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-gcp-pd-csi-driver-operator
time="2021-05-10T03:19:08Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-gcp-pd-csi-driver-operator
time="2021-05-10T03:19:08Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/openshift-gcp-pd-csi-driver-operator secret=openshift-cluster-csi-drivers/gcp-pd-cloud-credentials
time="2021-05-10T03:19:10Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-05-10T03:19:10Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-05-10T03:19:10Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-05-10T03:19:11Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-gcp secret=openshift-machine-api/gcp-cloud-credentials
time="2021-05-10T03:19:11Z" level=info msg="reconciling clusteroperator status"
time="2021-05-10T03:19:11Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=info msg="clusteroperator status updated" controller=status
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.accessLevels.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.accessPolicies.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.accessPolicies.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.accessZones.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.gcpUserAccessBindings.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.policies.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.policies.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of accesscontextmanager.servicePerimeters.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of assuredworkloads.operations.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of assuredworkloads.workload.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.accounts.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.accounts.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.budgets.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.credits.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.resourceAssociations.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of billing.subscriptions.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudprivatecatalogproducer.associations.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudprivatecatalogproducer.catalogs.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudprivatecatalogproducer.catalogs.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudsupport.accounts.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of cloudsupport.accounts.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of consumerprocurement.accounts.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of consumerprocurement.orders.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of earlyaccesscenter.campaigns.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of earlyaccesscenter.customerAllowlists.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.folders.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.folders.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.organizations.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.tagKeys.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.tagKeys.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.tagValues.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of resourcemanager.tagValues.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of securitycenter.notificationconfig.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of servicemanagement.consumerSettings.getIamPolicy at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:11Z" level=warning msg="Ignoring permission checking of servicemanagement.consumerSettings.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:12Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2021-05-10T03:19:12Z" level=info msg="reconciling clusteroperator status"
time="2021-05-10T03:19:12Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress-gcp
time="2021-05-10T03:19:12Z" level=info msg="clusteroperator status updated" controller=status
time="2021-05-10T03:19:13Z" level=warning msg="Ignoring permission checking of resourcemanager.projects.list at project level" actuator=gcp cr=openshift-cloud-credential-operator/openshift-ingress-gcp
time="2021-05-10T03:19:13Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress-gcp secret=openshift-ingress-operator/cloud-credentials
time="2021-05-10T03:19:13Z" level=info msg="reconciling clusteroperator status"

Comment 5 wang lin 2021-05-10 05:01:01 UTC
Hi Joel, I found the 4.7 has the same issue, do we need to backport this fix to 4.7?

Comment 6 Joel Diaz 2021-05-10 14:34:38 UTC
Yes, I will start a backport to 4.7.

Comment 9 errata-xmlrpc 2021-07-27 23:03:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.