Bug 1908217 - CI: Server-Side Apply should work for oauth.openshift.io/v1: has no tokens
Summary: CI: Server-Side Apply should work for oauth.openshift.io/v1: has no tokens
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.6
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.7.0
Assignee: Standa Laznicka
QA Contact: pmali
URL:
Whiteboard:
Depends On:
Blocks: 1930724
TreeView+ depends on / blocked
 
Reported: 2020-12-16 05:29 UTC by W. Trevor King
Modified: 2021-02-24 15:45 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:44:54 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 25780 0 None closed Bug 1908217: server-side apply e2e: fix oauthclientauthorization race with KCM 2021-02-19 06:21:51 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:45:16 UTC

Description W. Trevor King 2020-12-16 05:29:57 UTC
Seems popular in CI:

$ w3m -dump -cols 200 'https://search.ci.openshift.org/?search=OAuthClientAuthorization.oauth.openshift.io+%22user%3Asystem%3Aserviceaccount%3A.*+is+invalid%3A+clientName%3A+Internal+error%3A+system%3Aserviceaccount%3A.*+has+no+tokens&maxAge=24h&type=junit' | grep 'failures match' | sort
periodic-ci-openshift-release-master-ocp-4.6-e2e-vsphere - 5 runs, 80% failed, 25% of failures match
periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere - 7 runs, 100% failed, 14% of failures match
periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere-upi - 7 runs, 86% failed, 17% of failures match
pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws - 2 runs, 50% failed, 100% of failures match
...
pull-ci-openshift-origin-master-e2e-gcp - 33 runs, 48% failed, 31% of failures match
pull-ci-openshift-ovn-kubernetes-master-e2e-openstack-ovn - 4 runs, 100% failed, 25% of failures match
pull-ci-openshift-ovn-kubernetes-release-4.6-e2e-vsphere-ovn - 20 runs, 100% failed, 15% of failures match
rehearse-13921-pull-ci-openshift-machine-config-operator-release-4.7-e2e-vsphere - 2 runs, 100% failed, 50% of failures match
...
rehearse-14305-pull-ci-openshift-cluster-update-keys-release-4.7-okd-e2e-aws - 2 runs, 50% failed, 100% of failures match
release-openshift-ocp-installer-e2e-gcp-4.6 - 3 runs, 67% failed, 100% of failures match
release-openshift-ocp-installer-e2e-gcp-4.7 - 3 runs, 100% failed, 33% of failures match
release-openshift-ocp-installer-e2e-gcp-rt-4.7 - 3 runs, 100% failed, 33% of failures match
release-openshift-origin-installer-e2e-gcp-4.8 - 12 runs, 42% failed, 20% of failures match

Example job [1] failed:

  [sig-api-machinery][Feature:ServerSideApply] Server-Side Apply should work for oauth.openshift.io/v1, Resource=oauthclientauthorizations [Suite:openshift/conformance/parallel]

with:

  fail [github.com/openshift/origin/test/extended/apiserver/apply.go:171]: 
  Unexpected error:
    <*errors.StatusError | 0xc00213b9a0>: {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "OAuthClientAuthorization.oauth.openshift.io \"user:system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg\" is invalid: clientName: Internal error: system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg has no tokens",
            Reason: "Invalid",
            Details: {
                Name: "user:system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg",
                Group: "oauth.openshift.io",
                Kind: "OAuthClientAuthorization",
                UID: "",
                Causes: [
                    {
                        Type: "InternalError",
                        Message: "Internal error: system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg has no tokens",
                        Field: "clientName",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 422,
        },
    }
    OAuthClientAuthorization.oauth.openshift.io "user:system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg" is invalid: clientName: Internal error: system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg has no tokens
occurred

Luckily for CI, a number of jobs like [2] will fail once, but pass on retest, in which case the initial failure is non-fatal.  But would still be nice to drive this flake out entirely.

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/490/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1339037164188471296
[2]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-4.6/1339014074452676608

Comment 5 errata-xmlrpc 2021-02-24 15:44:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.