Bug 1908217

Summary: CI: Server-Side Apply should work for oauth.openshift.io/v1: has no tokens
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: apiserver-authAssignee: Standa Laznicka <slaznick>
Status: CLOSED ERRATA QA Contact: pmali
Severity: low Docs Contact:
Priority: low    
Version: 4.6CC: aos-bugs, mfojtik, pmali, scheng, xxia
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:44:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1930724    

Description W. Trevor King 2020-12-16 05:29:57 UTC 
Seems popular in CI:

$ w3m -dump -cols 200 'https://search.ci.openshift.org/?search=OAuthClientAuthorization.oauth.openshift.io+%22user%3Asystem%3Aserviceaccount%3A.*+is+invalid%3A+clientName%3A+Internal+error%3A+system%3Aserviceaccount%3A.*+has+no+tokens&maxAge=24h&type=junit' | grep 'failures match' | sort
periodic-ci-openshift-release-master-ocp-4.6-e2e-vsphere - 5 runs, 80% failed, 25% of failures match
periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere - 7 runs, 100% failed, 14% of failures match
periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere-upi - 7 runs, 86% failed, 17% of failures match
pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws - 2 runs, 50% failed, 100% of failures match
...
pull-ci-openshift-origin-master-e2e-gcp - 33 runs, 48% failed, 31% of failures match
pull-ci-openshift-ovn-kubernetes-master-e2e-openstack-ovn - 4 runs, 100% failed, 25% of failures match
pull-ci-openshift-ovn-kubernetes-release-4.6-e2e-vsphere-ovn - 20 runs, 100% failed, 15% of failures match
rehearse-13921-pull-ci-openshift-machine-config-operator-release-4.7-e2e-vsphere - 2 runs, 100% failed, 50% of failures match
...
rehearse-14305-pull-ci-openshift-cluster-update-keys-release-4.7-okd-e2e-aws - 2 runs, 50% failed, 100% of failures match
release-openshift-ocp-installer-e2e-gcp-4.6 - 3 runs, 67% failed, 100% of failures match
release-openshift-ocp-installer-e2e-gcp-4.7 - 3 runs, 100% failed, 33% of failures match
release-openshift-ocp-installer-e2e-gcp-rt-4.7 - 3 runs, 100% failed, 33% of failures match
release-openshift-origin-installer-e2e-gcp-4.8 - 12 runs, 42% failed, 20% of failures match

Example job [1] failed:

  [sig-api-machinery][Feature:ServerSideApply] Server-Side Apply should work for oauth.openshift.io/v1, Resource=oauthclientauthorizations [Suite:openshift/conformance/parallel]

with:

  fail [github.com/openshift/origin/test/extended/apiserver/apply.go:171]: 
  Unexpected error:
    <*errors.StatusError | 0xc00213b9a0>: {
        ErrStatus: {
            TypeMeta: {Kind: "Status", APIVersion: "v1"},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "OAuthClientAuthorization.oauth.openshift.io \"user:system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg\" is invalid: clientName: Internal error: system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg has no tokens",
            Reason: "Invalid",
            Details: {
                Name: "user:system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg",
                Group: "oauth.openshift.io",
                Kind: "OAuthClientAuthorization",
                UID: "",
                Causes: [
                    {
                        Type: "InternalError",
                        Message: "Internal error: system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg has no tokens",
                        Field: "clientName",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 422,
        },
    }
    OAuthClientAuthorization.oauth.openshift.io "user:system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg" is invalid: clientName: Internal error: system:serviceaccount:e2e-test-server-side-apply-q4xfw:clientg has no tokens
occurred

Luckily for CI, a number of jobs like [2] will fail once, but pass on retest, in which case the initial failure is non-fatal.  But would still be nice to drive this flake out entirely.

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/490/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1339037164188471296
[2]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-4.6/1339014074452676608
Comment 5 errata-xmlrpc 2021-02-24 15:44:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633