Bug 2043042

Summary: [Serial] [sig-auth][Feature:OAuthServer] [RequestHeaders] [IdP] test RequestHeaders IdP [Suite:openshift/conformance/serial]
Product: OpenShift Container Platform Reporter: Devan Goodwin <dgoodwin>
Component: oauth-apiserverAssignee: Sebastian Łaskawiec <slaskawi>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: high Docs Contact:
Priority: high    
Version: 4.10CC: aos-bugs, kostrows, mfojtik, sippy, surbania, wlewis
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:41:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Devan Goodwin 2022-01-20 14:22:33 UTC
[Serial] [sig-auth][Feature:OAuthServer] [RequestHeaders] [IdP] test RequestHeaders IdP [Suite:openshift/conformance/serial]

is failing frequently in CI, see:

Primary concern here is the serial suite where this is a top failure.

Comment 1 Sebastian Łaskawiec 2022-01-24 08:36:45 UTC
I checked 3 job runs from the following search [1]:
- blob:https://prow.ci.openshift.org/06b592ea-45f5-44dd-b5c9-382498df8490
- blob:https://prow.ci.openshift.org/a5fa04f4-d849-4613-873a-a276c09b4af9
- blob:https://prow.ci.openshift.org/f72cefe4-26d8-44f6-9158-025e74f393d9

In all 3 cases it seems the deployment deployment/oauth-openshift.openshift-authentication hasn't been properly rotated:
-   Message: (string) (len=137) "OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: observed generation is 3, desired generation is 4."
-   Message: (string) (len=138) "OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 2/3 pods have been updated to the latest generation"
-   Message: (string) (len=137) "OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: observed generation is 3, desired generation is 4."

The above line is the last one in the waiting loop from [2]. This means, the CAO flipped to progressing=false and generation 3, which released the waiting loop. Note that we were expecting generation 4 here, not 3. So this means, the tests were running on a stale OAuth configuration.

As for the fix, I believe we need to add deployment generation comparison to the [2] loop. Then, we release the waiting loop only if the current deployment generation matches the expected one. 

[1] https://search.ci.openshift.org/?search=%5C%5BSerial%5C%5D+%5C%5Bsig-auth%5C%5D%5C%5BFeature%3AOAuthServer%5C%5D+%5C%5BRequestHeaders%5C%5D+%5C%5BIdP%5C%5D+test+RequestHeaders+IdP+%5C%5BSuite%3Aopenshift%2Fconformance%2Fserial%5C%5D&maxAge=336h&context=1&type=junit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
[2] https://github.com/openshift/origin/blob/2d4f36ec6b2c141d787c150b542823bf39c40254/test/extended/oauth/requestheaders.go#L446-L473

Comment 3 Xingxing Xia 2022-01-27 00:38:18 UTC
Quickly checked the PR, understand what it improved is to make the test stable by waiting auth pods to really rotate and adding the X-CSRF-Token header to avoid misled. So, since it is test only, moving to VERIFIED directly to save time for other busy Auth work items that I owed and synced with mgmt.

Comment 6 errata-xmlrpc 2022-03-10 16:41:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.