Bug 2043042 - [Serial] [sig-auth][Feature:OAuthServer] [RequestHeaders] [IdP] test RequestHeaders IdP [Suite:openshift/conformance/serial]
Summary: [Serial] [sig-auth][Feature:OAuthServer] [RequestHeaders] [IdP] test RequestH...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oauth-apiserver
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Sebastian Łaskawiec
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-20 14:22 UTC by Devan Goodwin
Modified: 2022-03-10 16:41 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:41:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 26777 0 None open Bug 2043042: Additional checks for requestheaders.go 2022-01-26 07:43:56 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:41:37 UTC

Description Devan Goodwin 2022-01-20 14:22:33 UTC
[Serial] [sig-auth][Feature:OAuthServer] [RequestHeaders] [IdP] test RequestHeaders IdP [Suite:openshift/conformance/serial]

is failing frequently in CI, see:
https://sippy.ci.openshift.org/sippy-ng/tests/4.10/analysis?test=%5BSerial%5D%20%5Bsig-auth%5D%5BFeature%3AOAuthServer%5D%20%5BRequestHeaders%5D%20%5BIdP%5D%20test%20RequestHeaders%20IdP%20%5BSuite%3Aopenshift%2Fconformance%2Fserial%5D

Primary concern here is the serial suite where this is a top failure.

Comment 1 Sebastian Łaskawiec 2022-01-24 08:36:45 UTC
I checked 3 job runs from the following search [1]:
- blob:https://prow.ci.openshift.org/06b592ea-45f5-44dd-b5c9-382498df8490
- blob:https://prow.ci.openshift.org/a5fa04f4-d849-4613-873a-a276c09b4af9
- blob:https://prow.ci.openshift.org/f72cefe4-26d8-44f6-9158-025e74f393d9

In all 3 cases it seems the deployment deployment/oauth-openshift.openshift-authentication hasn't been properly rotated:
-   Message: (string) (len=137) "OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: observed generation is 3, desired generation is 4."
-   Message: (string) (len=138) "OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 2/3 pods have been updated to the latest generation"
-   Message: (string) (len=137) "OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: observed generation is 3, desired generation is 4."

The above line is the last one in the waiting loop from [2]. This means, the CAO flipped to progressing=false and generation 3, which released the waiting loop. Note that we were expecting generation 4 here, not 3. So this means, the tests were running on a stale OAuth configuration.

As for the fix, I believe we need to add deployment generation comparison to the [2] loop. Then, we release the waiting loop only if the current deployment generation matches the expected one. 

[1] https://search.ci.openshift.org/?search=%5C%5BSerial%5C%5D+%5C%5Bsig-auth%5C%5D%5C%5BFeature%3AOAuthServer%5C%5D+%5C%5BRequestHeaders%5C%5D+%5C%5BIdP%5C%5D+test+RequestHeaders+IdP+%5C%5BSuite%3Aopenshift%2Fconformance%2Fserial%5C%5D&maxAge=336h&context=1&type=junit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
[2] https://github.com/openshift/origin/blob/2d4f36ec6b2c141d787c150b542823bf39c40254/test/extended/oauth/requestheaders.go#L446-L473

Comment 3 Xingxing Xia 2022-01-27 00:38:18 UTC
Quickly checked the PR, understand what it improved is to make the test stable by waiting auth pods to really rotate and adding the X-CSRF-Token header to avoid misled. So, since it is test only, moving to VERIFIED directly to save time for other busy Auth work items that I owed and synced with mgmt.

Comment 6 errata-xmlrpc 2022-03-10 16:41:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.