Bug 1967398 - authentication operator still uses previous deleted pod ip rather than the new created pod ip to do health check
Summary: authentication operator still uses previous deleted pod ip rather than the ne...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.8
Hardware: x86_64
OS: Linux
Target Milestone: ---
: 4.8.0
Assignee: Standa Laznicka
QA Contact: pmali
Depends On:
TreeView+ depends on / blocked
Reported: 2021-06-03 06:02 UTC by liyao
Modified: 2021-07-27 23:11 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: A stale condition might have cause the authentication operator to appear degraded after upgrade even though there were no problems. Consequence: False-positive cluster degradation. Fix: Remove old and unused conditions from the operator's status. Result: The authentication operator should correctly report as "Degraded" only when there is an actual problem.
Clone Of:
Last Closed: 2021-07-27 23:11:18 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-authentication-operator pull 449 0 None open Bug 1967398: operator: add OAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded to stale conditions 2021-06-03 10:56:13 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:11:29 UTC

Description liyao 2021-06-03 06:02:08 UTC
Description of problem:
During the upgrade from 4.7.0-0.nightly-2021-05-17-040457 to 4.8.0-0.nightly-2021-05-19-092807, it fails with authentication degraded.

Version-Release number of selected component (if applicable):

4.7.0-0.nightly-2021-05-17-040457 to

How reproducible:
Not sure

Steps to Reproduce:

Actual results:
Upgrade from 4.7.0-0.nightly-2021-05-17-040457 to 4.8.0-0.nightly-2021-05-19-092807 hangs with authentication degraded:
oc describe co authentication shows:
    Last Transition Time:  2021-05-19T14:41:33Z
    Message:               OAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Get "": context canceled
    Reason:                OAuthServiceEndpointsCheckEndpointAccessibleController_SyncError
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2021-05-19T14:42:48Z
    Message:               All is well
    Reason:                AsExpected
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2021-05-19T14:44:48Z
    Message:               All is well
    Reason:                AsExpected
    Status:                True
    Type:                  Available

Check the must gather log, is the ip of the pod which belongs to openshift-authentication but the pod is deleted at "May 19 14:40:35.293321" and new pods are created around "2021-05-19T14:40+" with new ips||

From the upgrade CI log, the health check happens at '[2021-05-19T17:12:11.721Z]', more than 2 hours later, but still uses the previous pod ip( not the new pod ip(|| to do health check. That's the failure reason for health check

must gather log link: http://file.rdu.redhat.com/~xxia/bug_1967398_must-gather.local.5095653185111688673.tar.gz

Expected results:
Upgrade from 4.7.0-0.nightly-2021-05-17-040457 to 4.8.0-0.nightly-2021-05-19-092807 successes.

Additional info:

matrix: 27_UPI on GCP with RHCOS && XPN

Comment 2 liyao 2021-06-08 05:56:29 UTC
Test upgrade from 4.7.0-0.nightly-2021-06-07-095830 to 4.8.0-0.nightly-2021-06-07-180258
$ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-06-07-180258 --force=true --allow-explicit-upgrade=true

During the upgrade process, force update the oauth configuration 5 times to redeploy new pods with new ips, original issue hanging with old pod's IP is gone
$ oc edit oauth cluster

Check cluster version after upgarde finished
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-07-180258   True        False         123m    Cluster version is 4.8.0-0.nightly-2021-06-07-180258

Comment 5 errata-xmlrpc 2021-07-27 23:11:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.