Bug 1859591 - authentication operator does not get ready when installing a cluster behind https proxy
Summary: authentication operator does not get ready when installing a cluster behind h...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oauth-proxy
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Standa Laznicka
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-22 14:20 UTC by Johnny Liu
Modified: 2020-10-27 16:17 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:16:41 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-authentication-operator pull 314 0 None closed Bug 1859591: version controller: honor system CA bundle when checking route health 2021-01-25 12:37:12 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:17:00 UTC

Description Johnny Liu 2020-07-22 14:20:59 UTC
Description of problem:


Version-Release number of selected component (if applicable):
4.6.0-0.nightly-2020-07-21-200036

How reproducible:
Always

Steps to Reproduce:
1. install a cluster behind https proxy
apiVersion: v1
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 0
metadata:
  name: jialiu-6474
platform:
  none: {}
pullSecret: HIDDEN
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  <--snip-->
  -----END CERTIFICATE-----
proxy:
  httpProxy: http://proxy-user1:XXX@10.0.77.163:3128
  httpsProxy: https://proxy-user1:XXXX@10.0.77.163:3130   ---> This is a real https proxy
  noProxy: test.no-proxy.com
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
  networkType: OpenShiftSDN
  machineNetwork:
  - cidr: 10.0.0.0/16
fips: true
publish: External
baseDomain: qe.devcluster.openshift.com
sshKey: ssh-rsa YYYYYYYZZZZZZ openshift-qe@redhat.com
2. Trigger installation
3. 

Actual results:
$ openshift-install wait-for install-complete --dir '/home/installer1/workspace/Launch Environment Flexy/workdir/install-dir'
level=info msg="Waiting up to 30m0s for the cluster at https://api.jialiu-6474.qe.devcluster.openshift.com:6443 to initialize..."
E0722 08:06:35.085767    5899 reflector.go:307] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: Get "https://api.jialiu-6474.qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dversion&resourceVersion=19598&timeoutSeconds=480&watch=true": net/http: TLS handshake timeout
level=info msg="Cluster operator authentication Progressing is True with OAuthVersionRoute_WaitingForRoute: OAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.jialiu-6474.qe.devcluster.openshift.com/healthz\" not successfull yet"
level=info msg="Cluster operator authentication Available is False with OAuthVersionRoute_RequestFailed: OAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.jialiu-6474.qe.devcluster.openshift.com/healthz\" failed: proxyconnect tcp: x509: certificate signed by unknown authority"
level=info msg="Cluster operator insights Disabled is False with AsExpected: "
level=fatal msg="failed to initialize the cluster: Cluster operator authentication is still updating"

[root@preserve-jialiu-ansible ~]# oc describe co authentication
Name:         authentication
Namespace:    
Labels:       <none>
Annotations:  exclude.release.openshift.io/internal-openshift-hosted: true
API Version:  config.openshift.io/v1
Kind:         ClusterOperator
Metadata:
  Creation Timestamp:  2020-07-22T11:52:46Z
  Generation:          1
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:exclude.release.openshift.io/internal-openshift-hosted:
      f:spec:
      f:status:
        .:
        f:extension:
        f:versions:
    Manager:      cluster-version-operator
    Operation:    Update
    Time:         2020-07-22T11:52:46Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:relatedObjects:
    Manager:         authentication-operator
    Operation:       Update
    Time:            2020-07-22T12:06:13Z
  Resource Version:  20249
  Self Link:         /apis/config.openshift.io/v1/clusteroperators/authentication
  UID:               4ff0a43c-e589-4bcd-8961-ca702bc77317
Spec:
Status:
  Conditions:
    Last Transition Time:  2020-07-22T12:06:09Z
    Reason:                AsExpected
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2020-07-22T12:05:26Z
    Message:               OAuthVersionRouteProgressing: Request to "https://oauth-openshift.apps.jialiu-6474.qe.devcluster.openshift.com/healthz" not successfull yet
    Reason:                OAuthVersionRoute_WaitingForRoute
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2020-07-22T12:05:26Z
    Message:               OAuthVersionRouteAvailable: HTTP request to "https://oauth-openshift.apps.jialiu-6474.qe.devcluster.openshift.com/healthz" failed: proxyconnect tcp: x509: certificate signed by unknown authority
    Reason:                OAuthVersionRoute_RequestFailed
    Status:                False
    Type:                  Available
    Last Transition Time:  2020-07-22T11:56:15Z
    Reason:                AsExpected
    Status:                True
    Type:                  Upgradeable
  Extension:               <nil>
  Related Objects:
    Group:      operator.openshift.io
    Name:       cluster
    Resource:   authentications
    Group:      config.openshift.io
    Name:       cluster
    Resource:   authentications
    Group:      config.openshift.io
    Name:       cluster
    Resource:   infrastructures
    Group:      config.openshift.io
    Name:       cluster
    Resource:   oauths
    Group:      route.openshift.io
    Name:       oauth-openshift
    Namespace:  openshift-authentication
    Resource:   routes
    Group:      
    Name:       oauth-openshift
    Namespace:  openshift-authentication
    Resource:   services
    Group:      
    Name:       openshift-config
    Resource:   namespaces
    Group:      
    Name:       openshift-config-managed
    Resource:   namespaces
    Group:      
    Name:       openshift-authentication
    Resource:   namespaces
    Group:      
    Name:       openshift-authentication-operator
    Resource:   namespaces
    Group:      
    Name:       openshift-ingress
    Resource:   namespaces
Events:         <none>


Expected results:
auth operator get ready behind https proxy

Additional info:

Comment 2 Standa Laznicka 2020-07-22 14:33:04 UTC
Last time I checked https:// proxy was unsupported:

https://docs.openshift.com/container-platform/4.5/networking/enable-cluster-wide-proxy.html - The URL scheme must be http; https is currently not supported

Comment 3 Standa Laznicka 2020-07-22 14:37:13 UTC
Prove me wrong or close the BZ as invalid please.

Comment 4 Johnny Liu 2020-07-22 14:48:46 UTC
Okay, sound good to me
But what is curious to me, I just checked 4.5 QE CI jobs history, 4.5 installation did not hit such problems.

Comment 6 Johnny Liu 2020-07-23 03:01:23 UTC
Some more details about why "https is not supported", https://github.com/openshift/openshift-docs/pull/16635#discussion_r327846123

Comment 7 Standa Laznicka 2020-08-04 08:29:26 UTC
Thank you for bearing with me and for the explanation about why we did not claim HTTPS proxy support yet. I found a bug related to the recent refactoring and posted a fix.

Comment 10 Johnny Liu 2020-08-07 07:27:57 UTC
Verified this bug with 4.6.0-0.nightly-2020-08-06-233752, and PASS.

Comment 12 errata-xmlrpc 2020-10-27 16:16:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.