Bug 1750953

Summary: Authentication operator does not reach full health. RouteHealthDegraded: failed to GET route: net/http: TLS handshake timeout
Product: OpenShift Container Platform Reporter: jabulisile
Component: NetworkingAssignee: Dan Mace <dmace>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: high CC: aos-bugs, dhansen, dmace, dustymabe, mfojtik, sttts
Version: 4.1.zKeywords: Reopened
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-24 19:46:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Must-gather none

Description jabulisile 2019-09-10 20:24:58 UTC
Created attachment 1613763 [details]
Must-gather

Description of problem:
Openshift 4.1.14 does not complete installation because the authentication operator remains in degraded status, as well as console. All other operators including ingress are available.

Version-Release number of selected component (if applicable):
./openshift-install v4.1.14-201908291507-dirty
built from commit 95db8aae6559f45b06afce8838a569597c7a4dc3
release image quay.io/openshift-release-dev/ocp-release@sha256:fd41c9bda9e0ff306954f1fd7af6428edff8c3989b75f9fe984968db66846231

How reproducible:
./openshift-install wait-for bootstrap-complete --dir=custom-install --log-level debug

Steps to Reproduce:
1./openshift-install wait-for bootstrap-complete --dir=custom-install --log-level debug


Actual results:
oc get co
NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                                 Unknown     Unknown       True       30h
cloud-credential                     4.1.14    True        True          True       30h
cluster-autoscaler                   4.1.14    True        False         False      30h
console                              4.1.14    False       True          False      29h
dns                                  4.1.14    True        False         False      30h
image-registry                       4.1.14    True        False         False      9h
ingress                              4.1.14    True        False         False      29h
kube-apiserver                       4.1.14    True        False         False      30h
kube-controller-manager              4.1.14    True        False         False      30h
kube-scheduler                       4.1.14    True        False         False      30h
machine-api                          4.1.14    True        False         False      30h
machine-config                       4.1.14    True        False         False      30h
marketplace                          4.1.14    True        False         False      30h
monitoring                           4.1.14    True        False         False      29h
network                              4.1.14    True        False         False      30h
node-tuning                          4.1.14    True        False         False      30h
openshift-apiserver                  4.1.14    True        False         False      30h
openshift-controller-manager         4.1.14    True        False         False      30h
openshift-samples                    4.1.14    True        False         False      30h
operator-lifecycle-manager           4.1.14    True        False         False      30h
operator-lifecycle-manager-catalog   4.1.14    True        False         False      30h
service-ca                           4.1.14    True        False         False      30h
service-catalog-apiserver            4.1.14    True        False         False      30h
service-catalog-controller-manager   4.1.14    True        False         False      30h
storage                              4.1.14    True        False         False      30h

Expected results:
Successful installation of cluster. 

Additional info:
Must-gather attached

Comment 1 Standa Laznicka 2019-09-12 13:11:41 UTC
Since the authentication operator logs:
```
2019-09-09T15:10:35.224083399Z E0909 15:10:35.224008       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: error checking current version: unable to check route health: failed to GET route: net/http: TLS handshake timeout
```
and prior to that, the route seemed to be causing problems in the logs as well, I'm guessing this will be problem in either routing or underlying SDN.

Moving to routing.

Comment 2 Dan Mace 2019-10-07 17:17:17 UTC
*** Bug 1759246 has been marked as a duplicate of this bug. ***

Comment 3 Daneyon Hansen 2019-10-22 16:53:30 UTC
I can not reproduce this bug.

$ ~/Downloads/openshift-install-mac-4.1.14/openshift-install version
/Users/daneyonhansen/Downloads/openshift-install-mac-4.1.14/openshift-install v4.1.14-201908291507-dirty
built from commit 95db8aae6559f45b06afce8838a569597c7a4dc3
release image quay.io/openshift-release-dev/ocp-release@sha256:fd41c9bda9e0ff306954f1fd7af6428edff8c3989b75f9fe984968db66846231
$ ~/Downloads/openshift-install-mac-4.1.14/openshift-install --dir bz1750953 create cluster
? SSH Public Key /Users/daneyonhansen/.ssh/id_rsa.pub
? Platform aws
? Region us-west-2
? Base Domain devcluster.openshift.com
? Cluster Name bz1750953
? Pull Secret [? for help] **********************************************************************************************************************************************INFO Creating infrastructure resources...         ***********************************************************************************************************************INFO Waiting up to 30m0s for the Kubernetes API at https://api.bz1750953.devcluster.openshift.com:6443... ***************************************************************INFO API v1.13.4+b758672 up                       
INFO Waiting up to 30m0s for bootstrapping to complete... 
INFO Destroying the bootstrap resources...        
INFO Waiting up to 30m0s for the cluster at https://api.bz1750953.devcluster.openshift.com:6443 to initialize... 
INFO Waiting up to 10m0s for the openshift-console route to be created... 
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/Users/daneyonhansen/code/go/src/github.com/openshift/installer/bz1750953/auth/kubeconfig' 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.bz1750953.devcluster.openshift.com 
INFO Login to the console with user: kubeadmin, password: 8I25x-SNhoh-NisvU-9ALps 
$ export KUBECONFIG=/Users/daneyonhansen/code/go/src/github.com/openshift/installer/bz1750953/auth/kubeconfig
$ oc get clusteroperators
NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                       4.1.14    True        False         False      16h
cloud-credential                     4.1.14    True        False         False      16h
cluster-autoscaler                   4.1.14    True        False         False      16h
console                              4.1.14    True        False         False      16h
dns                                  4.1.14    True        False         False      16h
image-registry                       4.1.14    True        False         False      16h
ingress                              4.1.14    True        False         False      16h
kube-apiserver                       4.1.14    True        False         False      16h
kube-controller-manager              4.1.14    True        False         False      16h
kube-scheduler                       4.1.14    True        False         False      16h
machine-api                          4.1.14    True        False         False      16h
machine-config                       4.1.14    True        False         False      16h
marketplace                          4.1.14    True        False         False      16h
monitoring                           4.1.14    True        False         False      16h
network                              4.1.14    True        False         False      16h
node-tuning                          4.1.14    True        False         False      16h
openshift-apiserver                  4.1.14    True        False         False      16h
openshift-controller-manager         4.1.14    True        False         False      16h
openshift-samples                    4.1.14    True        False         False      16h
operator-lifecycle-manager           4.1.14    True        False         False      16h
operator-lifecycle-manager-catalog   4.1.14    True        False         False      16h
service-ca                           4.1.14    True        False         False      16h
service-catalog-apiserver            4.1.14    True        False         False      16h
service-catalog-controller-manager   4.1.14    True        False         False      16h
storage                              4.1.14    True        False         False      16h

Comment 4 Dan Mace 2019-10-23 21:14:21 UTC
This one shouldn't have been closed, I'm using it to try and consolidate tracking the related auth/ingress flake. Sorry for the misunderstanding!

Comment 5 Dan Mace 2019-10-24 19:46:42 UTC

*** This bug has been marked as a duplicate of bug 1765276 ***