Bug 1876571

Summary:	Openshift installation fails - authentication and network operators are in degraded state
Product:	OpenShift Container Platform	Reporter:	Itzik Brown <itbrown>
Component:	Networking	Assignee:	Maysa Macedo <mdemaced>
Networking sub component:	kuryr	QA Contact:	GenadiC <gcheresh>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	high	CC:	wjiang, wsun
Version:	4.6	Keywords:	TestBlocker
Target Milestone:	---
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 16:38:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Itzik Brown 2020-09-07 14:11:58 UTC

Description of problem:
As described

Version-Release number of selected component (if applicable):
OCP 4.6.0-0.nightly-2020-09-04-093211
OSP 13 2020-06-11.1

It seems that there are missing security group rules 
e.g. missing 443/TCP for openshift-authentication service

$ openstack security group rule list |grep ea5a68c1-d499-4358-98a3-bc890be16cc4
| 23920e92-b301-48cd-beac-b77a37209fc6 | None        | None           |             | None                                 | ea5a68c1-d499-4358-98a3-bc890be16cc4 |
| 8667a067-c408-4238-96e9-4cd144d100d0 | tcp         | None           | 1025:1025   | None                                 | ea5a68c1-d499-4358-98a3-bc890be16cc4 |
| 89ceda08-21aa-458c-944d-2f83713e1385 | None        | None           |             | None  




How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
$ oc get pods -n openshift-kuryr
NAME                                   READY   STATUS    RESTARTS   AGE
kuryr-cni-2wrjc                        1/1     Running   0          10h
kuryr-cni-cmwx5                        1/1     Running   3          11h
kuryr-cni-fxsgb                        1/1     Running   1          11h
kuryr-cni-gggfk                        1/1     Running   2          11h
kuryr-cni-twmsg                        1/1     Running   2          11h
kuryr-cni-v4x9p                        1/1     Running   106        7h48m
kuryr-controller-7b457fcd9-gs5tn       1/1     Running   10         125m
kuryr-dns-admission-controller-m56w5   1/1     Running   0          11h
kuryr-dns-admission-controller-w9shl   1/1     Running   0          11h
kuryr-dns-admission-controller-wb9b4   1/1     Running   0          11h

Comment 3 weiwei jiang 2020-09-10 07:27:42 UTC

Checked with 4.6.0-0.nightly-2020-09-09-224210, and it's fixed now.

$ oc get co 
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.nightly-2020-09-09-224210   True        False         False      37m
cloud-credential                           4.6.0-0.nightly-2020-09-09-224210   True        False         False      85m
cluster-autoscaler                         4.6.0-0.nightly-2020-09-09-224210   True        False         False      72m
config-operator                            4.6.0-0.nightly-2020-09-09-224210   True        False         False      76m
console                                    4.6.0-0.nightly-2020-09-09-224210   True        False         False      38m
csi-snapshot-controller                    4.6.0-0.nightly-2020-09-09-224210   True        False         False      37m
dns                                        4.6.0-0.nightly-2020-09-09-224210   True        False         False      74m
etcd                                       4.6.0-0.nightly-2020-09-09-224210   True        False         False      79m
image-registry                             4.6.0-0.nightly-2020-09-09-224210   True        False         False      44m
ingress                                    4.6.0-0.nightly-2020-09-09-224210   True        False         False      65m
insights                                   4.6.0-0.nightly-2020-09-09-224210   True        False         False      72m
kube-apiserver                             4.6.0-0.nightly-2020-09-09-224210   True        False         False      78m
kube-controller-manager                    4.6.0-0.nightly-2020-09-09-224210   True        False         False      76m
kube-scheduler                             4.6.0-0.nightly-2020-09-09-224210   True        False         False      76m
kube-storage-version-migrator              4.6.0-0.nightly-2020-09-09-224210   True        False         False      43m
machine-api                                4.6.0-0.nightly-2020-09-09-224210   True        False         False      68m
machine-approver                           4.6.0-0.nightly-2020-09-09-224210   True        False         False      74m
machine-config                             4.6.0-0.nightly-2020-09-09-224210   True        False         False      71m
marketplace                                4.6.0-0.nightly-2020-09-09-224210   True        False         False      38m
monitoring                                 4.6.0-0.nightly-2020-09-09-224210   True        False         False      37m
network                                    4.6.0-0.nightly-2020-09-09-224210   True        False         False      80m
node-tuning                                4.6.0-0.nightly-2020-09-09-224210   True        False         False      76m
openshift-apiserver                        4.6.0-0.nightly-2020-09-09-224210   True        False         False      42m
openshift-controller-manager               4.6.0-0.nightly-2020-09-09-224210   True        False         False      72m
openshift-samples                          4.6.0-0.nightly-2020-09-09-224210   True        False         False      72m
operator-lifecycle-manager                 4.6.0-0.nightly-2020-09-09-224210   True        False         False      76m
operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-09-09-224210   True        False         False      76m
operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-09-09-224210   True        False         False      24m
service-ca                                 4.6.0-0.nightly-2020-09-09-224210   True        False         False      79m
storage                                    4.6.0-0.nightly-2020-09-09-224210   True        False         False      38m

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-09-09-224210   True        False         47m     Cluster version is 4.6.0-0.nightly-2020-09-09-224210

$ oc get nodes -o wide 
NAME                                 STATUS   ROLES    AGE   VERSION                INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
wj46ioskr910a-klxng-master-0         Ready    master   85m   v1.19.0-rc.2+40d85fc   192.168.2.182   <none>        Red Hat Enterprise Linux CoreOS 46.82.202009091306-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-11.rhaos4.6.gitf83564f.el8-rc.1
wj46ioskr910a-klxng-master-1         Ready    master   85m   v1.19.0-rc.2+40d85fc   192.168.0.142   <none>        Red Hat Enterprise Linux CoreOS 46.82.202009091306-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-11.rhaos4.6.gitf83564f.el8-rc.1
wj46ioskr910a-klxng-master-2         Ready    master   86m   v1.19.0-rc.2+40d85fc   192.168.1.19    <none>        Red Hat Enterprise Linux CoreOS 46.82.202009091306-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-11.rhaos4.6.gitf83564f.el8-rc.1
wj46ioskr910a-klxng-worker-0-khcrx   Ready    worker   67m   v1.19.0-rc.2+40d85fc   192.168.1.22    <none>        Red Hat Enterprise Linux CoreOS 46.82.202009091306-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-11.rhaos4.6.gitf83564f.el8-rc.1
wj46ioskr910a-klxng-worker-0-ndjtw   Ready    worker   68m   v1.19.0-rc.2+40d85fc   192.168.1.129   <none>        Red Hat Enterprise Linux CoreOS 46.82.202009091306-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-11.rhaos4.6.gitf83564f.el8-rc.1
wj46ioskr910a-klxng-worker-0-q9c66   Ready    worker   66m   v1.19.0-rc.2+40d85fc   192.168.1.171   <none>        Red Hat Enterprise Linux CoreOS 46.82.202009091306-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-11.rhaos4.6.gitf83564f.el8-rc.1


$ oc describe co authentication 
Name:         authentication
Namespace:    
Labels:       <none>
Annotations:  exclude.release.openshift.io/internal-openshift-hosted: true
API Version:  config.openshift.io/v1
Kind:         ClusterOperator
Metadata:
  Creation Timestamp:  2020-09-10T05:58:24Z
  Generation:          1
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:exclude.release.openshift.io/internal-openshift-hosted:
      f:spec:
      f:status:
        .:
        f:extension:
    Manager:      cluster-version-operator
    Operation:    Update
    Time:         2020-09-10T05:58:24Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:relatedObjects:
        f:versions:
    Manager:         authentication-operator
    Operation:       Update
    Time:            2020-09-10T06:48:59Z
  Resource Version:  77033
  Self Link:         /apis/config.openshift.io/v1/clusteroperators/authentication
  UID:               9aecec81-66e6-455f-a1b4-50d2deff6d3e
Spec:
Status:
  Conditions:
    Last Transition Time:  2020-09-10T06:37:36Z
    Reason:                AsExpected
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2020-09-10T06:48:32Z
    Reason:                AsExpected
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2020-09-10T06:48:46Z
    Message:               OAuthServerDeploymentAvailable: availableReplicas==2
    Reason:                AsExpected
    Status:                True
    Type:                  Available
    Last Transition Time:  2020-09-10T06:05:51Z
    Reason:                AsExpected
    Status:                True
    Type:                  Upgradeable
  Extension:               <nil>
  Related Objects:
    Group:      operator.openshift.io
    Name:       cluster
    Resource:   authentications
    Group:      config.openshift.io
    Name:       cluster
    Resource:   authentications
    Group:      config.openshift.io
    Name:       cluster
    Resource:   infrastructures
    Group:      config.openshift.io
    Name:       cluster
    Resource:   oauths
    Group:      route.openshift.io
    Name:       oauth-openshift
    Namespace:  openshift-authentication
    Resource:   routes
    Group:      
    Name:       oauth-openshift
    Namespace:  openshift-authentication
    Resource:   services
    Group:      
    Name:       openshift-config
    Resource:   namespaces
    Group:      
    Name:       openshift-config-managed
    Resource:   namespaces
    Group:      
    Name:       openshift-authentication
    Resource:   namespaces
    Group:      
    Name:       openshift-authentication-operator
    Resource:   namespaces
    Group:      
    Name:       openshift-ingress
    Resource:   namespaces
    Group:      
    Name:       openshift-oauth-apiserver
    Resource:   namespaces
  Versions:
    Name:     oauth-apiserver
    Version:  4.6.0-0.nightly-2020-09-09-224210
    Name:     operator
    Version:  4.6.0-0.nightly-2020-09-09-224210
    Name:     oauth-openshift
    Version:  4.6.0-0.nightly-2020-09-09-224210_openshift
Events:       <none>

Comment 5 errata-xmlrpc 2020-10-27 16:38:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196