Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1893803

Summary: false-positive ReadyIngressNodes_NoReadyIngressNodes: Auth operator makes risky "worker" assumption when guessing about ingress availability
Product: OpenShift Container Platform Reporter: Scott Dodson <sdodson>
Component: apiserver-authAssignee: Standa Laznicka <slaznick>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.6CC: aos-bugs, lmohanty, mfojtik, mifiedle, pmali, sttts, wking
Target Milestone: ---Keywords: Upgrades
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1893386 Environment:
Last Closed: 2020-11-16 14:37:43 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1893386    
Bug Blocks:    

Description Scott Dodson 2020-11-02 16:29:59 UTC 
+++ This bug was initially created as a clone of Bug #1893386 +++

Description of problem:

In 4.6, the auth operator grew logic to check if the router could schedule pods, but that logic assumes the router will be scheduled on "worker"-labeled nodes [1].

Version-Release number of selected component (if applicable):

4.6 and later.

How reproducible:

100%, when you have no vanilla 'worker' nodes.

Steps to Reproduce:

1. Have no vanilla 'worker' nodes, but have a bunch of custom compute pools [1]
2. Try to survive on 4.6+

Actual results:

Watch the authentication operator complain: Available=False ReadyIngressNodes_NoReadyIngressNodes ReadyIngressNodesAvailable: Authentication require functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes and 3 master nodes (none are schedulable or ready for ingress pods).

Expected results:

Auth operator minds its own business, and the ingress operator complains when it is unscheduled (bug 1881155, [3]) ;).

Additional info:

Auth operator going Available=False on this can hang updates, e.g. updates from 4.5 into 4.6 for folks without vanilla compute nodes.

Workaround: scale up at least one node with a 'node-role.kubernetes.io/worker' label and an empty value.

[1]: https://github.com/openshift/cluster-authentication-operator/pull/344/files#diff-74035431d399f5431916d8624ce3080db323d3b4762cb875651311d703168425R66
[2]: https://github.com/openshift/machine-config-operator/blob/0170e082a8b8228373bd841d17555fff2cfb51b7/docs/custom-pools.md#custom-pools
[3]: https://github.com/openshift/cluster-ingress-operator/pull/465

--- Additional comment from W. Trevor King on 2020-10-30 22:14:13 EDT ---

The check for "am I impacted by this?" looks like a ReadyIngressNodes_NoReadyIngressNodes Available=False authentication operator:

$ oc get -o json clusteroperator authentication | jq -r '.status.conditions[] | select(.type == "Available") | .lastTransitionTime + " " + .type + " " + .status + " " + (.reason // "-") + " " + (.message // "-")'
2020-10-29T12:49:29Z Available False ReadyIngressNodes_NoReadyIngressNodes ReadyIngressNodesAvailable: Authentication require functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes and 3 master nodes (none are schedulable or ready for ingress pods).

combined with a lack of 'worker' nodes:

$ oc get -o json nodes | jq -r '.items[] | [.status.conditions[] | select(.type == "Ready")][0] as $ready | $ready.lastTransitionTime + " " + $ready.status + " " + .metadata.name + " " + (.metadata.labels | to_entries[] | select(.key | startswith("node-role.kubernetes.io/")).key| tostring)' | sort
2020-08-28T08:41:54Z True worker-0...local node-role.kubernetes.io/app
2020-08-28T09:45:56Z True worker-2...local node-role.kubernetes.io/app
2020-10-29T13:54:51Z True worker-1...local node-role.kubernetes.io/app
2020-10-30T09:59:20Z True infra-2...local node-role.kubernetes.io/app
2020-10-30T09:59:20Z True infra-2...local node-role.kubernetes.io/infra
2020-10-30T10:01:38Z True master-1...local node-role.kubernetes.io/master
2020-10-30T10:04:24Z True master-0...local node-role.kubernetes.io/master
2020-10-30T10:07:02Z True infra-1...local node-role.kubernetes.io/app
2020-10-30T10:07:02Z True infra-1...local node-role.kubernetes.io/infra
2020-10-30T10:07:27Z True master-2...local node-role.kubernetes.io/master
2020-10-30T10:10:10Z True infra-0...local node-role.kubernetes.io/app
2020-10-30T10:10:10Z True infra-0...local node-role.kubernetes.io/infra

This example cluster has nodes with "worker-..." names, but the roles are all app, infra, or master.

--- Additional comment from W. Trevor King on 2020-10-30 22:50:47 EDT ---

Took a stab at a CCX rule: https://gitlab.cee.redhat.com/ccx/ccx-rules-ocp/-/merge_requests/238

--- Additional comment from Stefan Schimanski on 2020-11-02 04:21:59 EST ---

> Auth operator minds its own business, and the ingress operator complains when it is unscheduled (bug 1881155, [3]) ;).

Half-serious: can we then assign this PR to ingress operator in order to produce good diagnostic output? Half of our reported auth issues are due to ingress not being able to serve auth info because of missing nodes.

--- Additional comment from W. Trevor King on 2020-11-02 04:43:54 EST ---

bug 1881155, linked from comment 0, is exactly that ingress/routing-assigned bug you ask for.

--- Additional comment from Lalatendu Mohanty on 2020-11-02 06:50:46 EST ---

Who is impacted?  If we have to block upgrade edges based on this issue, which edges would need blocking?
  example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet
  example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time
What is the impact?  Is it serious enough to warrant blocking edges?
  example: Up to 2 minute disruption in edge routing
  example: Up to 90seconds of API downtime
  example: etcd loses quorum and you have to restore from backup
How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)?
  example: Issue resolves itself after five minutes
  example: Admin uses oc to fix things
  example: Admin must SSH to hosts, restore from backups, or other non standard admin activities
Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)?
  example: No, it’s always been like this we just never noticed
  example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1
Comment 1 Scott Dodson 2020-11-02 16:30:32 UTC 
This is a blocker for 4.6.2.
Comment 4 Mike Fiedler 2020-11-04 02:10:12 UTC 
This failed to verify on 4.6.0-0.nightly-2020-11-03-172112

Summary: upgrading a happy 4.5.16 cluster with ingress running on nodes labeled node-role.kubernetes.io/infra resulted in an upgrade stuck with auth with this status

Status:                                                                                                                                                                                                                                                                                   
  Conditions:                                                                                                                                                                                                                                                                             
    Last Transition Time:  2020-11-03T23:50:44Z                                                                                                                                                                                                                                           
    Reason:                AsExpected                                                                                                                                                                                                                                                     
    Status:                False                                                                                                                                                                                                                                                          
    Type:                  Degraded                                                                                                                                                                                                                                                       
    Last Transition Time:  2020-11-04T00:30:43Z                                                                                                                                                                                                                                           
    Reason:                AsExpected                                                                                                                                                                                                                                                     
    Status:                False                                                                                                                                                                                                                                                          
    Type:                  Progressing                                                                                                                                                                                                                                                    
    Last Transition Time:  2020-11-04T00:29:32Z                                                                                                                                                                                                                                           
    Message:               ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).                     
    Reason:                ReadyIngressNodes_NoReadyIngressNodes                                                                                                                                                                                                                          
    Status:                False                                                                                                                                                                                                                                                          
    Type:                  Available                                                                                                                                                                                                                                                      
    Last Transition Time:  2020-11-03T22:43:43Z                                                                                                                                                                                                                                           
    Reason:                AsExpected                                                                                                                                                                                                                                                     
    Status:                True                                                                                                                                                                                                                                                           
    Type:                  Upgradeable 

I will get oc adm must-gather.

Details:

Verification steps based on https://bugzilla.redhat.com/show_bug.cgi?id=1881155#c5.  Thanks to @wking.

1. Started with a basic AWS 4.5.16 cluster with 3 nodes labeled master and 3 worker

2. before the test starts, auth and ingress are happy

2020-11-03T22:43:43Z authentication Upgradeable True AsExpected -
2020-11-03T23:00:14Z authentication Available True AsExpected -
2020-11-03T23:01:08Z authentication Progressing False AsExpected -
2020-11-03T23:49:56Z ingress Degraded False NoIngressControllersDegraded -
2020-11-03T23:50:44Z authentication Degraded False AsExpected -
2020-11-03T23:53:40Z ingress Available True - desired and current number of IngressControllers are equal
2020-11-03T23:53:40Z ingress Progressing False - desired and current number of IngressControllers are equal

3. Label all worker nodes with node-role.kubernetes.io/infra= and remove node-role.kubernetes.io/worker-

 # oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-131-118.us-east-2.compute.internal   Ready    infra    65m   v1.18.3+2fbd7c7
ip-10-0-134-15.us-east-2.compute.internal    Ready    master   74m   v1.18.3+2fbd7c7
ip-10-0-168-147.us-east-2.compute.internal   Ready    master   74m   v1.18.3+2fbd7c7
ip-10-0-187-74.us-east-2.compute.internal    Ready    infra    65m   v1.18.3+2fbd7c7
ip-10-0-201-177.us-east-2.compute.internal   Ready    master   73m   v1.18.3+2fbd7c7
ip-10-0-222-88.us-east-2.compute.internal    Ready    infra    65m   v1.18.3+2fbd7c7

Now, patch ingress to allow ingress pods to schedule on matchLabel infra

oc -n openshift-ingress-operator patch ingresscontroller default --type json -p '[{"op": "add", "path": "/spec/nodePlacement", "value": {"nodeSelector": {"matchLabels": {"node-role.kubernetes.io/infra": ""}}}}]'

4. kill all openshift-ingress pods and let them reschedule on infra nodes via matchLabels

ingress pods are now running ok with infra matchLabels

# oc get pods -n openshift-ingress 
NAME                              READY   STATUS    RESTARTS   AGE
router-default-76c5ff76db-l7wh6   1/1     Running   0          71s
router-default-76c5ff76db-wq8wz   1/1     Running   0          71s

5. on 4.5.16 both auth and ingress are happy at this point with ingress running on infra matchLabels with the patched IngressController - unlike 4.6+ auth does not go to Available=False

2020-11-03T22:43:43Z authentication Upgradeable True AsExpected -                                                                                                                                                                                                                         
2020-11-03T23:00:14Z authentication Available True AsExpected -                                                                                                                                                                                                                           
2020-11-03T23:01:08Z authentication Progressing False AsExpected -                                                                                                                                                                                                                        
2020-11-03T23:49:56Z ingress Degraded False NoIngressControllersDegraded -                                                                                                                                                                                                                
2020-11-03T23:50:44Z authentication Degraded False AsExpected -                                                                                                                                                                                                                           
2020-11-03T23:53:40Z ingress Available True - desired and current number of IngressControllers are equal                                                                                                                                                                                  
2020-11-03T23:53:40Z ingress Progressing False - desired and current number of IngressControllers are equal 

oc get co also shows everybody happy.

6. Now start the upgrade to release:4.6.0-0.nightly-2020-11-03-172112.  

Upgrade goes smooth for a while but gets stuck on auth with the status listed above.   

# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.16    True        True          107m    Unable to apply 4.6.0-0.nightly-2020-11-03-172112: the cluster operator authentication has not yet successfully rolled out

# oc get co                                                                                                                                                                                                                                                       
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE                                                                                                                                                                 
authentication                             4.6.0-0.nightly-2020-11-03-172112   False       False         False      52m                                                                                                                                                                   
cloud-credential                           4.6.0-0.nightly-2020-11-03-172112   True        False         False      167m                                                                                                                                                                  
cluster-autoscaler                         4.6.0-0.nightly-2020-11-03-172112   True        False         False      153m                                                                                                                                                                  
config-operator                            4.6.0-0.nightly-2020-11-03-172112   True        False         False      154m                                                                                                                                                                  
console                                    4.6.0-0.nightly-2020-11-03-172112   True        False         False      52m                                                                                                                                                                   
csi-snapshot-controller                    4.6.0-0.nightly-2020-11-03-172112   True        False         False      154m                                                                                                                                                                  
dns                                        4.5.16                              True        False         False      157m                                                                                                                                                                  
etcd                                       4.6.0-0.nightly-2020-11-03-172112   True        False         False      157m                                                                                                                                                                  
image-registry                             4.6.0-0.nightly-2020-11-03-172112   True        False         False      150m                                                                                                                                                                  
ingress                                    4.6.0-0.nightly-2020-11-03-172112   True        False         False      52m                                                                                                                                                                   
insights                                   4.6.0-0.nightly-2020-11-03-172112   True        False         False      154m                                                                                                                                                                  
kube-apiserver                             4.6.0-0.nightly-2020-11-03-172112   True        False         False      156m                                                                                                                                                                  
kube-controller-manager                    4.6.0-0.nightly-2020-11-03-172112   True        False         False      156m                                                                                                                                                                  
kube-scheduler                             4.6.0-0.nightly-2020-11-03-172112   True        False         False      153m                                                                                                                                                                  
kube-storage-version-migrator              4.6.0-0.nightly-2020-11-03-172112   True        False         False      150m                                                                                                                                                                  
machine-api                                4.6.0-0.nightly-2020-11-03-172112   True        False         False      151m                                                                                                                                                                  
machine-approver                           4.6.0-0.nightly-2020-11-03-172112   True        False         False      155m                                                                                                                                                                  
machine-config                             4.5.16                              True        False         False      157m                                                                                                                                                                  
marketplace                                4.6.0-0.nightly-2020-11-03-172112   True        False         False      52m                                                                                                                                                                   
monitoring                                 4.6.0-0.nightly-2020-11-03-172112   True        False         False      144m                                                                                                                                                                  
network                                    4.5.16                              True        False         False      158m                                                                                                                                                                  
node-tuning                                4.6.0-0.nightly-2020-11-03-172112   True        False         False      52m                                                                                                                                                                   
openshift-apiserver                        4.6.0-0.nightly-2020-11-03-172112   True        False         False      153m                                                                                                                                                                  
openshift-controller-manager               4.6.0-0.nightly-2020-11-03-172112   True        False         False      154m                                                                                                                                                                  
openshift-samples                          4.6.0-0.nightly-2020-11-03-172112   True        False         False      52m                                                                                                                                                                   
operator-lifecycle-manager                 4.6.0-0.nightly-2020-11-03-172112   True        False         False      157m                                                                                                                                                                  
operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-11-03-172112   True        False         False      157m                                                                                                                                                                  
operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-11-03-172112   True        False         False      51m                                                                                                                                                                   
service-ca                                 4.6.0-0.nightly-2020-11-03-172112   True        False         False      158m                                                                                                                                                                  
storage                                    4.6.0-0.nightly-2020-11-03-172112   True        False         False      52m                                                                                                                                                                   


root@ip-172-31-64-58: ~ # oc describe co auth                                                                                                                                                                                                                                             
Name:         authentication                                                                                                                                                                                                                                                              
Namespace:                                                                                                                                                                                                                                                                                
Labels:       <none>                                                                                                                                                                                                                                                                      
Annotations:  exclude.release.openshift.io/internal-openshift-hosted: true                                                                                                                                                                                                                
API Version:  config.openshift.io/v1                                                                                                                                                                                                                                                      
Kind:         ClusterOperator                                                                                                                                                                                                                                                             
Metadata:                                                                                                                                                                                                                                                                                 
  Creation Timestamp:  2020-11-03T22:35:11Z                                                                                                                                                                                                                                               
  Generation:          1                                                                                                                                                                                                                                                                  
  Managed Fields:                                                                                                                                                                                                                                                                         
    API Version:  config.openshift.io/v1                                                                                                                                                                                                                                                  
    Fields Type:  FieldsV1   
    fieldsV1:                                                                                                                                                                                                                                                                             
      f:metadata:                                                                                                                                                                                                                                                                         
        f:annotations:                                                                                                                                                                                                                                                                    
          .:                                                                                                                                                                                                                                                                              
          f:exclude.release.openshift.io/internal-openshift-hosted:                                                                                                                                                                                                                       
      f:spec:                                                                                                                                                                                                                                                                             
      f:status:                                                                                                                                                                                                                                                                           
        .:                                                                                                                                                                                                                                                                                
        f:extension:                                                                                                                                                                                                                                                                      
    Manager:      cluster-version-operator                                                                                                                                                                                                                                                
    Operation:    Update                                                                                                                                                                                                                                                                  
    Time:         2020-11-03T22:35:11Z                                                                                                                                                                                                                                                    
    API Version:  config.openshift.io/v1                                                                                                                                                                                                                                                  
    Fields Type:  FieldsV1                                                                                                                                                                                                                                                                
    fieldsV1:                                                                                                                                                                                                                                                                             
      f:status:                                                                                                                                                                                                                                                                           
        f:conditions:                                                                                                                                                                                                                                                                     
        f:relatedObjects:                                                                                                                                                                                                                                                                 
        f:versions:                                                                                                                                                                                                                                                                       
    Manager:         authentication-operator                                                                                                                                                                                                                                              
    Operation:       Update                                                                                                                                                                                                                                                               
    Time:            2020-11-04T00:31:13Z                                                                                                                                                                                                                                                 
  Resource Version:  61547
  Self Link:         /apis/config.openshift.io/v1/clusteroperators/authentication
  UID:               71de293d-2939-4b84-af17-f941dbaf82f8
Spec:
Status:
  Conditions:
    Last Transition Time:  2020-11-03T23:50:44Z
    Reason:                AsExpected
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2020-11-04T00:30:43Z
    Reason:                AsExpected
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2020-11-04T00:29:32Z
    Message:               ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).
    Reason:                ReadyIngressNodes_NoReadyIngressNodes
    Status:                False
    Type:                  Available
    Last Transition Time:  2020-11-03T22:43:43Z
    Reason:                AsExpected
    Status:                True
    Type:                  Upgradeable
  Extension:               <nil>
  Related Objects:
    Group:      operator.openshift.io
    Name:       cluster
    Resource:   authentications
    Group:      config.openshift.io
    Name:       cluster
    Resource:   authentications
    Group:      config.openshift.io
    Name:       cluster
    Resource:   infrastructures
    Group:      config.openshift.io
    Name:       cluster
    Resource:   oauths
    Group:      route.openshift.io
    Name:       oauth-openshift
    Namespace:  openshift-authentication
    Resource:   routes
    Group:      
    Name:       oauth-openshift
    Namespace:  openshift-authentication
    Resource:   services
    Group:      
    Name:       openshift-config
    Resource:   namespaces
    Group:      
    Name:       openshift-config-managed
    Resource:   namespaces
    Group:      
    Name:       openshift-authentication
    Resource:   namespaces
    Group:      
    Name:       openshift-authentication-operator
    Resource:   namespaces
    Group:      
    Name:       openshift-ingress
    Resource:   namespaces
    Group:      
    Name:       openshift-oauth-apiserver
    Resource:   namespaces
  Versions:
    Name:     operator
    Version:  4.6.0-0.nightly-2020-11-03-172112
    Name:     oauth-openshift
    Version:  4.6.0-0.nightly-2020-11-03-172112_openshift
    Name:     oauth-apiserver
    Version:  4.6.0-0.nightly-2020-11-03-172112
Events:       <none>
                                                                           
# oc get pods -n openshift-ingress
NAME                              READY   STATUS    RESTARTS   AGE
router-default-5995d6bbbf-2tqxk   1/1     Running   0          89m
router-default-5995d6bbbf-dwfwr   1/1     Running   0          89m
Comment 6 Xingxing Xia 2020-11-04 07:31:18 UTC 
Thanks Mike used upgrade to test it. From bug 1881155#c5, the reproducer can be seen in fresh env without doing upgrade. So I launched a fresh 4.6.0-0.nightly-2020-11-03-172112 env, then followed bug 1881155#c5 steps, I can also reproduce Mike's error:
$ oc get node
NAME                                              STATUS   ROLES    AGE     VERSION
ip-10-0-129-211.ap-southeast-1.compute.internal   Ready    master   4h24m   v1.19.0+9f84db3
ip-10-0-135-171.ap-southeast-1.compute.internal   Ready    infra    4h13m   v1.19.0+9f84db3
ip-10-0-184-212.ap-southeast-1.compute.internal   Ready    infra    4h13m   v1.19.0+9f84db3
ip-10-0-186-226.ap-southeast-1.compute.internal   Ready    master   4h24m   v1.19.0+9f84db3
ip-10-0-206-99.ap-southeast-1.compute.internal    Ready    infra    4h13m   v1.19.0+9f84db3
ip-10-0-207-229.ap-southeast-1.compute.internal   Ready    master   4h25m   v1.19.0+9f84db3
$ oc get pods -n openshift-ingress -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE                                              NOMINATED NODE   READINESS GATES
router-default-7d6df7cf44-f9ffg   1/1     Running   0          10m   10.129.2.79   ip-10-0-206-99.ap-southeast-1.compute.internal    <none>           <none>
router-default-7d6df7cf44-gf8f9   1/1     Running   0          10m   10.128.2.17   ip-10-0-184-212.ap-southeast-1.compute.internal   <none>           <none>
$ oc get co | grep -v "4.6.0-0.nightly-2020-11-03-172112.*T.*F.*F"
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.nightly-2020-11-03-172112   False       False         False      13m
$ oc describe co authentication
Name:         authentication
...
    Message:               ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).
    Reason:                ReadyIngressNodes_NoReadyIngressNodes
    Status:                False
    Type:                  Available
...
I guess the PR has some problem in function numberOfCustomIngressTargets which wrongly got 0 custom target nodes.
Comment 7 Scott Dodson 2020-11-04 17:53:43 UTC 
I'm dropping the blocker flag because this continues to fail in CI. WE have numerous fixes already in 4.6.2 that we do not wish to delay any longer. We'll just have to deal with this failing upgrades until 4.6.3.
Comment 9 Xingxing Xia 2020-11-05 10:23:38 UTC 
Verified in 4.6.0-0.nightly-2020-11-05-024238 using the steps of bug 1893386#c14. Everything is fine. The mad co/authentication is not reproduced.
Comment 12 errata-xmlrpc 2020-11-16 14:37:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.4 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4987
Comment 13 W. Trevor King 2021-04-05 17:47:44 UTC 
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475