1792139 – RHEL7 worker nodes may go to NotReady,SchedulingDisabled while upgrading from 4.2.12 to 4.3.0

Bug 1792139 - RHEL7 worker nodes may go to NotReady,SchedulingDisabled while upgrading from 4.2.12 to 4.3.0

Summary: RHEL7 worker nodes may go to NotReady,SchedulingDisabled while upgrading fro...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Installer
Sub Component:
Version:	4.3.0
Hardware:	Unspecified
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.4.0
Assignee:	Russell Teague
QA Contact:	Gaoyun Pei
Docs Contact:
URL:
Whiteboard:
Depends On:	1786274
Blocks:
TreeView+	depends on / blocked

Reported:	2020-01-17 06:56 UTC by Gaoyun Pei
Modified:	2020-05-04 11:25 UTC (History)
CC List:	16 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: Machine config was not properly updated by MCO because package installs updated files on disk Consequence: MCO would not process config updates on RHEL nodes Fix: Added machine config apply back to upgrade steps and added proxy config for image pulls Result: Machine configs were properly applied after package updates during upgrade.
Clone Of:	1786274
Environment:
Last Closed:	2020-05-04 11:24:47 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift openshift-ansible pull 12069	0	None	closed	Bug 1793078: Fix machine config apply on upgrade	2020-11-12 10:20:20 UTC
Red Hat Product Errata	RHBA-2020:0581	0	None	None	None	2020-05-04 11:25:19 UTC

Comment 2 Scott Dodson 2020-01-30 20:02:43 UTC

https://github.com/openshift/openshift-ansible/pull/12069 fixed application of machine config in such a way that the workflow described in docs should now be successful

Comment 4 Gaoyun Pei 2020-02-07 12:25:17 UTC

During the upgrade of a 4.3.1 cluster(with rhcos&rhel worker) to 4.4, no RHEL worker would be stuck at NotReady,SchedulingDisabled, which is different from when doing so in "4.2 to 4.3".

# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-07-012035   True        False         4m27s   Cluster version is 4.4.0-0.nightly-2020-02-07-012035

# oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.0-0.nightly-2020-02-07-012035   True        False         False      3h55m
cloud-credential                           4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h23m
cluster-autoscaler                         4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h12m
console                                    4.4.0-0.nightly-2020-02-07-012035   True        False         False      33m
csi-snapshot-controller                    4.4.0-0.nightly-2020-02-07-012035   True        False         False      3m56s
dns                                        4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h16m
etcd                                       4.4.0-0.nightly-2020-02-07-012035   True        False         False      19m
image-registry                             4.4.0-0.nightly-2020-02-07-012035   True        False         False      7m40s
ingress                                    4.4.0-0.nightly-2020-02-07-012035   True        False         False      3m44s
insights                                   4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h18m
kube-apiserver                             4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h15m
kube-controller-manager                    4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h16m
kube-scheduler                             4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h16m
kube-storage-version-migrator              4.4.0-0.nightly-2020-02-07-012035   True        False         False      7m46s
machine-api                                4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h17m
machine-config                             4.4.0-0.nightly-2020-02-07-012035   True        False         False      21m
marketplace                                4.4.0-0.nightly-2020-02-07-012035   True        False         False      23m
monitoring                                 4.4.0-0.nightly-2020-02-07-012035   True        False         False      8m11s
network                                    4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h18m
node-tuning                                4.4.0-0.nightly-2020-02-07-012035   True        False         False      3h27m
openshift-apiserver                        4.4.0-0.nightly-2020-02-07-012035   True        False         False      23m
openshift-controller-manager               4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h16m
openshift-samples                          4.4.0-0.nightly-2020-02-07-012035   True        False         False      3h27m
operator-lifecycle-manager                 4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h17m
operator-lifecycle-manager-catalog         4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h17m
operator-lifecycle-manager-packageserver   4.4.0-0.nightly-2020-02-07-012035   True        False         False      33m
service-ca                                 4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h18m
service-catalog-apiserver                  4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h15m
service-catalog-controller-manager         4.4.0-0.nightly-2020-02-07-012035   True        False         False      4h15m
storage                                    4.4.0-0.nightly-2020-02-07-012035   True        False         False      3h27m

# oc get clusterversion -o json|jq -r '.items[0].status.history[]|.startedTime + "|" + .completionTime + "|" + .state + "|" + .version'
2020-02-07T08:36:05Z|2020-02-07T12:09:47Z|Completed|4.4.0-0.nightly-2020-02-07-012035
2020-02-07T07:50:36Z|2020-02-07T08:18:35Z|Completed|4.3.1


After cluster upgrade finished, run upgrade playbook against all the RHEL workers. 
The whole cluster was upgrade to an expected status in the end.

# oc get node -o wide
NAME                                        STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                CONTAINER-RUNTIME
ip-10-0-49-70.us-east-2.compute.internal    Ready    worker   3h44m   v1.17.1   10.0.49.70    <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)                    3.10.0-1062.12.1.el7.x86_64   cri-o://1.17.0-0.4.rc1.rhaos4.4.git5842752.el7-rc1
ip-10-0-51-166.us-east-2.compute.internal   Ready    worker   4h11m   v1.17.1   10.0.51.166   <none>        Red Hat Enterprise Linux CoreOS 44.81.202002061902-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.16.2-15.dev.rhaos4.3.gita83f883.el8
ip-10-0-57-164.us-east-2.compute.internal   Ready    master   4h23m   v1.17.1   10.0.57.164   <none>        Red Hat Enterprise Linux CoreOS 44.81.202002061902-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.16.2-15.dev.rhaos4.3.gita83f883.el8
ip-10-0-57-166.us-east-2.compute.internal   Ready    master   4h23m   v1.17.1   10.0.57.166   <none>        Red Hat Enterprise Linux CoreOS 44.81.202002061902-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.16.2-15.dev.rhaos4.3.gita83f883.el8
ip-10-0-59-5.us-east-2.compute.internal     Ready    worker   3h44m   v1.17.1   10.0.59.5     <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)                    3.10.0-1062.12.1.el7.x86_64   cri-o://1.17.0-0.4.rc1.rhaos4.4.git5842752.el7-rc1
ip-10-0-67-153.us-east-2.compute.internal   Ready    master   4h23m   v1.17.1   10.0.67.153   <none>        Red Hat Enterprise Linux CoreOS 44.81.202002061902-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.16.2-15.dev.rhaos4.3.gita83f883.el8
ip-10-0-69-163.us-east-2.compute.internal   Ready    worker   4h10m   v1.17.1   10.0.69.163   <none>        Red Hat Enterprise Linux CoreOS 44.81.202002061902-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.16.2-15.dev.rhaos4.3.gita83f883.el8

Mark this bug as verified in openshift-ansible-4.4.0-202002070656.git.178.3e1c275.el7.noarch.rpm.

Comment 6 errata-xmlrpc 2020-05-04 11:24:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581

Note You need to log in before you can comment on or make changes to this bug.