1914892 – Upgrading to 4.6.10 nightly failed with RHEL worker nodes: Failed to find /dev/disk/by-label/root

Bug 1914892 - Upgrading to 4.6.10 nightly failed with RHEL worker nodes: Failed to find /dev/disk/by-label/root

Summary: Upgrading to 4.6.10 nightly failed with RHEL worker nodes: Failed to find /de...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Machine Config Operator
Sub Component:
Version:	4.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.6.z
Assignee:	Yu Qi Zhang
QA Contact:	Michael Nguyen
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	Partner1908842 (view as bug list)
Depends On:	1913154
Blocks:
TreeView+	depends on / blocked

Reported:	2021-01-11 12:38 UTC by OpenShift BugZilla Robot
Modified:	2021-04-05 17:47 UTC (History)
CC List:	14 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-01-25 20:02:18 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift machine-config-operator pull 2330	0	None	closed	[release-4.6] Bug 1914892: update.go: only set BFQ scheduler for masters	2021-02-08 14:49:24 UTC
Red Hat Product Errata	RHSA-2021:0171	0	None	None	None	2021-01-25 20:02:27 UTC

Comment 4 Jianwei Hou 2021-01-19 10:00:28 UTC

Verified in 4.6.0-0.nightly-2021-01-18-070340. We had run two upgrades covering upgrades from 4.6.12 and from 4.5.25, both upgrades successfully upgraded RHCOS and RHEL workers. The issue no longer persists.

01-19 16:41:44  --------------upgrade succ(time cost 54 minutes), Cluster version is 4.6.0-0.nightly-2021-01-18-070340---------------
01-19 16:41:44  
01-19 16:41:44  
01-19 16:41:44  sleep if node notready before do next node check.....
01-19 16:41:44  
01-19 16:41:44  [2021-01-19,08:41:32]
01-19 16:41:44  NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
01-19 16:41:44  master   rendered-master-0d6e76293bba10240ebb4d089e03ff55   True      False      False      3              3                   3                     0                      3h14m
01-19 16:41:44  worker   rendered-worker-163e41816932cd1e67bdbf0988031104   True      False      False      5              5                   5                     0                      3h14m
01-19 16:41:44  exit from upgrade loop
01-19 16:41:44  
01-19 16:41:44  **************Post Action after upgrade succ****************
01-19 16:41:44  
01-19 16:41:44  Post action: #oc get node: NAME                                                         STATUS   ROLES    AGE     VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
01-19 16:41:44  miyadav191308-mxn66-master-0.c.openshift-qe.internal         Ready    master   3h15m   v1.19.0+3b01205   10.0.0.5      <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-master-1.c.openshift-qe.internal         Ready    master   3h15m   v1.19.0+3b01205   10.0.0.4      <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-master-2.c.openshift-qe.internal         Ready    master   3h15m   v1.19.0+3b01205   10.0.0.6      <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-w-a-l-rhel-0                             Ready    worker   116m    v1.19.0+3b01205   10.0.32.5     <none>        Red Hat Enterprise Linux Server 7.9 (Maipo)                    3.10.0-1160.11.1.el7.x86_64    cri-o://1.19.0-118.rhaos4.6.gitf51f94a.el7
01-19 16:41:44  miyadav191308-mxn66-w-a-l-rhel-1                             Ready    worker   116m    v1.19.0+3b01205   10.0.32.6     <none>        Red Hat Enterprise Linux Server 7.9 (Maipo)                    3.10.0-1160.11.1.el7.x86_64    cri-o://1.19.0-118.rhaos4.6.gitf51f94a.el7
01-19 16:41:44  miyadav191308-mxn66-worker-a-9rwb2.c.openshift-qe.internal   Ready    worker   3h3m    v1.19.0+3b01205   10.0.32.2     <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-worker-b-6s4wm.c.openshift-qe.internal   Ready    worker   3h3m    v1.19.0+3b01205   10.0.32.3     <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-worker-c-tqknn.c.openshift-qe.internal   Ready    worker   3h3m    v1.19.0+3b01205   10.0.32.4     <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8


01-19 16:41:44  Post action: #oc get co:NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
01-19 16:41:44  authentication                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      11m
01-19 16:41:44  cloud-credential                           4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h16m
01-19 16:41:44  cluster-autoscaler                         4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h13m
01-19 16:41:44  config-operator                            4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m
01-19 16:41:44  console                                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      29m
01-19 16:41:44  csi-snapshot-controller                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      29m
01-19 16:41:44  dns                                        4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h12m
01-19 16:41:44  etcd                                       4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h12m
01-19 16:41:44  image-registry                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h3m
01-19 16:41:44  ingress                                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h2m
01-19 16:41:44  insights                                   4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m
01-19 16:41:44  kube-apiserver                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h11m
01-19 16:41:44  kube-controller-manager                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h11m
01-19 16:41:44  kube-scheduler                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h10m
01-19 16:41:44  kube-storage-version-migrator              4.6.0-0.nightly-2021-01-18-070340   True        False         False      31m
01-19 16:41:44  machine-api                                4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h4m
01-19 16:41:44  machine-approver                           4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h13m
01-19 16:41:44  machine-config                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      25m
01-19 16:41:44  marketplace                                4.6.0-0.nightly-2021-01-18-070340   True        False         False      28m
01-19 16:41:44  monitoring                                 4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h1m
01-19 16:41:44  network                                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m
01-19 16:41:44  node-tuning                                4.6.0-0.nightly-2021-01-18-070340   True        False         False      55m
01-19 16:41:44  openshift-apiserver                        4.6.0-0.nightly-2021-01-18-070340   True        False         False      27m
01-19 16:41:44  openshift-controller-manager               4.6.0-0.nightly-2021-01-18-070340   True        False         False      54m
01-19 16:41:44  openshift-samples                          4.6.0-0.nightly-2021-01-18-070340   True        False         False      55m
01-19 16:41:44  operator-lifecycle-manager                 4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h12m
01-19 16:41:44  operator-lifecycle-manager-catalog         4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h13m
01-19 16:41:44  operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2021-01-18-070340   True        False         False      29m
01-19 16:41:44  service-ca                                 4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m
01-19 16:41:44  storage                                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m

Comment 6 errata-xmlrpc 2021-01-25 20:02:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6.13 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0171

Comment 7 rvanderp 2021-01-28 17:43:24 UTC

*** PartnerBug 1908842 has been marked as a duplicate of this bug. ***

Comment 8 W. Trevor King 2021-04-05 17:47:16 UTC

Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475

Note You need to log in before you can comment on or make changes to this bug.