Bug 1914892 - Upgrading to 4.6.10 nightly failed with RHEL worker nodes: Failed to find /dev/disk/by-label/root
Summary: Upgrading to 4.6.10 nightly failed with RHEL worker nodes: Failed to find /de...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.z
Assignee: Yu Qi Zhang
QA Contact: Michael Nguyen
URL:
Whiteboard:
: 1908842 (view as bug list)
Depends On: 1913154
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-11 12:38 UTC by OpenShift BugZilla Robot
Modified: 2021-04-05 17:47 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-25 20:02:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2330 0 None closed [release-4.6] Bug 1914892: update.go: only set BFQ scheduler for masters 2021-02-08 14:49:24 UTC
Red Hat Product Errata RHSA-2021:0171 0 None None None 2021-01-25 20:02:27 UTC

Comment 4 Jianwei Hou 2021-01-19 10:00:28 UTC
Verified in 4.6.0-0.nightly-2021-01-18-070340. We had run two upgrades covering upgrades from 4.6.12 and from 4.5.25, both upgrades successfully upgraded RHCOS and RHEL workers. The issue no longer persists.

01-19 16:41:44  --------------upgrade succ(time cost 54 minutes), Cluster version is 4.6.0-0.nightly-2021-01-18-070340---------------
01-19 16:41:44  
01-19 16:41:44  
01-19 16:41:44  sleep if node notready before do next node check.....
01-19 16:41:44  
01-19 16:41:44  [2021-01-19,08:41:32]
01-19 16:41:44  NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
01-19 16:41:44  master   rendered-master-0d6e76293bba10240ebb4d089e03ff55   True      False      False      3              3                   3                     0                      3h14m
01-19 16:41:44  worker   rendered-worker-163e41816932cd1e67bdbf0988031104   True      False      False      5              5                   5                     0                      3h14m
01-19 16:41:44  exit from upgrade loop
01-19 16:41:44  
01-19 16:41:44  **************Post Action after upgrade succ****************
01-19 16:41:44  
01-19 16:41:44  Post action: #oc get node: NAME                                                         STATUS   ROLES    AGE     VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
01-19 16:41:44  miyadav191308-mxn66-master-0.c.openshift-qe.internal         Ready    master   3h15m   v1.19.0+3b01205   10.0.0.5      <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-master-1.c.openshift-qe.internal         Ready    master   3h15m   v1.19.0+3b01205   10.0.0.4      <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-master-2.c.openshift-qe.internal         Ready    master   3h15m   v1.19.0+3b01205   10.0.0.6      <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-w-a-l-rhel-0                             Ready    worker   116m    v1.19.0+3b01205   10.0.32.5     <none>        Red Hat Enterprise Linux Server 7.9 (Maipo)                    3.10.0-1160.11.1.el7.x86_64    cri-o://1.19.0-118.rhaos4.6.gitf51f94a.el7
01-19 16:41:44  miyadav191308-mxn66-w-a-l-rhel-1                             Ready    worker   116m    v1.19.0+3b01205   10.0.32.6     <none>        Red Hat Enterprise Linux Server 7.9 (Maipo)                    3.10.0-1160.11.1.el7.x86_64    cri-o://1.19.0-118.rhaos4.6.gitf51f94a.el7
01-19 16:41:44  miyadav191308-mxn66-worker-a-9rwb2.c.openshift-qe.internal   Ready    worker   3h3m    v1.19.0+3b01205   10.0.32.2     <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-worker-b-6s4wm.c.openshift-qe.internal   Ready    worker   3h3m    v1.19.0+3b01205   10.0.32.3     <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8
01-19 16:41:44  miyadav191308-mxn66-worker-c-tqknn.c.openshift-qe.internal   Ready    worker   3h3m    v1.19.0+3b01205   10.0.32.4     <none>        Red Hat Enterprise Linux CoreOS 46.82.202101161820-0 (Ootpa)   4.18.0-193.40.1.el8_2.x86_64   cri-o://1.19.1-3.rhaos4.6.gitc9c3658.el8


01-19 16:41:44  Post action: #oc get co:NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
01-19 16:41:44  authentication                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      11m
01-19 16:41:44  cloud-credential                           4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h16m
01-19 16:41:44  cluster-autoscaler                         4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h13m
01-19 16:41:44  config-operator                            4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m
01-19 16:41:44  console                                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      29m
01-19 16:41:44  csi-snapshot-controller                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      29m
01-19 16:41:44  dns                                        4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h12m
01-19 16:41:44  etcd                                       4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h12m
01-19 16:41:44  image-registry                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h3m
01-19 16:41:44  ingress                                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h2m
01-19 16:41:44  insights                                   4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m
01-19 16:41:44  kube-apiserver                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h11m
01-19 16:41:44  kube-controller-manager                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h11m
01-19 16:41:44  kube-scheduler                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h10m
01-19 16:41:44  kube-storage-version-migrator              4.6.0-0.nightly-2021-01-18-070340   True        False         False      31m
01-19 16:41:44  machine-api                                4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h4m
01-19 16:41:44  machine-approver                           4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h13m
01-19 16:41:44  machine-config                             4.6.0-0.nightly-2021-01-18-070340   True        False         False      25m
01-19 16:41:44  marketplace                                4.6.0-0.nightly-2021-01-18-070340   True        False         False      28m
01-19 16:41:44  monitoring                                 4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h1m
01-19 16:41:44  network                                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m
01-19 16:41:44  node-tuning                                4.6.0-0.nightly-2021-01-18-070340   True        False         False      55m
01-19 16:41:44  openshift-apiserver                        4.6.0-0.nightly-2021-01-18-070340   True        False         False      27m
01-19 16:41:44  openshift-controller-manager               4.6.0-0.nightly-2021-01-18-070340   True        False         False      54m
01-19 16:41:44  openshift-samples                          4.6.0-0.nightly-2021-01-18-070340   True        False         False      55m
01-19 16:41:44  operator-lifecycle-manager                 4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h12m
01-19 16:41:44  operator-lifecycle-manager-catalog         4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h13m
01-19 16:41:44  operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2021-01-18-070340   True        False         False      29m
01-19 16:41:44  service-ca                                 4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m
01-19 16:41:44  storage                                    4.6.0-0.nightly-2021-01-18-070340   True        False         False      3h14m

Comment 6 errata-xmlrpc 2021-01-25 20:02:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6.13 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0171

Comment 7 rvanderp 2021-01-28 17:43:24 UTC
*** Bug 1908842 has been marked as a duplicate of this bug. ***

Comment 8 W. Trevor King 2021-04-05 17:47:16 UTC
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475


Note You need to log in before you can comment on or make changes to this bug.