I still see the issue with 4.7.0-0.nightly-2021-01-06-222035, however in this instance the error I see is "error enabling unit: Failed to execute operation: File exists\n"

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.9     True        True          64m     Working towards 4.7.0-0.nightly-2021-01-06-222035: 84% complete


$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.7.0-0.nightly-2021-01-06-222035   True        False         False      2m33s
baremetal                                  4.7.0-0.nightly-2021-01-06-222035   True        False         False      41m
cloud-credential                           4.7.0-0.nightly-2021-01-06-222035   True        False         False      126m
cluster-autoscaler                         4.7.0-0.nightly-2021-01-06-222035   True        False         False      122m
config-operator                            4.7.0-0.nightly-2021-01-06-222035   True        False         False      123m
console                                    4.7.0-0.nightly-2021-01-06-222035   True        False         False      7m31s
csi-snapshot-controller                    4.7.0-0.nightly-2021-01-06-222035   True        False         False      7m26s
dns                                        4.7.0-0.nightly-2021-01-06-222035   True        False         False      121m
etcd                                       4.7.0-0.nightly-2021-01-06-222035   True        False         False      121m
image-registry                             4.7.0-0.nightly-2021-01-06-222035   True        False         False      113m
ingress                                    4.7.0-0.nightly-2021-01-06-222035   True        False         False      113m
insights                                   4.7.0-0.nightly-2021-01-06-222035   True        False         False      123m
kube-apiserver                             4.7.0-0.nightly-2021-01-06-222035   True        False         False      120m
kube-controller-manager                    4.7.0-0.nightly-2021-01-06-222035   True        False         False      120m
kube-scheduler                             4.7.0-0.nightly-2021-01-06-222035   True        False         False      120m
kube-storage-version-migrator              4.7.0-0.nightly-2021-01-06-222035   True        False         False      12m
machine-api                                4.7.0-0.nightly-2021-01-06-222035   True        False         False      119m
machine-approver                           4.7.0-0.nightly-2021-01-06-222035   True        False         False      122m
machine-config                             4.6.9                               False       True          True       25m
marketplace                                4.7.0-0.nightly-2021-01-06-222035   True        False         False      13m
monitoring                                 4.7.0-0.nightly-2021-01-06-222035   True        False         False      112m
network                                    4.7.0-0.nightly-2021-01-06-222035   True        False         False      28m
node-tuning                                4.7.0-0.nightly-2021-01-06-222035   True        False         False      38m
openshift-apiserver                        4.7.0-0.nightly-2021-01-06-222035   True        False         False      5m20s
openshift-controller-manager               4.7.0-0.nightly-2021-01-06-222035   True        False         False      121m
openshift-samples                          4.7.0-0.nightly-2021-01-06-222035   True        False         False      38m
operator-lifecycle-manager                 4.7.0-0.nightly-2021-01-06-222035   True        False         False      122m
operator-lifecycle-manager-catalog         4.7.0-0.nightly-2021-01-06-222035   True        False         False      122m
operator-lifecycle-manager-packageserver   4.7.0-0.nightly-2021-01-06-222035   True        False         False      7m6s
service-ca                                 4.7.0-0.nightly-2021-01-06-222035   True        False         False      123m
storage                                    4.7.0-0.nightly-2021-01-06-222035   True        False         False      6m33s


$ oc get nodes -o wide
NAME                                        STATUS                     ROLES    AGE    VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-52-165.us-east-2.compute.internal   Ready                      master   126m   v1.20.0+b1e9f0d   10.0.52.165   <none>        Red Hat Enterprise Linux CoreOS 47.83.202101060443-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitd388528.el8.39
ip-10-0-53-183.us-east-2.compute.internal   Ready                      worker   67m    v1.19.0+9c69bdc   10.0.53.183   <none>        Red Hat Enterprise Linux Server 7.9 (Maipo)                    3.10.0-1160.11.1.el7.x86_64    cri-o://1.19.0-118.rhaos4.6.gitf51f94a.el7
ip-10-0-60-33.us-east-2.compute.internal    Ready,SchedulingDisabled   worker   67m    v1.19.0+9c69bdc   10.0.60.33    <none>        Red Hat Enterprise Linux Server 7.9 (Maipo)                    3.10.0-1160.11.1.el7.x86_64    cri-o://1.19.0-118.rhaos4.6.gitf51f94a.el7
ip-10-0-63-181.us-east-2.compute.internal   Ready                      master   126m   v1.20.0+b1e9f0d   10.0.63.181   <none>        Red Hat Enterprise Linux CoreOS 47.83.202101060443-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitd388528.el8.39
ip-10-0-74-41.us-east-2.compute.internal    Ready                      master   126m   v1.20.0+b1e9f0d   10.0.74.41    <none>        Red Hat Enterprise Linux CoreOS 47.83.202101060443-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitd388528.el8.39


$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-15480014538085b4c551d98d493e248d   True      False      False      3              3                   3                     0                      124m
worker   rendered-worker-8057ddcdd5ed8f83f444d8ea4f9963c4   False     True       True       2              0                   0                     1                      124m


$ oc describe mcp worker
Name:         worker
Namespace:    
Labels:       machineconfiguration.openshift.io/mco-built-in=
              pools.operator.machineconfiguration.openshift.io/worker=
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1
Kind:         MachineConfigPool
Metadata:
  Creation Timestamp:  2021-01-07T09:32:57Z
  Generation:          5
  Managed Fields:
    API Version:  machineconfiguration.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:machineconfiguration.openshift.io/mco-built-in:
          f:pools.operator.machineconfiguration.openshift.io/worker:
      f:spec:
        .:
        f:configuration:
        f:machineConfigSelector:
          .:
          f:matchLabels:
            .:
            f:machineconfiguration.openshift.io/role:
        f:nodeSelector:
          .:
          f:matchLabels:
            .:
            f:node-role.kubernetes.io/worker:
        f:paused:
    Manager:      machine-config-operator
    Operation:    Update
    Time:         2021-01-07T09:32:57Z
    API Version:  machineconfiguration.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:configuration:
          f:name:
          f:source:
      f:status:
        .:
        f:conditions:
        f:configuration:
          .:
          f:name:
          f:source:
        f:degradedMachineCount:
        f:machineCount:
        f:observedGeneration:
        f:readyMachineCount:
        f:unavailableMachineCount:
        f:updatedMachineCount:
    Manager:         machine-config-controller
    Operation:       Update
    Time:            2021-01-07T10:32:30Z
  Resource Version:  74388
  UID:               42d6af38-c165-46a3-8d1d-9c86e156411b
Spec:
  Configuration:
    Name:  rendered-worker-3b8eb936c7f50c4d1664fb66186e51fc
    Source:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         00-worker
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-container-runtime
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-kubelet
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-fips
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-generated-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-ssh
  Machine Config Selector:
    Match Labels:
      machineconfiguration.openshift.io/role:  worker
  Node Selector:
    Match Labels:
      node-role.kubernetes.io/worker:  
  Paused:                              false
Status:
  Conditions:
    Last Transition Time:  2021-01-07T09:35:17Z
    Message:               
    Reason:                
    Status:                False
    Type:                  RenderDegraded
    Last Transition Time:  2021-01-07T11:16:38Z
    Message:               
    Reason:                
    Status:                False
    Type:                  Updated
    Last Transition Time:  2021-01-07T11:16:38Z
    Message:               All nodes are updating to rendered-worker-3b8eb936c7f50c4d1664fb66186e51fc
    Reason:                
    Status:                True
    Type:                  Updating
    Last Transition Time:  2021-01-07T11:18:48Z
    Message:               Node ip-10-0-60-33.us-east-2.compute.internal is reporting: "error enabling unit: Failed to execute operation: File exists\n"
    Reason:                1 nodes are reporting degraded status on sync
    Status:                True
    Type:                  NodeDegraded
    Last Transition Time:  2021-01-07T11:18:48Z
    Message:               
    Reason:                
    Status:                True
    Type:                  Degraded
  Configuration:
    Name:  rendered-worker-8057ddcdd5ed8f83f444d8ea4f9963c4
    Source:
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   00-worker
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-worker-container-runtime
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-worker-kubelet
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-fips
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-generated-registries
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-ssh
  Degraded Machine Count:     1
  Machine Count:              2
  Observed Generation:        5
  Ready Machine Count:        0
  Unavailable Machine Count:  1
  Updated Machine Count:      0
Events:
  Type    Reason            Age   From                                    Message
  ----    ------            ----  ----                                    -------
  Normal  SetDesiredConfig  97m   machineconfigcontroller-nodecontroller  Targeted node ip-10-0-63-136.us-east-2.compute.internal to config rendered-worker-8057ddcdd5ed8f83f444d8ea4f9963c4
  Normal  SetDesiredConfig  94m   machineconfigcontroller-nodecontroller  Targeted node ip-10-0-71-35.us-east-2.compute.internal to config rendered-worker-8057ddcdd5ed8f83f444d8ea4f9963c4
  Normal  SetDesiredConfig  92m   machineconfigcontroller-nodecontroller  Targeted node ip-10-0-53-113.us-east-2.compute.internal to config rendered-worker-8057ddcdd5ed8f83f444d8ea4f9963c4
  Normal  SetDesiredConfig  20m   machineconfigcontroller-nodecontroller  Targeted node ip-10-0-60-33.us-east-2.compute.internal to config rendered-worker-3b8eb936c7f50c4d1664fb66186e51fc


$ oc describe node ip-10-0-60-33.us-east-2.compute.internal
Name:               ip-10-0-60-33.us-east-2.compute.internal
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m4.xlarge
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=us-east-2
                    failure-domain.beta.kubernetes.io/zone=us-east-2a
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-0-60-33.us-east-2.compute.internal
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/worker=
                    node.kubernetes.io/instance-type=m4.xlarge
                    node.openshift.io/os_id=rhel
                    topology.ebs.csi.aws.com/zone=us-east-2a
                    topology.kubernetes.io/region=us-east-2
                    topology.kubernetes.io/zone=us-east-2a
Annotations:        csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-06d3f727f99a1d2da"}
                    k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"shared","interface-id":"br-ex_ip-10-0-60-33.us-east-2.compute.internal","mac-address":"02:20:5c:5f:8e:92","ip-addresse...
                    k8s.ovn.org/node-chassis-id: 80072e11-3efc-4667-9f3f-63f3d8a2282a
                    k8s.ovn.org/node-local-nat-ip: {"default":["169.254.10.233"]}
                    k8s.ovn.org/node-mgmt-port-mac-address: 02:ef:71:ca:a0:b9
                    k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.0.60.33/20"}
                    k8s.ovn.org/node-subnets: {"default":"10.131.2.0/23"}
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-8057ddcdd5ed8f83f444d8ea4f9963c4
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-3b8eb936c7f50c4d1664fb66186e51fc
                    machineconfiguration.openshift.io/reason: error enabling unit: Failed to execute operation: File exists
                    machineconfiguration.openshift.io/ssh: accessed
                    machineconfiguration.openshift.io/state: Degraded
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 07 Jan 2021 15:59:02 +0530
Taints:             node.kubernetes.io/unschedulable:NoSchedule
Unschedulable:      true
Lease:
  HolderIdentity:  ip-10-0-60-33.us-east-2.compute.internal
  AcquireTime:     <unset>
  RenewTime:       Thu, 07 Jan 2021 17:07:14 +0530
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 07 Jan 2021 17:05:04 +0530   Thu, 07 Jan 2021 15:59:02 +0530   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 07 Jan 2021 17:05:04 +0530   Thu, 07 Jan 2021 15:59:02 +0530   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 07 Jan 2021 17:05:04 +0530   Thu, 07 Jan 2021 15:59:02 +0530   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 07 Jan 2021 17:05:04 +0530   Thu, 07 Jan 2021 15:59:52 +0530   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   10.0.60.33
  Hostname:     ip-10-0-60-33.us-east-2.compute.internal
  InternalDNS:  ip-10-0-60-33.us-east-2.compute.internal
Capacity:
  attachable-volumes-aws-ebs:  39
  cpu:                         4
  ephemeral-storage:           31444972Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      16264968Ki
  pods:                        250
Allocatable:
  attachable-volumes-aws-ebs:  39
  cpu:                         3500m
  ephemeral-storage:           27905944324
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      15113992Ki
  pods:                        250
System Info:
  Machine ID:                             9a1c7f6b38b4416bb786db538b6ff55a
  System UUID:                            EC2BE0A1-C9B2-442F-0610-ED4DD17F8AB7
  Boot ID:                                37e728b8-9ae2-4ea6-a7da-4e4ed61e5e1e
  Kernel Version:                         3.10.0-1160.11.1.el7.x86_64
  OS Image:                               Red Hat Enterprise Linux Server 7.9 (Maipo)
  Operating System:                       linux
  Architecture:                           amd64
  Container Runtime Version:              cri-o://1.19.0-118.rhaos4.6.gitf51f94a.el7
  Kubelet Version:                        v1.19.0+9c69bdc
  Kube-Proxy Version:                     v1.19.0+9c69bdc
ProviderID:                               aws:///us-east-2a/i-06d3f727f99a1d2da
Non-terminated Pods:                      (12 in total)
  Namespace                               Name                             CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                               ----                             ------------  ----------  ---------------  -------------  ---
  openshift-cluster-csi-drivers           aws-ebs-csi-driver-node-jpr6b    30m (0%)      0 (0%)      150Mi (1%)       0 (0%)         38m
  openshift-cluster-node-tuning-operator  tuned-zww9c                      10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         38m
  openshift-dns                           dns-default-j47hl                65m (1%)      0 (0%)      131Mi (0%)       0 (0%)         24m
  openshift-image-registry                node-ca-m2jps                    10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         38m
  openshift-ingress-canary                ingress-canary-44gwr             10m (0%)      0 (0%)      20Mi (0%)        0 (0%)         39m
  openshift-machine-config-operator       machine-config-daemon-g8tcl      40m (1%)      0 (0%)      100Mi (0%)       0 (0%)         23m
  openshift-monitoring                    node-exporter-fd95q              9m (0%)       0 (0%)      210Mi (1%)       0 (0%)         39m
  openshift-multus                        multus-8pgxt                     10m (0%)      0 (0%)      150Mi (1%)       0 (0%)         32m
  openshift-multus                        network-metrics-daemon-6vqj9     20m (0%)      0 (0%)      120Mi (0%)       0 (0%)         33m
  openshift-network-diagnostics           network-check-target-mzzdv       10m (0%)      0 (0%)      150Mi (1%)       0 (0%)         34m
  openshift-ovn-kubernetes                ovnkube-node-vs6mv               30m (0%)      0 (0%)      620Mi (4%)       0 (0%)         34m
  openshift-ovn-kubernetes                ovs-node-46vsh                   100m (2%)     0 (0%)      300Mi (2%)       0 (0%)         33m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         344m (9%)     0 (0%)
  memory                      2011Mi (13%)  0 (0%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0
Events:
  Type    Reason              Age   From                                               Message
  ----    ------              ----  ----                                               -------
  Normal  NodeNotSchedulable  20m   kubelet, ip-10-0-60-33.us-east-2.compute.internal  Node ip-10-0-60-33.us-east-2.compute.internal status is now: NodeNotSchedulable

To add to my previous comment, the error I see is reported in bug https://bugzilla.redhat.com/show_bug.cgi?id=1913536.

@Ben Howard could you please review this as I did not see write to scheduler error reported in this bug which I guess is fixed in this bug?

Yes I believe you are running into https://bugzilla.redhat.com/show_bug.cgi?id=1913536. I will try to verify this fix by verifying a 4.6->4.6 upgrade with the backported fix in https://bugzilla.redhat.com/show_bug.cgi?id=1913316, as 4.6 should not be affected by this.

I was able to confirm this fix.

I created a release image that included this fix from the 4.6.0-0.ci release stream then upgraded my 4.6.9 cluster to it.

The upgrade is successful, even with the RHEL worker (though I had to work around https://bugzilla.redhat.com/show_bug.cgi?id=1913154).

I see this in the MCD logs on the RHEL node:

$ oc logs machine-config-daemon-jl7rb -c machine-config-daemon  | grep sched
I0107 17:49:12.140304    1880 controlplane.go:50] Device /sys/devices/pci0000:00/0000:00:1e.0/0000:05:01.0/0000:06:0a.0/virtio1/block/vda does not support the bfq scheduler

I then rolled back to 4.6.9 where I hit the error (expectedly as 4.6.9 does not contain the fix):

E0107 18:36:01.363148   41769 writer.go:135] Marking Degraded due to: write /sys/devices/pci0000:00/0000:00:1e.0/0000:05:01.0/0000:06:0a.0/virtio1/block/vda/queue/scheduler: invalid argument

Sunil, you can consider this verified by me.

Thank you Seth,

I will mark this as Verified as the error I see will be fixed in https://bugzilla.redhat.com/show_bug.cgi?id=1913536

No docs needed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633

Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475