Bug 1861404 - MCO panic when upgrading AWS cluster to 4.6 from 4.1
Summary: MCO panic when upgrading AWS cluster to 4.6 from 4.1
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Kirsten Garrison
QA Contact: Michael Nguyen
URL:
Whiteboard:
: 1870548 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-28 14:22 UTC by Yang Yang
Modified: 2020-09-19 22:02 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2057 None closed Bug 1861404: sync Infra in ControllerConfigSpec 2020-09-15 21:01:33 UTC

Description Yang Yang 2020-07-28 14:22:06 UTC
Description of problem:
I am upgrading AWS cluster from 4.1 to 4.6 but seeing a panic in machine-config-operator when upgrading to 4.6

Version-Release number of selected component (if applicable):
4.6.0-0.nightly-2020-07-25-091217

How reproducible:
2/2

Steps to Reproduce:
1.Upgrade IPI AWS cluster from 4.1 to 4.6, like this:
4.1.0-0.nightly-2020-05-28-040321, 
4.2.0-0.nightly-2020-07-18-024525,
4.3.0-0.nightly-2020-07-22-001315,
4.4.0-0.nightly-2020-07-22-132117,
4.5.0-0.nightly-2020-07-22-163134,
4.6.0-0.nightly-2020-07-25-091217
2.
3.

Actual results:
Upgrading to 4.6 failed and panic is in MCO
version   4.5.0-0.nightly-2020-07-22-163134   True        True          118m    Unable to apply 4.6.0-0.nightly-2020-07-25-091217: the cluster operator machine-config has not yet successfully rolled out

# oc logs machine-config-operator-67fdf89c85-q84kn -n openshift-machine-config-operator 
I0728 01:51:52.463753       1 start.go:46] Version: 4.6.0-0.nightly-2020-07-25-091217 (Raw: v4.6.0-202007250017.p0-dirty, Hash: ab2673986646c62bc6599931d22f050ffd5871db) 
I0728 01:51:52.466810       1 leaderelection.go:242] attempting to acquire leader lease  openshift-machine-config-operator/machine-config... 
I0728 01:53:50.286343       1 leaderelection.go:252] successfully acquired lease openshift-machine-config-operator/machine-config 
I0728 01:53:50.915868       1 operator.go:270] Starting MachineConfigOperator 
E0728 01:53:53.093372       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) 
goroutine 224 [running]: 
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1814240, 0x2a35120) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3 
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82 
panic(0x1814240, 0x2a35120) 
    /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/panic.go:969 +0x166 
github.com/openshift/machine-config-operator/lib/resourcemerge.ensureControllerConfigSpec(0xc002c5980f, 0xc00003c618, 0xc0029f3d20, 0xb, 0x0, 0x0, 0xc0029f3e48, 0x3, 0xc002c329c0, 0x29, ...) 
    /go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:79 +0x19f 
github.com/openshift/machine-config-operator/lib/resourcemerge.EnsureControllerConfig(0xc002c5980f, 0xc00003c500, 0x16c2301, 0x10, 0xc002c32a50, 0x24, 0xc002c1d4e0, 0x19, 0x0, 0x0, ...) 
    /go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:19 +0xd4 
github.com/openshift/machine-config-operator/lib/resourceapply.ApplyControllerConfig(0x7f9b99923210, 0xc000140880, 0xc00003c280, 0x7f9b99923210, 0xc000140880, 0x5ac9, 0x5b42) 
    /go/src/github.com/openshift/machine-config-operator/lib/resourceapply/machineconfig.go:67 +0x185 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncMachineConfigController(0xc000354000, 0xc000a26000, 0xc0021cc80e, 0x29bfde5178e6) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:468 +0x438 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncAll(0xc000354000, 0xc002c59ca8, 0x6, 0x6, 0xc0007b1c01, 0x413893) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:69 +0x177 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).sync(0xc000354000, 0xc0001548a0, 0x30, 0x0, 0x0) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:362 +0x40a 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).processNextWorkItem(0xc000354000, 0x203000) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:318 +0xd2 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).worker(0xc000354000) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:307 +0x2b 
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0007f20a0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f 
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0007f20a0, 0x1cbf1c0, 0xc000912180, 0xc000908001, 0xc0004d00c0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3 
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007f20a0, 0x3b9aca00, 0x0, 0x1, 0xc0004d00c0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2 
k8s.io/apimachinery/pkg/util/wait.Until(0xc0007f20a0, 0x3b9aca00, 0xc0004d00c0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d 
created by github.com/openshift/machine-config-operator/pkg/operator.(*Operator).Run 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:276 +0x3dc 
panic: runtime error: invalid memory address or nil pointer dereference [recovered] 
    panic: runtime error: invalid memory address or nil pointer dereference 
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a8 pc=0x13aac6f] 
 
goroutine 224 [running]: 
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105 
panic(0x1814240, 0x2a35120) 
    /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/panic.go:969 +0x166 
github.com/openshift/machine-config-operator/lib/resourcemerge.ensureControllerConfigSpec(0xc002c5980f, 0xc00003c618, 0xc0029f3d20, 0xb, 0x0, 0x0, 0xc0029f3e48, 0x3, 0xc002c329c0, 0x29, ...) 
    /go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:79 +0x19f 
github.com/openshift/machine-config-operator/lib/resourcemerge.EnsureControllerConfig(0xc002c5980f, 0xc00003c500, 0x16c2301, 0x10, 0xc002c32a50, 0x24, 0xc002c1d4e0, 0x19, 0x0, 0x0, ...) 
    /go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:19 +0xd4 
github.com/openshift/machine-config-operator/lib/resourceapply.ApplyControllerConfig(0x7f9b99923210, 0xc000140880, 0xc00003c280, 0x7f9b99923210, 0xc000140880, 0x5ac9, 0x5b42) 
    /go/src/github.com/openshift/machine-config-operator/lib/resourceapply/machineconfig.go:67 +0x185 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncMachineConfigController(0xc000354000, 0xc000a26000, 0xc0021cc80e, 0x29bfde5178e6) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:468 +0x438 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncAll(0xc000354000, 0xc002c59ca8, 0x6, 0x6, 0xc0007b1c01, 0x413893) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:69 +0x177 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).sync(0xc000354000, 0xc0001548a0, 0x30, 0x0, 0x0) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:362 +0x40a 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).processNextWorkItem(0xc000354000, 0x203000) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:318 +0xd2 
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).worker(0xc000354000) 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:307 +0x2b 
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0007f20a0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f 
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0007f20a0, 0x1cbf1c0, 0xc000912180, 0xc000908001, 0xc0004d00c0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3 
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007f20a0, 0x3b9aca00, 0x0, 0x1, 0xc0004d00c0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2 
k8s.io/apimachinery/pkg/util/wait.Until(0xc0007f20a0, 0x3b9aca00, 0xc0004d00c0) 
    /go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d 
created by github.com/openshift/machine-config-operator/pkg/operator.(*Operator).Run 
    /go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:276 +0x3dc 

Expected results:
Upgrade is successful.

Additional info:

Comment 2 Yang Yang 2020-07-28 14:32:13 UTC
Must gather tarball is online.  	
https://drive.google.com/file/d/1NhdY8_tz8aNgZCJUYxc5Kz_TJFrhprkB/view?usp=sharing

Comment 3 W. Trevor King 2020-08-18 03:30:46 UTC
4.6.0-0.nightly-2020-07-25-091217 is old, and since then things like [1] have landed.  Although from the attached must-gather, this isn't a vSphere cluster:

$ tar -xOz must-gather.local.3184524102915563670/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-611f823bcbd1b627ba7c3b4f558094dd08b4625567a49d56e8687295ae454d26/cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml <must-gather.tar.gz | grep -A5 platform
  platform: AWS
  platformStatus:
    aws:
      region: us-east-2
    type: AWS

So I'm not sure what's going on there.

[1]: https://github.com/openshift/machine-config-operator/pull/1951

Comment 4 Kirsten Garrison 2020-08-18 14:58:09 UTC
This particular panic was already fixed (line 79 in machineconfig.go) https://github.com/openshift/machine-config-operator/pull/1935/files

@yangyang since that was an old nightly (that doesn't exist anymore) can you verify that you still see this in your 4.1->...-> 4.6 upgrades using a recent nightly?

If not, I will dupe this to the old closed BZ.

Comment 5 Wenjing Zheng 2020-08-18 15:43:11 UTC
I still can see the bug with 4.6.0-0.nightly-2020-08-18-055142 which has some must-gather info here: https://bugzilla.redhat.com/show_bug.cgi?id=1866554#c12

Comment 6 Kirsten Garrison 2020-08-18 17:32:10 UTC
@Wenjing that must gather doesn't have any MCO logs..

Comment 7 Yang Yang 2020-08-19 06:47:15 UTC
I still found this issue recently when upgrading from 4.1.0-0.nightly-2020-07-29-210856	-> 4.2.0-0.nightly-2020-08-06-223716 -> 4.3.0-0.nightly-2020-08-10-122110 -> 4.4.0-0.nightly-2020-08-10-180247 -> 4.5.0-0.nightly-2020-08-10-150345 -> 4.6.0-0.nightly-2020-08-10-150008

It's different from bz1858026. In this issue, mco upgrades to 4.5 successfully but upgrade to 4.6 failed.

NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-08-10-150345   True        True          119m    Unable to apply 4.6.0-0.nightly-2020-08-10-150008: the cluster operator machine-config has not yet successfully rolled out

https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/upgrade_CI/4281/console

Comment 8 Kirsten Garrison 2020-08-19 17:11:06 UTC
How often are you seeing this? Have you reproduced this in any of the recent nightlies available from https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/#4.6.0-0.nightly

Comment 9 Kirsten Garrison 2020-08-19 17:21:48 UTC
I'm seeing:
2020-08-11T03:08:25.484223874Z I0811 03:08:25.484186       1 kubelet_config_controller.go:318] Error syncing kubeletconfig cluster: GenerateMachineConfigsforRole failed with error cannot generate MachineConfigs when no platformStatus.type is set
2020-08-11T03:08:25.773143916Z I0811 03:08:25.773102       1 template_controller.go:366] Error syncing controllerconfig machine-config-controller: failed to create MachineConfig for role master: cannot generate MachineConfigs when no platformStatus.type is set
2020-08-11T03:08:25.806605111Z I0811 03:08:25.806563       1 container_runtime_config_controller.go:369] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error cannot generate MachineConfigs when no platformStatus.type is set
2020-08-11T03:08:26.099569478Z I0811 03:08:26.099504       1 kubelet_config_controller.go:318] Error syncing kubeletconfig cluster: GenerateMachineConfigsforRole failed with error cannot generate MachineConfigs when no platformStatus.type is set

But I'm also seeing:
  platformStatus:
    aws:
      region: us-east-2
    type: AWS

Comment 11 Yang Yang 2020-08-20 10:10:14 UTC
It's reproducible when upgrading from 4.1.38-x86_64 -> 4.2.36-x86_64 -> 4.3.33-x86_64 -> 4.4.17-x86_64 -> 4.5.6-x86_64 -> 4.6.0-0.nightly-2020-08-18-165040.

https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/upgrade_CI/4403/console

Comment 12 Antonio Murdaca 2020-08-20 11:13:16 UTC
*** Bug 1870548 has been marked as a duplicate of this bug. ***

Comment 13 Antonio Murdaca 2020-08-20 11:24:31 UTC
This is caused by our ControllerConfig not syncing the infra object coming from 4.1 unfortunately (

Comment 14 Nick Stielau 2020-09-01 20:30:29 UTC
Antonio, I'm trying to decode your last comment.  Do we understand the problem?  Do we know what a fix might look like?

Comment 15 Kirsten Garrison 2020-09-01 23:16:10 UTC
@Nick Yes to both: we understand the problem and understand the path a fix might take.

Comment 16 Kirsten Garrison 2020-09-02 17:28:20 UTC
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1812179

Comment 19 Micah Abbott 2020-09-19 22:00:53 UTC
Verified with 4.6.0-0.nightly-2020-09-19-004228

(6+ hrs to go from 4.1 -> 4.6!)

```
$ oc get clusterversion                                                                                                                                                                                                                                    
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS              
version   4.1.41    True        False         41m     Cluster version is 4.1.41          

$ oc adm upgrade --allow-explicit-upgrade=true --allow-upgrade-with-warnings=true --force=true --to-image=quay.io/openshift-release-dev/ocp-release:4.2.36-x86_64                                                                                          
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead                                                                                                                               
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to preceed anyway              
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.                                    
Updating to release image quay.io/openshift-release-dev/ocp-release:4.2.36-x86_64                                                                     

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.2.36    True        False         6s      Cluster version is 4.2.36

$ oc adm upgrade --allow-explicit-upgrade=true --allow-upgrade-with-warnings=true --force=true --to-image=quay.io/openshift-release-dev/ocp-release:4.3.35-x86_64
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to preceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image quay.io/openshift-release-dev/ocp-release:4.3.35-x86_64

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.35    True        False         116s    Cluster version is 4.3.35

$ oc adm upgrade --allow-explicit-upgrade=true --allow-upgrade-with-warnings=true --force=true --to-image=quay.io/openshift-release-dev/ocp-release:4.4.23-x86_64
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to preceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image quay.io/openshift-release-dev/ocp-release:4.4.23-x86_64

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.23    True        False         25m     Cluster version is 4.4.23

$ oc adm upgrade --allow-explicit-upgrade=true --allow-upgrade-with-warnings=true --force=true --to-image=quay.io/openshift-release-dev/ocp-release:4.5.10-x86_64
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to preceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image quay.io/openshift-release-dev/ocp-release:4.5.10-x86_64

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.10    True        False         17m     Cluster version is 4.5.10

$ oc adm upgrade --allow-explicit-upgrade=true --allow-upgrade-with-warnings=true --force=true --to-image=registry.svc.ci.openshift.org/ocp/release@sha256:2f6222aecdfe27eae59131d1e698e06e48bb8bcefec336a39c4d2cd761a621ac                                
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to preceed anyway                                                                                                                                                                             
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.                                                                                                                                                                                                   
Updating to release image registry.svc.ci.openshift.org/ocp/release@sha256:2f6222aecdfe27eae59131d1e698e06e48bb8bcefec336a39c4d2cd761a621ac   

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-09-19-004228   True        False         34s     Cluster version is 4.6.0-0.nightly-2020-09-19-004228

$ oc describe co/machine-config
Name:         machine-config
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1
Kind:         ClusterOperator
Metadata:
  Creation Timestamp:  2020-09-19T15:50:31Z
  Generation:          1
  Resource Version:    230034
  Self Link:           /apis/config.openshift.io/v1/clusteroperators/machine-config
  UID:                 d938ac7b-fa8f-11ea-97bf-022c5ecfda63
Spec:
Status:
  Conditions:
    Last Transition Time:  2020-09-19T21:55:43Z
    Message:               Cluster has deployed 4.6.0-0.nightly-2020-09-19-004228
    Status:                True
    Type:                  Available
    Last Transition Time:  2020-09-19T21:55:43Z
    Message:               Cluster version is 4.6.0-0.nightly-2020-09-19-004228
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2020-09-19T21:31:53Z
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2020-09-19T17:27:43Z
    Reason:                AsExpected
    Status:                True
    Type:                  Upgradeable
  Extension:
    Master:  all 3 nodes are at latest configuration rendered-master-ea3d50e6005eb46bd31d38c7af0e1890
    Worker:  all 3 nodes are at latest configuration rendered-worker-89b4196319e54095cad4b003daa21e96
  Related Objects:
    Group:     
    Name:      openshift-machine-config-operator
    Resource:  namespaces
    Group:     machineconfiguration.openshift.io
    Name:      
    Resource:  machineconfigpools
    Group:     machineconfiguration.openshift.io
    Name:      
    Resource:  controllerconfigs
    Group:     machineconfiguration.openshift.io
    Name:      
    Resource:  kubeletconfigs
    Group:     machineconfiguration.openshift.io
    Name:      
    Resource:  containerruntimeconfigs
    Group:     machineconfiguration.openshift.io
    Name:      
    Resource:  machineconfigs
    Group:     
    Name:      
    Resource:  nodes
  Versions:
    Name:     operator
    Version:  4.6.0-0.nightly-2020-09-19-004228
Events:       <none>

$ oc get nodes
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-0-129-165.us-west-2.compute.internal   Ready    master   6h9m    v1.19.0+7f9e863
ip-10-0-135-149.us-west-2.compute.internal   Ready    worker   6h4m    v1.19.0+7f9e863
ip-10-0-147-43.us-west-2.compute.internal    Ready    worker   6h3m    v1.19.0+7f9e863
ip-10-0-154-82.us-west-2.compute.internal    Ready    master   6h10m   v1.19.0+7f9e863
ip-10-0-171-208.us-west-2.compute.internal   Ready    master   6h10m   v1.19.0+7f9e863
ip-10-0-173-169.us-west-2.compute.internal   Ready    worker   6h3m    v1.19.0+7f9e863
```

Comment 20 Micah Abbott 2020-09-19 22:02:11 UTC
```
$ oc -n openshift-machine-config-operator logs machine-config-operator-7b58dc6cff-fkvbd
I0919 21:46:37.540011       1 start.go:43] Version: 4.6.0-0.nightly-2020-09-19-004228 (Raw: v4.6.0-202009181332.p0-dirty, Hash: c08c048584ef0bf18ab2dd88fdddd93279e1c6a1)
I0919 21:46:37.543321       1 leaderelection.go:243] attempting to acquire leader lease  openshift-machine-config-operator/machine-config...
I0919 21:48:33.252997       1 leaderelection.go:253] successfully acquired lease openshift-machine-config-operator/machine-config
W0919 21:48:33.366368       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 21:48:33.376676       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 21:48:33.406841       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
I0919 21:48:33.780030       1 operator.go:253] Starting MachineConfigOperator
W0919 21:48:33.803280       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 21:54:40.888815       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
I0919 21:55:43.170205       1 event.go:282] Event(v1.ObjectReference{Kind:"", Namespace:"", Name:"machine-config", UID:"d938ac7b-fa8f-11ea-97bf-022c5ecfda63", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/machine-config-operator version changed from [{operator 4.5.10}] to [{operator 4.6.0-0.nightly-2020-09-19-004228}]
W0919 21:55:43.972427       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 21:56:05.332981       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 21:57:25.219700       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 21:57:43.308575       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 21:58:13.600420       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 21:58:20.218655       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 22:00:19.114058       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 22:00:20.570039       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 22:00:20.814491       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 22:00:25.672157       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0919 22:00:32.262745       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
```


Note You need to log in before you can comment on or make changes to this bug.