Bug 1852047

Summary: e2e-azure-upgrade-4.4-stable-to-4.5-ci failing with API unreachable for ~28% of the upgrade time
Product: OpenShift Container Platform Reporter: Abu Kashem <akashem>
Component: Machine Config OperatorAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.5CC: aos-bugs, jack.ottofaro, lmohanty, mfojtik, miabbott, nelluri, nstielau, sbatsche, scuppett, sdodson, vrutkovs, wking, xxia
Target Milestone: ---Keywords: Upgrades
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1850057 Environment:
Last Closed: 2020-10-27 16:10:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1852056    
Bug Blocks: 1850057    

Comment 2 Stefan Schimanski 2020-08-03 12:02:37 UTC
Moving to MCO as parent bug.

Comment 3 Sam Batschelet 2020-08-20 23:54:18 UTC
*** Bug 1861017 has been marked as a duplicate of this bug. ***

Comment 6 Micah Abbott 2020-09-19 20:53:53 UTC
Verified with 4.6.0-0.nightly-2020-09-19-060512

```
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-09-19-060512   True        False         20m     Cluster version is 4.6.0-0.nightly-2020-09-19-060512


$ oc adm upgrade --allow-explicit-upgrade=true --allow-upgrade-with-warnings=true --force=true --to-image=registry.svc.ci.openshift.org/ocp/release@sha256:5a09dd570d6cc87a41b55bcacc39958dc3de1877cd1945a4f479570df8083778

...

$ oc get clusterversion
NAME      VERSION                        AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.ci-2020-09-19-153015   True        False         13m     Cluster version is 4.6.0-0.ci-2020-09-19-153015


$ oc get events -o json | jq '.items | map(select(.source.component == "machineconfigcontroller-nodecontroller" or .source.component == "machineconfigdaemon")) | sort_by(.firstTimestamp | fromdate) | map(.firstTimestamp + " " + .involvedObject
.kind + " " + .involvedObject.name + ": " + .message)'                                                                                                         
[                                                                                                                                                              
  "2020-09-19T17:39:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-master-1, currentConfig rendered-master-9d39db7fc2ec3a03099836ae174057df to Done",
  "2020-09-19T17:39:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-master-2, currentConfig rendered-master-9d39db7fc2ec3a03099836ae174057df to Done",            
  "2020-09-19T17:39:54Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1 now has machineconfiguration.openshift.io/state=Done",               
  "2020-09-19T17:39:54Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2 now has machineconfiguration.openshift.io/state=Done",                                                                                                                                                                              
  "2020-09-19T17:39:55Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-master-0, currentConfig rendered-master-9d39db7fc2ec3a03099836ae174057df to Done",
  "2020-09-19T17:39:55Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0 now has machineconfiguration.openshift.io/state=Done",
  "2020-09-19T17:49:15Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x, currentConfig rendered-worker-ff560ececef24a9e8da6f01097187105 to Done",
  "2020-09-19T17:49:18Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7, currentConfig rendered-worker-ff560ececef24a9e8da6f01097187105 to Done",
  "2020-09-19T17:49:30Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq, currentConfig rendered-worker-ff560ececef24a9e8da6f01097187105 to Done",
  "2020-09-19T19:05:06Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x to config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",                                                       
  "2020-09-19T19:05:07Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Draining node to update config.",
  "2020-09-19T19:06:37Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: ",                                                                                      
  "2020-09-19T19:06:46Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Changes to OS staged",                                                   
  "2020-09-19T19:06:46Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Written pending config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",
  "2020-09-19T19:06:46Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Node will reboot into config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",                                                                                                                                                                        
  "2020-09-19T19:07:35Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x, currentConfig rendered-worker-e6c79d53ce9a19fa5793a06663af7c76 to Done",
  "2020-09-19T19:07:40Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7 to config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",
  "2020-09-19T19:07:41Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Draining node to update config.",                                              
  "2020-09-19T19:09:05Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: ",                                                                                                                                                                                                                                                     
  "2020-09-19T19:09:13Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Changes to OS staged",                                                                                                                                                                                                                                 
  "2020-09-19T19:09:13Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Written pending config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",
  "2020-09-19T19:09:13Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Node will reboot into config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",                                                                                                                                                                        
  "2020-09-19T19:10:02Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7, currentConfig rendered-worker-e6c79d53ce9a19fa5793a06663af7c76 to Done",
  "2020-09-19T19:10:07Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq to config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",                               
  "2020-09-19T19:10:08Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Draining node to update config.",
  "2020-09-19T19:11:16Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: ",                                                                                      
  "2020-09-19T19:11:27Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Changes to OS staged",                                                                                                                                                                                                                                 
  "2020-09-19T19:11:27Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Written pending config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",   
  "2020-09-19T19:11:27Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Node will reboot into config rendered-worker-e6c79d53ce9a19fa5793a06663af7c76",
  "2020-09-19T19:12:20Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq, currentConfig rendered-worker-e6c79d53ce9a19fa5793a06663af7c76 to Done",
  "2020-09-19T19:15:22Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7 to config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",
  "2020-09-19T19:15:23Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Draining node to update config.",
  "2020-09-19T19:16:48Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: ",                                                                                                                                                                                                                                                     
  "2020-09-19T19:16:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Changes to OS staged",
  "2020-09-19T19:16:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Written pending config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",
  "2020-09-19T19:16:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Node will reboot into config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",
  "2020-09-19T19:17:42Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7, currentConfig rendered-worker-c357fae6e3fdfa250b30478995e1fb05 to Done",
  "2020-09-19T19:17:47Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x to config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",
  "2020-09-19T19:17:48Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Draining node to update config.",
  "2020-09-19T19:18:38Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: ",                                                                                                                                                                                                                                                     
  "2020-09-19T19:18:45Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Changes to OS staged",                                                                  
  "2020-09-19T19:18:45Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Written pending config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",                                                                                                                                                                              
  "2020-09-19T19:18:45Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Node will reboot into config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",
  "2020-09-19T19:19:35Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x, currentConfig rendered-worker-c357fae6e3fdfa250b30478995e1fb05 to Done",
  "2020-09-19T19:19:40Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq to config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",
  "2020-09-19T19:19:41Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Draining node to update config.",                                                                                                                                                                                                                      
  "2020-09-19T19:20:39Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: ",                                                                                      
  "2020-09-19T19:20:47Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Changes to OS staged",                                                   
  "2020-09-19T19:20:47Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Written pending config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",      
  "2020-09-19T19:20:47Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Node will reboot into config rendered-worker-c357fae6e3fdfa250b30478995e1fb05",
  "2020-09-19T19:21:39Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq, currentConfig rendered-worker-c357fae6e3fdfa250b30478995e1fb05 to Done",                      
  "2020-09-19T19:30:37Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq to config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",
  "2020-09-19T19:30:38Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Draining node to update config.",                                  
  "2020-09-19T19:30:48Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: ",                                                                                      
  "2020-09-19T19:30:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Changes to OS staged",                                                                                                                                                                                                                                 
  "2020-09-19T19:30:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Written pending config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",                                                                                                                                                                              
  "2020-09-19T19:30:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Node will reboot into config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",
  "2020-09-19T19:31:47Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq, currentConfig rendered-worker-27733f7362bcf053ebffdd905ae1ccff to Done",
  "2020-09-19T19:31:52Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7 to config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",
  "2020-09-19T19:31:53Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Draining node to update config.",
  "2020-09-19T19:33:22Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: ",
  "2020-09-19T19:33:27Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Changes to OS staged",
  "2020-09-19T19:33:27Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Written pending config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",
  "2020-09-19T19:33:27Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Node will reboot into config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",
  "2020-09-19T19:34:16Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7, currentConfig rendered-worker-27733f7362bcf053ebffdd905ae1ccff to Done",
  "2020-09-19T19:34:21Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x to config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",
  "2020-09-19T19:34:22Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Draining node to update config.",
  "2020-09-19T19:35:14Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: ",
  "2020-09-19T19:35:20Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Changes to OS staged",
  "2020-09-19T19:35:20Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Written pending config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",
  "2020-09-19T19:35:20Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Node will reboot into config rendered-worker-27733f7362bcf053ebffdd905ae1ccff",
  "2020-09-19T19:36:09Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x, currentConfig rendered-worker-27733f7362bcf053ebffdd905ae1ccff to Done",
  "2020-09-19T20:24:07Z MachineConfigPool master: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-master-0 to config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:24:07Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0 now has machineconfiguration.openshift.io/desiredConfig=rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:24:07Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7 to config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:24:08Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0: Draining node to update config.",
  "2020-09-19T20:24:08Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Draining node to update config.",
  "2020-09-19T20:24:08Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0 now has machineconfiguration.openshift.io/state=Working",
  "2020-09-19T20:24:34Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0: Upgrading OS",
  "2020-09-19T20:24:34Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0: Updating from oscontainer registry.svc.ci.openshift.org/ocp/4.6-2020-09-19-153015@sha256:70e0547a93a5aec6926a2ab59ddd16de94b976cbfeae2ee7b64a4341c7404cde",
  "2020-09-19T20:25:07Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0: Changes to OS staged",
  "2020-09-19T20:25:07Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0: Written pending config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:25:07Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0: Node will reboot into config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:25:31Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Upgrading OS",
  "2020-09-19T20:25:31Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Updating from oscontainer registry.svc.ci.openshift.org/ocp/4.6-2020-09-19-153015@sha256:70e0547a93a5aec6926a2ab59ddd16de94b976cbfeae2ee7b64a4341c7404cde",
  "2020-09-19T20:25:51Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Changes to OS staged",
  "2020-09-19T20:25:52Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Written pending config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:25:52Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Node will reboot into config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:26:48Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-b-25cc7, currentConfig rendered-worker-399818b87c3feddaf26848a7ea9fcae0 to Done",
  "2020-09-19T20:26:53Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x to config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:26:54Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Draining node to update config.",
  "2020-09-19T20:27:37Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-0: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-master-0, currentConfig rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e to Done",
  "2020-09-19T20:27:42Z MachineConfigPool master: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-master-1 to config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:27:42Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1 now has machineconfiguration.openshift.io/desiredConfig=rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:27:43Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1: Draining node to update config.",
  "2020-09-19T20:27:43Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Upgrading OS",
  "2020-09-19T20:27:43Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Updating from oscontainer registry.svc.ci.openshift.org/ocp/4.6-2020-09-19-153015@sha256:70e0547a93a5aec6926a2ab59ddd16de94b976cbfeae2ee7b64a4341c7404cde",
  "2020-09-19T20:27:43Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1 now has machineconfiguration.openshift.io/state=Working",
  "2020-09-19T20:28:04Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Changes to OS staged",
  "2020-09-19T20:28:05Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Written pending config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:28:05Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Node will reboot into config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:28:23Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1: Upgrading OS",
  "2020-09-19T20:28:23Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1: Updating from oscontainer registry.svc.ci.openshift.org/ocp/4.6-2020-09-19-153015@sha256:70e0547a93a5aec6926a2ab59ddd16de94b976cbfeae2ee7b64a4341c7404cde",
  "2020-09-19T20:28:57Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1: Changes to OS staged",
  "2020-09-19T20:28:57Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1: Written pending config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:28:57Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1: Node will reboot into config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:29:05Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-c-pvg2x, currentConfig rendered-worker-399818b87c3feddaf26848a7ea9fcae0 to Done",
  "2020-09-19T20:30:20Z MachineConfigPool worker: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq to config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:30:21Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Draining node to update config.",
  "2020-09-19T20:30:59Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-1: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-master-1, currentConfig rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e to Done",
  "2020-09-19T20:31:04Z MachineConfigPool master: Targeted node ci-ln-j3tbpx2-f76d1-lrx8m-master-2 to config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:31:04Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2 now has machineconfiguration.openshift.io/desiredConfig=rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:31:05Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2: Draining node to update config.",
  "2020-09-19T20:31:05Z MachineConfigPool master: Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2 now has machineconfiguration.openshift.io/state=Working",
  "2020-09-19T20:31:15Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Upgrading OS",
  "2020-09-19T20:31:15Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Updating from oscontainer registry.svc.ci.openshift.org/ocp/4.6-2020-09-19-153015@sha256:70e0547a93a5aec6926a2ab59ddd16de94b976cbfeae2ee7b64a4341c7404cde",
  "2020-09-19T20:31:39Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Changes to OS staged",
  "2020-09-19T20:31:39Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Written pending config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:31:39Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Node will reboot into config rendered-worker-399818b87c3feddaf26848a7ea9fcae0",
  "2020-09-19T20:31:42Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2: Upgrading OS",
  "2020-09-19T20:31:42Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2: Updating from oscontainer registry.svc.ci.openshift.org/ocp/4.6-2020-09-19-153015@sha256:70e0547a93a5aec6926a2ab59ddd16de94b976cbfeae2ee7b64a4341c7404cde",
  "2020-09-19T20:32:13Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2: Changes to OS staged",
  "2020-09-19T20:32:13Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2: Written pending config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:32:13Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2: Node will reboot into config rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e",
  "2020-09-19T20:32:46Z Node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-worker-d-zs8jq, currentConfig rendered-worker-399818b87c3feddaf26848a7ea9fcae0 to Done",
  "2020-09-19T20:34:39Z Node ci-ln-j3tbpx2-f76d1-lrx8m-master-2: Setting node ci-ln-j3tbpx2-f76d1-lrx8m-master-2, currentConfig rendered-master-6f55b2b0f63b23f320b4c4e89b38b86e to Done"
]
```

Comment 8 errata-xmlrpc 2020-10-27 16:10:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196

Comment 9 W. Trevor King 2021-04-05 17:36:27 UTC
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475