Bug 1930791

Summary: WMCO patch pub-key-hash annotation to Linux node
Product: OpenShift Container Platform Reporter: gaoshang <sgao>
Component: Windows ContainersAssignee: Sebastian Soto <ssoto>
Status: CLOSED ERRATA QA Contact: gaoshang <sgao>
Severity: high Docs Contact:
Priority: medium    
Version: 4.8CC: aos-bugs, ssoto, team-winc
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1934281 1945247 (view as bug list) Environment:
Last Closed: 2021-08-03 20:29:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1934281, 1945247, 1945248    

Description gaoshang 2021-02-19 15:16:28 UTC
Description of problem:
After replaced cloud-private-key secret with a new private key, found WMCO is patching pub-key-hash annotation to Linux node, including both master/work node.

# oc describe node sgao-a-bnxdr-master-0 | grep -A 20 Annotation
Annotations:        k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 0a:58:0a:80:00:03
                    k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"shared","interface-id":"br-ex_sgao-a-bnxdr-master-0","mac-address":"00:0d:3a:42:6b:39","ip-addresses":["10.0.0.8/19"],...
                    k8s.ovn.org/node-chassis-id: 32358f7c-bbf9-422a-893c-1d74482d5a3a
                    k8s.ovn.org/node-local-nat-ip: {"default":["169.254.8.64"]}
                    k8s.ovn.org/node-mgmt-port-mac-address: 4e:b8:d0:ce:cc:27
                    k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.0.0.8/19"}
                    k8s.ovn.org/node-subnets: {"default":"10.128.0.0/23"}
                    machine.openshift.io/machine: openshift-machine-api/sgao-a-bnxdr-master-0
                    machineconfiguration.openshift.io/currentConfig: rendered-master-4effb05b75d538407efd2b99f6c31c06
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-4effb05b75d538407efd2b99f6c31c06
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
---------------------------------------------------------------------------------------------------
                    windowsmachineconfig.openshift.io/pub-key-hash: 
---------------------------------------------------------------------------------------------------
CreationTimestamp:  Thu, 18 Feb 2021 22:07:37 -0500
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  sgao-a-bnxdr-master-0
  AcquireTime:     <unset>

Version-Release number of selected component (if applicable):
OCP version: 4.7.0-0.nightly-2021-02-18-110409
WMCO commit: f1f40153af071e9778d3ada5ec6dc93e9adfaa9d

How reproducible:
Always

Steps to Reproduce:
1, WMCO installed and Windows nodes has been bootstrapped
# oc get nodes -l kubernetes.io/os=windows
NAME            STATUS   ROLES    AGE   VERSION
windows-k9t5g   Ready    worker   98m   v1.20.0-1030+cac2421340a449

2, Delete secret cloud-private-key and create with another private key

# oc delete secret cloud-private-key
secret "cloud-private-key" deleted

# oc create secret generic cloud-private-key --from-file=private-key.pem=/root/.ssh/openshift-dev.pem 
secret/cloud-private-key created

3, Check WMCO log, found WMCO is patching pub-key-hash annotation to Linux node, and seems like there's duplicate message in log
"DEBUG	secret_controller	patched node object"
"INFO	secret_controller	updating secret"

# oc logs -f deployment.apps/windows-machine-config-operator
...
2021-02-19T13:52:08.366Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:52:08.370Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:52:19.235Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:52:19.242Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:53:56.589Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:01.223Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:01.228Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:16.637Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-master-0", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.654Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-master-1", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.667Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-master-2", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.699Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-worker-centralus1-cplbh", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.711Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-worker-centralus2-rtqm8", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.739Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-worker-centralus3-rkbdd", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.750Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "windows-k9t5g", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.751Z	INFO	secret_controller	updating secret	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "name": "windows-user-data"}
2021-02-19T13:54:16.752Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "windows-k9t5g"}
2021-02-19T13:54:16.765Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:16.772Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:16.773Z	INFO	windowsmachine-controller	deleting machine	{"name": "windows-k9t5g"}
2021-02-19T13:54:16.803Z	INFO	windowsmachine-controller	machine has been remediated by deletion	{"name": "windows-k9t5g"}
2021-02-19T13:54:16.803Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "windowsmachine-controller", "request": "openshift-machine-api/windows-k9t5g"}
2021-02-19T13:54:16.803Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "windows-k9t5g"}
2021-02-19T13:54:16.803Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Machine","namespace":"openshift-machine-api","name":"windows-k9t5g","uid":"63a6d9aa-715e-4fd2-9baf-2c9a0cb2beb9","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"185934"}, "reason": "MachineDeleted", "message": "Machine windows-k9t5g has been remediated by deleting the Machine object"}
2021-02-19T13:54:16.816Z	INFO	windowsmachine-controller	deleting machine	{"name": "windows-k9t5g"}
2021-02-19T13:54:16.828Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "windowsmachine-controller", "request": "openshift-machine-api/windows-k9t5g"}
2021-02-19T13:54:16.828Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "windows-k9t5g"}
2021-02-19T13:54:16.837Z	DEBUG	windowsmachine-controller	machine not provisioned	{"phase": "Deleting"}
2021-02-19T13:54:16.855Z	INFO	metrics	Prometheus configured	{"endpoints": "windows-machine-config-operator-metrics", "port": 9182, "name": "metrics"}
2021-02-19T13:54:16.856Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "windowsmachine-controller", "request": "openshift-machine-api/windows-k9t5g"}
2021-02-19T13:54:16.856Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "windows-tzgf5"}


Actual results:
WMCO is trying to patch annotation to Linux node
 
Expected results:
WMCO should not patch annotation to Linux node

Additional info:

Comment 1 gaoshang 2021-03-09 03:05:40 UTC
This bug has been verified on OCP 4.8.0-0.nightly-2021-03-06-055252 and passed, thanks.

Version:
WMCO commit: b0dd4992bf0fb4cdff2b433f6a9ece3960447842
OCP version: 4.8.0-0.nightly-2021-03-06-055252

Steps:

After repeat steps in bug, found WMCO is not patching pub-key-hash annotation to Linux node anymore.

# oc logs -f deployment.apps/windows-machine-config-operator
...

2021-03-09T02:41:26.532Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "ip-10-0-145-235.us-east-2.compute.internal", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-03-09T02:41:26.532Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "sgao-w886x-windows-worker-us-east-2a-9srsq"}
2021-03-09T02:41:26.540Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "ip-10-0-147-105.us-east-2.compute.internal", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-03-09T02:41:26.540Z	INFO	secret_controller	updating secret	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "name": "windows-user-data"}

Comment 2 gaoshang 2021-03-09 06:55:34 UTC
Also verified annotations applied by previous versions of WMCO will be removed.

Steps:

Version:
WMCO commit: b0dd4992bf0fb4cdff2b433f6a9ece3960447842
OCP version: 4.8.0-0.nightly-2021-03-06-055252

1, Patch pub-key-hash annotation to Linux node

# oc patch node ip-10-0-130-66.us-east-2.compute.internal -p '{"metadata":{"annotations":{"windowsmachineconfig.openshift.io/pub-key-hash":""}}}'
node/ip-10-0-130-66.us-east-2.compute.internal patched

# oc describe node ip-10-0-130-66.us-east-2.compute.internal | grep -A 20 Annotation
Annotations:        csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-007263ab6e26bb9eb"}
                    k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 0a:58:0a:80:02:03
                    k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"shared","interface-id":"br-ex_ip-10-0-130-66.us-east-2.compute.internal","mac-address":"02:a2:e6:76:68:36","ip-address...
                    k8s.ovn.org/node-chassis-id: b8cd6f57-7606-41bf-97f7-a31f55e1aa5b
                    k8s.ovn.org/node-local-nat-ip: {"default":["169.254.0.168"]}
                    k8s.ovn.org/node-mgmt-port-mac-address: 2a:b1:8c:ff:42:2c
                    k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.0.130.66/19"}
                    k8s.ovn.org/node-subnets: {"default":"10.128.2.0/23"}
                    machine.openshift.io/machine: openshift-machine-api/sgao-w886x-worker-us-east-2a-d4zm9
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-e19e58fb3bc1d744149c30405a03bd64
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-e19e58fb3bc1d744149c30405a03bd64
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
---------------------------------------------------------------------------------------------------
                    windowsmachineconfig.openshift.io/pub-key-hash: 
---------------------------------------------------------------------------------------------------
CreationTimestamp:  Sun, 07 Mar 2021 23:37:19 -0500
Taints:             <none>
Unschedulable:      false
Lease:

2, Restart WMCO by deleting pod, check after WMCO restarted, pub-key-hash annotation on Linux node will be removed.

# oc delete pod/windows-machine-config-operator-865cc5d78c-9ghqw
pod "windows-machine-config-operator-865cc5d78c-9ghqw" deleted

# oc describe node ip-10-0-130-66.us-east-2.compute.internal | grep -A 20 Annotation
Annotations:        csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-007263ab6e26bb9eb"}
                    k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 0a:58:0a:80:02:03
                    k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"shared","interface-id":"br-ex_ip-10-0-130-66.us-east-2.compute.internal","mac-address":"02:a2:e6:76:68:36","ip-address...
                    k8s.ovn.org/node-chassis-id: b8cd6f57-7606-41bf-97f7-a31f55e1aa5b
                    k8s.ovn.org/node-local-nat-ip: {"default":["169.254.0.168"]}
                    k8s.ovn.org/node-mgmt-port-mac-address: 2a:b1:8c:ff:42:2c
                    k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.0.130.66/19"}
                    k8s.ovn.org/node-subnets: {"default":"10.128.2.0/23"}
                    machine.openshift.io/machine: openshift-machine-api/sgao-w886x-worker-us-east-2a-d4zm9
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-e19e58fb3bc1d744149c30405a03bd64
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-e19e58fb3bc1d744149c30405a03bd64
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 07 Mar 2021 23:37:19 -0500
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  ip-10-0-130-66.us-east-2.compute.internal
  AcquireTime:     <unset>

Comment 3 Sebastian Soto 2021-03-31 14:24:52 UTC
*** Bug 1945247 has been marked as a duplicate of this bug. ***

Comment 6 errata-xmlrpc 2021-08-03 20:29:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Container Platform for Windows Containers 3.0.0 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3001