Bug 1945248 - WMCO patch pub-key-hash annotation to Linux node
Summary: WMCO patch pub-key-hash annotation to Linux node
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Windows Containers
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.7.z
Assignee: Sebastian Soto
QA Contact: gaoshang
URL:
Whiteboard:
Depends On: 1930791
Blocks: 1934281 1945247
TreeView+ depends on / blocked
 
Reported: 2021-03-31 14:22 UTC by OpenShift BugZilla Robot
Modified: 2021-06-23 05:39 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-23 05:39:31 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift windows-machine-config-operator pull 370 0 None closed Bug 1945248: [wmco] Stop adding pub-key-hash label to all nodes 2021-05-03 21:36:32 UTC
Red Hat Product Errata RHSA-2021:2130 0 None None None 2021-06-23 05:39:42 UTC

Description OpenShift BugZilla Robot 2021-03-31 14:22:44 UTC 
+++ This bug was initially created as a clone of Bug #1930791 +++

Description of problem:
After replaced cloud-private-key secret with a new private key, found WMCO is patching pub-key-hash annotation to Linux node, including both master/work node.

# oc describe node sgao-a-bnxdr-master-0 | grep -A 20 Annotation
Annotations:        k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 0a:58:0a:80:00:03
                    k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"shared","interface-id":"br-ex_sgao-a-bnxdr-master-0","mac-address":"00:0d:3a:42:6b:39","ip-addresses":["10.0.0.8/19"],...
                    k8s.ovn.org/node-chassis-id: 32358f7c-bbf9-422a-893c-1d74482d5a3a
                    k8s.ovn.org/node-local-nat-ip: {"default":["169.254.8.64"]}
                    k8s.ovn.org/node-mgmt-port-mac-address: 4e:b8:d0:ce:cc:27
                    k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.0.0.8/19"}
                    k8s.ovn.org/node-subnets: {"default":"10.128.0.0/23"}
                    machine.openshift.io/machine: openshift-machine-api/sgao-a-bnxdr-master-0
                    machineconfiguration.openshift.io/currentConfig: rendered-master-4effb05b75d538407efd2b99f6c31c06
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-4effb05b75d538407efd2b99f6c31c06
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
---------------------------------------------------------------------------------------------------
                    windowsmachineconfig.openshift.io/pub-key-hash: 
---------------------------------------------------------------------------------------------------
CreationTimestamp:  Thu, 18 Feb 2021 22:07:37 -0500
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  sgao-a-bnxdr-master-0
  AcquireTime:     <unset>

Version-Release number of selected component (if applicable):
OCP version: 4.7.0-0.nightly-2021-02-18-110409
WMCO commit: f1f40153af071e9778d3ada5ec6dc93e9adfaa9d

How reproducible:
Always

Steps to Reproduce:
1, WMCO installed and Windows nodes has been bootstrapped
# oc get nodes -l kubernetes.io/os=windows
NAME            STATUS   ROLES    AGE   VERSION
windows-k9t5g   Ready    worker   98m   v1.20.0-1030+cac2421340a449

2, Delete secret cloud-private-key and create with another private key

# oc delete secret cloud-private-key
secret "cloud-private-key" deleted

# oc create secret generic cloud-private-key --from-file=private-key.pem=/root/.ssh/openshift-dev.pem 
secret/cloud-private-key created

3, Check WMCO log, found WMCO is patching pub-key-hash annotation to Linux node, and seems like there's duplicate message in log
"DEBUG	secret_controller	patched node object"
"INFO	secret_controller	updating secret"

# oc logs -f deployment.apps/windows-machine-config-operator
...
2021-02-19T13:52:08.366Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:52:08.370Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:52:19.235Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:52:19.242Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:53:56.589Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:01.223Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:01.228Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:16.637Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-master-0", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.654Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-master-1", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.667Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-master-2", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.699Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-worker-centralus1-cplbh", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.711Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-worker-centralus2-rtqm8", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.739Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "sgao-a-bnxdr-worker-centralus3-rkbdd", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.750Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "windows-k9t5g", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-02-19T13:54:16.751Z	INFO	secret_controller	updating secret	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "name": "windows-user-data"}
2021-02-19T13:54:16.752Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "windows-k9t5g"}
2021-02-19T13:54:16.765Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:16.772Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "secret_controller", "request": "openshift-windows-machine-config-operator/cloud-private-key"}
2021-02-19T13:54:16.773Z	INFO	windowsmachine-controller	deleting machine	{"name": "windows-k9t5g"}
2021-02-19T13:54:16.803Z	INFO	windowsmachine-controller	machine has been remediated by deletion	{"name": "windows-k9t5g"}
2021-02-19T13:54:16.803Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "windowsmachine-controller", "request": "openshift-machine-api/windows-k9t5g"}
2021-02-19T13:54:16.803Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "windows-k9t5g"}
2021-02-19T13:54:16.803Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Machine","namespace":"openshift-machine-api","name":"windows-k9t5g","uid":"63a6d9aa-715e-4fd2-9baf-2c9a0cb2beb9","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"185934"}, "reason": "MachineDeleted", "message": "Machine windows-k9t5g has been remediated by deleting the Machine object"}
2021-02-19T13:54:16.816Z	INFO	windowsmachine-controller	deleting machine	{"name": "windows-k9t5g"}
2021-02-19T13:54:16.828Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "windowsmachine-controller", "request": "openshift-machine-api/windows-k9t5g"}
2021-02-19T13:54:16.828Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "windows-k9t5g"}
2021-02-19T13:54:16.837Z	DEBUG	windowsmachine-controller	machine not provisioned	{"phase": "Deleting"}
2021-02-19T13:54:16.855Z	INFO	metrics	Prometheus configured	{"endpoints": "windows-machine-config-operator-metrics", "port": 9182, "name": "metrics"}
2021-02-19T13:54:16.856Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "windowsmachine-controller", "request": "openshift-machine-api/windows-k9t5g"}
2021-02-19T13:54:16.856Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "windows-tzgf5"}


Actual results:
WMCO is trying to patch annotation to Linux node
 
Expected results:
WMCO should not patch annotation to Linux node

Additional info:

--- Additional comment from sgao on 2021-03-09 03:05:40 UTC ---

This bug has been verified on OCP 4.8.0-0.nightly-2021-03-06-055252 and passed, thanks.

Version:
WMCO commit: b0dd4992bf0fb4cdff2b433f6a9ece3960447842
OCP version: 4.8.0-0.nightly-2021-03-06-055252

Steps:

After repeat steps in bug, found WMCO is not patching pub-key-hash annotation to Linux node anymore.

# oc logs -f deployment.apps/windows-machine-config-operator
...

2021-03-09T02:41:26.532Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "ip-10-0-145-235.us-east-2.compute.internal", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-03-09T02:41:26.532Z	DEBUG	windowsmachine-controller	reconciling	{"namespace": "openshift-machine-api", "name": "sgao-w886x-windows-worker-us-east-2a-9srsq"}
2021-03-09T02:41:26.540Z	DEBUG	secret_controller	patched node object	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "node": "ip-10-0-147-105.us-east-2.compute.internal", "patch": "[{\"op\":\"add\",\"path\":\"/metadata/annotations/windowsmachineconfig.openshift.io~1pub-key-hash\",\"value\":\"\"}]"}
2021-03-09T02:41:26.540Z	INFO	secret_controller	updating secret	{"namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "namespace": "openshift-windows-machine-config-operator", "name": "cloud-private-key", "name": "windows-user-data"}

--- Additional comment from sgao on 2021-03-09 06:55:34 UTC ---

Also verified annotations applied by previous versions of WMCO will be removed.

Steps:

Version:
WMCO commit: b0dd4992bf0fb4cdff2b433f6a9ece3960447842
OCP version: 4.8.0-0.nightly-2021-03-06-055252

1, Patch pub-key-hash annotation to Linux node

# oc patch node ip-10-0-130-66.us-east-2.compute.internal -p '{"metadata":{"annotations":{"windowsmachineconfig.openshift.io/pub-key-hash":""}}}'
node/ip-10-0-130-66.us-east-2.compute.internal patched

# oc describe node ip-10-0-130-66.us-east-2.compute.internal | grep -A 20 Annotation
Annotations:        csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-007263ab6e26bb9eb"}
                    k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 0a:58:0a:80:02:03
                    k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"shared","interface-id":"br-ex_ip-10-0-130-66.us-east-2.compute.internal","mac-address":"02:a2:e6:76:68:36","ip-address...
                    k8s.ovn.org/node-chassis-id: b8cd6f57-7606-41bf-97f7-a31f55e1aa5b
                    k8s.ovn.org/node-local-nat-ip: {"default":["169.254.0.168"]}
                    k8s.ovn.org/node-mgmt-port-mac-address: 2a:b1:8c:ff:42:2c
                    k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.0.130.66/19"}
                    k8s.ovn.org/node-subnets: {"default":"10.128.2.0/23"}
                    machine.openshift.io/machine: openshift-machine-api/sgao-w886x-worker-us-east-2a-d4zm9
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-e19e58fb3bc1d744149c30405a03bd64
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-e19e58fb3bc1d744149c30405a03bd64
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
---------------------------------------------------------------------------------------------------
                    windowsmachineconfig.openshift.io/pub-key-hash: 
---------------------------------------------------------------------------------------------------
CreationTimestamp:  Sun, 07 Mar 2021 23:37:19 -0500
Taints:             <none>
Unschedulable:      false
Lease:

2, Restart WMCO by deleting pod, check after WMCO restarted, pub-key-hash annotation on Linux node will be removed.

# oc delete pod/windows-machine-config-operator-865cc5d78c-9ghqw
pod "windows-machine-config-operator-865cc5d78c-9ghqw" deleted

# oc describe node ip-10-0-130-66.us-east-2.compute.internal | grep -A 20 Annotation
Annotations:        csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-007263ab6e26bb9eb"}
                    k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 0a:58:0a:80:02:03
                    k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"shared","interface-id":"br-ex_ip-10-0-130-66.us-east-2.compute.internal","mac-address":"02:a2:e6:76:68:36","ip-address...
                    k8s.ovn.org/node-chassis-id: b8cd6f57-7606-41bf-97f7-a31f55e1aa5b
                    k8s.ovn.org/node-local-nat-ip: {"default":["169.254.0.168"]}
                    k8s.ovn.org/node-mgmt-port-mac-address: 2a:b1:8c:ff:42:2c
                    k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.0.130.66/19"}
                    k8s.ovn.org/node-subnets: {"default":"10.128.2.0/23"}
                    machine.openshift.io/machine: openshift-machine-api/sgao-w886x-worker-us-east-2a-d4zm9
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-e19e58fb3bc1d744149c30405a03bd64
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-e19e58fb3bc1d744149c30405a03bd64
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 07 Mar 2021 23:37:19 -0500
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  ip-10-0-130-66.us-east-2.compute.internal
  AcquireTime:     <unset>
Comment 1 gaoshang 2021-05-07 09:59:15 UTC 
This bug has been verified on OCP 4.7.0-0.nightly-2021-05-05-092347 and passed, thanks.

Version:
WMCO commit: d93f4e7a3b0410b9df460fcd40af45f22c9a54d5
OCP version: 4.7.0-0.nightly-2021-05-05-092347

Steps:

After repeat steps in bug, found WMCO will not patch pub-key-hash annotation to Linux node anymore, and Linux node that patched pub-key-hash annotation by previous WMCO version has been fixed.

# oc get nodes
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-0-130-180.us-east-2.compute.internal   Ready    worker   45m     v1.20.0-1030+cac2421340a449
ip-10-0-131-236.us-east-2.compute.internal   Ready    worker   6h1m    v1.20.0+e3fdce4
ip-10-0-134-137.us-east-2.compute.internal   Ready    worker   33m     v1.20.0-1030+cac2421340a449
ip-10-0-143-115.us-east-2.compute.internal   Ready    master   6h11m   v1.20.0+e3fdce4
ip-10-0-161-222.us-east-2.compute.internal   Ready    master   6h10m   v1.20.0+e3fdce4
ip-10-0-177-49.us-east-2.compute.internal    Ready    worker   6h3m    v1.20.0+e3fdce4
ip-10-0-200-46.us-east-2.compute.internal    Ready    master   6h10m   v1.20.0+e3fdce4
ip-10-0-223-109.us-east-2.compute.internal   Ready    worker   6h2m    v1.20.0+e3fdce4


# oc describe node ip-10-0-131-236.us-east-2.compute.internal | grep -A 20 Annotation
Annotations:        csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-073e142c8967efd04"}
                    k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 0a:58:0a:81:02:03
                    k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"shared","interface-id":"br-ex_ip-10-0-131-236.us-east-2.compute.internal","mac-address":"02:08:83:c5:32:3e","ip-addres...
                    k8s.ovn.org/node-chassis-id: a56fe244-947b-4d6d-a535-5c5d4e984fa2
                    k8s.ovn.org/node-local-nat-ip: {"default":["169.254.11.157"]}
                    k8s.ovn.org/node-mgmt-port-mac-address: 76:d5:ee:00:2c:99
                    k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.0.131.236/19"}
                    k8s.ovn.org/node-subnets: {"default":"10.129.2.0/23"}
                    machine.openshift.io/machine: openshift-machine-api/sgao-47-wwdld-worker-us-east-2a-d57tp
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-40c6dbb2c449fa99d97002efcbb8a3d3
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-40c6dbb2c449fa99d97002efcbb8a3d3
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
--------------------------------------------------------------------------------------------
                    windowsmachineconfig.openshift.io/pub-key-hash: ---- removed
--------------------------------------------------------------------------------------------

CreationTimestamp:  Thu, 06 May 2021 23:17:09 -0400
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  ip-10-0-131-236.us-east-2.compute.internal
Comment 4 errata-xmlrpc 2021-06-23 05:39:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Windows Container Support for Red Hat OpenShift 2.0.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2130
Note You need to log in before you can comment on or make changes to this bug.