Description of problem: After configured a BYOH Windows instance with DNS name in config map, it got deconfigured immediately, this happens on UPI cluster on baremetal, BYOH Windows instance configured with IP address have not this issue. Following is the DNS name and config map used: PS C:\Users\Administrator> nslookup 10.0.55.187 Server: ip-10-0-0-2.us-east-2.compute.internal Address: 10.0.0.2 Name: ip-10-0-55-187.us-east-2.compute.internal Address: 10.0.55.187 # cat configmap_byoh.yaml kind: ConfigMap apiVersion: v1 metadata: name: windows-instances namespace: openshift-windows-machine-config-operator data: ip-10-0-55-187.us-east-2.compute.internal: |- username=Administrator # oc get node -l kubernetes.io/os=windows NAME STATUS ROLES AGE VERSION sgao-win1 Ready,SchedulingDisabled worker 8m15s v1.22.1-1660+bbcc9aea9e4bef # oc describe node sgao-win1 Name: sgao-win1 Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=windows kubernetes.io/arch=amd64 kubernetes.io/hostname=sgao-win1 kubernetes.io/os=windows node-role.kubernetes.io/worker= node.kubernetes.io/windows-build=10.0.17763 node.openshift.io/os_id=Windows windowsmachineconfig.openshift.io/byoh=true Annotations: k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 00-15-5D-C9-F5-8A k8s.ovn.org/hybrid-overlay-node-subnet: 10.132.10.0/24 volumes.kubernetes.io/controller-managed-attach-detach: true windowsmachineconfig.openshift.io/pub-key-hash: 1df2c166b1c401180523270e9cf6bc2cd2724b9279ea65668a3b95298525a0f5 windowsmachineconfig.openshift.io/username: -----BEGIN ENCRYPTED DATA-----<wmcoMarker><wmcoMarker>wx4EBwMIGyHM95CxsERgtbij7q4k3mYrEsaFVNoTO8jS5gF07WsxBH7z0Xp/aegs<wmcoMarker>VVx3CEY4... windowsmachineconfig.openshift.io/version: 3.1.0+d5fd8c8 CreationTimestamp: Mon, 06 Sep 2021 06:36:54 -0400 Taints: node.kubernetes.io/unschedulable:NoSchedule os=Windows:NoSchedule Unschedulable: true Lease: HolderIdentity: sgao-win1 AcquireTime: <unset> RenewTime: Mon, 06 Sep 2021 06:44:08 -0400 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Mon, 06 Sep 2021 06:41:56 -0400 Mon, 06 Sep 2021 06:36:54 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Mon, 06 Sep 2021 06:41:56 -0400 Mon, 06 Sep 2021 06:36:54 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Mon, 06 Sep 2021 06:41:56 -0400 Mon, 06 Sep 2021 06:36:54 -0400 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Mon, 06 Sep 2021 06:41:56 -0400 Mon, 06 Sep 2021 06:37:04 -0400 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.0.55.187 Hostname: sgao-win1 Capacity: cpu: 2 ephemeral-storage: 31455228Ki memory: 8125980Ki pods: 250 Allocatable: cpu: 1500m ephemeral-storage: 27915396253 memory: 6975004Ki pods: 250 System Info: Machine ID: sgao-win1 System UUID: EC277FBA-77CD-9B78-69E1-578CDC479EFA Boot ID: Kernel Version: 10.0.17763.2061 OS Image: Windows Server 2019 Datacenter Operating System: windows Architecture: amd64 Container Runtime Version: docker://20.10.6 Kubelet Version: v1.22.1-1660+bbcc9aea9e4bef Kube-Proxy Version: v1.22.1-1660+bbcc9aea9e4bef Non-terminated Pods: (0 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 0 (0%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeNotSchedulable 164m (x2 over 5h18m) kubelet Node sgao-win1 status is now: NodeNotSchedulable Normal Starting 139m kubelet Starting kubelet. Normal NodeHasSufficientMemory 139m (x2 over 139m) kubelet Node sgao-win1 status is now: NodeHasSufficientPID Warning CheckLimitsForResolvConf 139m kubelet open c:\k\etc\resolv.conf: The system cannot find the file specified. Normal NodeReady 139m kubelet Node sgao-win1 status is now: NodeReady Normal NodeNotSchedulable 139m kubelet Node sgao-win1 status is now: NodeNotSchedulable Normal Starting 136m kubelet Starting kubelet. Normal NodeHasSufficientMemory 136m kubelet Node sgao-win1 status is now: NodeHasSufficientMemory Normal NodeSchedulable 135m kubelet Node sgao-win1 status is now: NodeSchedulable Normal NodeNotSchedulable 135m (x2 over 136m) kubelet Node sgao-win1 status is now: NodeNotSchedulable Normal Starting 134m kubelet Starting kubelet. Warning CheckLimitsForResolvConf 134m kubelet open c:\k\etc\resolv.conf: The system cannot find the file specified. Normal NodeHasSufficientMemory 134m kubelet Node sgao-win1 status is now: NodeHasSufficientMemory Normal NodeNotSchedulable 134m kubelet Node sgao-win1 status is now: NodeNotSchedulable Normal Starting 131m kubelet Starting kubelet. Normal NodeHasSufficientMemory 131m kubelet Node sgao-win1 status is now: NodeHasSufficientMemory Normal NodeHasSufficientPID 131m kubelet Node sgao-win1 status is now: NodeHasSufficientPID Normal NodeSchedulable 130m kubelet Node sgao-win1 status is now: NodeSchedulable Normal NodeNotSchedulable 130m (x2 over 131m) kubelet Node sgao-win1 status is now: NodeNotSchedulable Normal Starting 124m kubelet Starting kubelet. Warning CheckLimitsForResolvConf 124m kubelet open c:\k\etc\resolv.conf: The system cannot find the file specified. Normal NodeHasSufficientMemory 124m (x2 over 124m) kubelet Node sgao-win1 status is now: NodeHasSufficientPID Normal NodeReady 123m kubelet Node sgao-win1 status is now: NodeReady Normal NodeNotSchedulable 123m kubelet Node sgao-win1 status is now: NodeNotSchedulable Normal Starting 121m kubelet Starting kubelet. Version-Release number of selected component (if applicable): OCP version: 4.9.0-0.nightly-2021-09-05-204238 WMCO mater branch commit: d5fd8c8d9b7ed21f4dc5eac1f410e893c305e840 How reproducible: Always Steps to Reproduce: 1, Install OCP 4.9 UPI on baremetal 2, Build WMCO locally with latest commit in master branch and install it on cluster 4, Manually install a BYOH Windows instance, configure it with DNS name in configmap Actual results: BYOH Windows instance got deconfigured imediately when it's Ready and stuck in "Configure" - "Deconfigure" cycle Expected results: BYOH Windows instance should be configured as node and in "Ready" status Additional info: Looks like this happens due to DNS name does not exist in node.Status.Addresses, see https://github.com/openshift/windows-machine-config-operator/blob/d5fd8c8d9b7ed21f4dc5eac1f410e893c305e840/controllers/configmap_controller.go#L215 # oc describe node sgao-win1 ... Addresses: InternalIP: 10.0.55.187 Hostname: sgao-win1 Capacity:
@sgao We are currently unable to reproduce this bug as we do not have access to a baremetal UPI setup (or any platform=none environment). Would it be possible for you to give me access to a QE cluster where this bug was seen?
@mohashai Sure, I'll prepare a cluster later today and DM to you, thanks.
Current status: - QE has seen this across multiple platforms - I have a fix up, PR is reviewable - successfully tested locally on 4.9 vSphere (and other platforms)
Fix is approved and ready to merge, but a change/deprecation in how to reference images to test is holding up our CI.
Hi, I try this bug on OCP 4.9.0-0.nightly-2021-09-16-215330, found WMCO configured BYOH Windows instance twice before Ready with DNS name in config map, while with IP address only configured one time, pls take a look, thanks. # oc get nodes -l kubernetes.io/os=windows NAME STATUS ROLES AGE VERSION sgao-win1 Ready worker 13m v1.22.1-1660+bbcc9aea9e4bef # oc logs -f deployment.apps/windows-machine-config-operator -n openshift-windows-machine-config-operator ... 2021-09-17T08:46:47.858Z INFO controller.secret secret not found, creating the secret {"secret": "openshift-windows-machine-config-operator/cloud-private-key", "name": "windows-user-data"} 2021-09-17T08:47:01.721Z INFO controllers.configmap processing {"instances in": "windows-instances"} 2021-09-17T08:47:01.721Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal initializing SSH connection 2021-09-17T08:47:31.950Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configuring 2021-09-17T08:47:32.073Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\k\\ mkdir C:\\k\\", "out": ""} 2021-09-17T08:47:32.086Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\Temp\\ mkdir C:\\Temp\\", "out": ""} 2021-09-17T08:47:32.099Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\k\\cni\\ mkdir C:\\k\\cni\\", "out": ""} 2021-09-17T08:47:32.112Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\k\\cni\\config\\ mkdir C:\\k\\cni\\config\\", "out": ""} 2021-09-17T08:47:32.125Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\var\\log\\ mkdir C:\\var\\log\\", "out": ""} 2021-09-17T08:47:32.138Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\var\\log\\kube-proxy\\ mkdir C:\\var\\log\\kube-proxy\\", "out": ""} 2021-09-17T08:47:32.151Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\var\\log\\hybrid-overlay\\ mkdir C:\\var\\log\\hybrid-overlay\\", "out": ""} 2021-09-17T08:47:32.151Z INFO wc ip-10-0-49-147.us-east-2.compute.internal transferring files 2021-09-17T08:47:33.503Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\hybrid-overlay-node.exe", "out": "False\r\n"} 2021-09-17T08:47:33.503Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload/hybrid-overlay-node.exe", "remote dir": "C:\\k\\"} 2021-09-17T08:47:38.803Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\wmcb.exe", "out": "False\r\n"} 2021-09-17T08:47:38.803Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload/wmcb.exe", "remote dir": "C:\\k\\"} 2021-09-17T08:47:40.947Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\windows_exporter.exe", "out": "False\r\n"} 2021-09-17T08:47:40.947Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload/windows_exporter.exe", "remote dir": "C:\\k\\"} 2021-09-17T08:47:43.109Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\\\win-overlay.exe", "out": "False\r\n"} 2021-09-17T08:47:43.109Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload//cni/win-overlay.exe", "remote dir": "C:\\k\\cni\\"} 2021-09-17T08:47:44.179Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\kube-proxy.exe", "out": "False\r\n"} 2021-09-17T08:47:44.179Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload//kube-node/kube-proxy.exe", "remote dir": "C:\\k\\"} 2021-09-17T08:47:49.514Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\Temp\\\\wget-ignore-cert.ps1", "out": "False\r\n"} 2021-09-17T08:47:49.514Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload//powershell/wget-ignore-cert.ps1", "remote dir": "C:\\Temp\\"} 2021-09-17T08:47:49.906Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\\\win-bridge.exe", "out": "False\r\n"} 2021-09-17T08:47:49.906Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload//cni/win-bridge.exe", "remote dir": "C:\\k\\cni\\"} 2021-09-17T08:47:50.956Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\kubelet.exe", "out": "False\r\n"} 2021-09-17T08:47:50.957Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload//kube-node/kubelet.exe", "remote dir": "C:\\k\\"} 2021-09-17T08:48:03.603Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\\\flannel.exe", "out": "False\r\n"} 2021-09-17T08:48:03.603Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload//cni/flannel.exe", "remote dir": "C:\\k\\cni\\"} 2021-09-17T08:48:04.551Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\\\host-local.exe", "out": "False\r\n"} 2021-09-17T08:48:04.551Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload//cni/host-local.exe", "remote dir": "C:\\k\\cni\\"} 2021-09-17T08:48:05.283Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\Temp\\\\hns.psm1", "out": "False\r\n"} 2021-09-17T08:48:05.283Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/payload//powershell/hns.psm1", "remote dir": "C:\\Temp\\"} 2021-09-17T08:48:05.370Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe create windows_exporter binPath=\"C:\\k\\windows_exporter.exe --collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info\" start=auto", "out": "[SC] CreateService SUCCESS\r\n"} 2021-09-17T08:48:05.394Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query windows_exporter", "out": "\r\nSERVICE_NAME: windows_exporter \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 1077 (0x435)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:48:28.874Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe start windows_exporter", "out": "\r\nSERVICE_NAME: windows_exporter \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 2 START_PENDING \r\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x7d0\r\n PID : 3400\r\n FLAGS : \r\n"} 2021-09-17T08:48:28.874Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured {"service": "windows_exporter", "args": "--collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info\""} 2021-09-17T08:48:31.090Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass C:\\Temp\\wget-ignore-cert.ps1 -server https://api-int.sgao-b49.qe.devcluster.openshift.com:22623/config/worker -output C:\\Windows\\Temp\\worker.ign -acceptHeader application/vnd.coreos.ignition+json`;version=3.1.0", "out": ""} 2021-09-17T08:48:32.886Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass C:\\k\\\\wmcb.exe initialize-kubelet --ignition-file C:\\Windows\\Temp\\worker.ign --kubelet-path C:\\k\\kubelet.exe --node-ip=10.0.49.147", "out": "Bootstrapping completed successfully"} 2021-09-17T08:48:32.886Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured kubelet {"cmd": "C:\\k\\\\wmcb.exe initialize-kubelet --ignition-file C:\\Windows\\Temp\\worker.ign --kubelet-path C:\\k\\kubelet.exe --node-ip=10.0.49.147", "output": "Bootstrapping completed successfully"} 2021-09-17T08:48:32.911Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal initializing SSH connection 2021-09-17T08:48:33.612Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass hostname", "out": "sgao-win1\r\n"} 2021-09-17T08:48:33.619Z INFO controllers.CertificateSigningRequests CSR approved {"CSR": "csr-2z97v"} 2021-09-17T08:48:36.243Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal initializing SSH connection 2021-09-17T08:48:36.657Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass hostname", "out": "sgao-win1\r\n"} 2021-09-17T08:48:36.663Z INFO controllers.CertificateSigningRequests CSR approved {"CSR": "csr-sc942"} 2021-09-17T08:49:02.917Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configure {"service": "hybrid-overlay-node", "args": "--node sgao-win1 --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log\" depend= kubelet"} 2021-09-17T08:49:02.967Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe create hybrid-overlay-node binPath=\"C:\\k\\hybrid-overlay-node.exe --node sgao-win1 --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log\" depend= kubelet start=auto", "out": "[SC] CreateService SUCCESS\r\n"} 2021-09-17T08:49:02.993Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query hybrid-overlay-node", "out": "\r\nSERVICE_NAME: hybrid-overlay-node \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 1077 (0x435)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:49:03.614Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe start hybrid-overlay-node", "out": "\r\nSERVICE_NAME: hybrid-overlay-node \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 2 START_PENDING \r\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x7d0\r\n PID : 3140\r\n FLAGS : \r\n"} 2021-09-17T08:49:03.650Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query hybrid-overlay-node", "out": "\r\nSERVICE_NAME: hybrid-overlay-node \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 4 RUNNING \r\n (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:51:04.556Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Get-HnsNetwork", "out": "\r\n\r\nActivityId : 54B65D47-6BE9-48C4-B145-AA86BBADBCC2\r\nAdditionalParams : \r\nCurrentEndpointCount : 0\r\nExtensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering \r\n Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=False; Name=Microsoft Azure \r\n VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; \r\n Name=Microsoft NDIS Capture}}\r\nFlags : 0\r\nHealth : @{AddressNotificationMissedCount=0; AddressNotificationSequenceNumber=0; \r\n InterfaceNotificationMissedCount=0; InterfaceNotificationSequenceNumber=0; LastErrorCode=0; \r\n LastUpdateTime=132763419822202348; RouteNotificationMissedCount=0; \r\n RouteNotificationSequenceNumber=0}\r\nID : 784735AF-170C-429D-81C9-AD155465183A\r\nIPv6 : False\r\nLayeredOn : 2DE3312D-C462-4485-BABD-C540871A24F1\r\nMacPools : {@{EndMacAddress=00-15-5D-3D-3F-FF; StartMacAddress=00-15-5D-3D-30-00}}\r\nMaxConcurrentEndpoints : 0\r\nName : nat\r\nNatName : ICSA73B71D1-2056-49FE-87D6-A192B359062C\r\nPolicies : {}\r\nResources : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=; \r\n ID=54B65D47-6BE9-48C4-B145-AA86BBADBCC2; PortOperationTime=0; State=1; SwitchOperationTime=0; \r\n VfpOperationTime=0; parentId=6819C11A-B854-4192-9334-B768B63C42AA}\r\nState : 1\r\nSubnets : {@{AdditionalParams=; AddressPrefix=172.17.48.0/20; GatewayAddress=172.17.48.1; Health=; \r\n ID=E61F9295-6575-4186-AD97-C15772C891B0; Policies=System.Object[]; State=0}}\r\nTotalEndpoints : 0\r\nType : nat\r\nVersion : 38654705669\r\n\r\nActivityId : 72D40D15-7BC8-4DE9-9F7C-E78089663511\r\nAdditionalParams : \r\nCurrentEndpointCount : 0\r\nDNSServerCompartment : 3\r\nDrMacAddress : 00-15-5D-87-45-3C\r\nExtensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering \r\n Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=True; Name=Microsoft Azure \r\n VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; \r\n Name=Microsoft NDIS Capture}}\r\nFlags : 0\r\nHealth : @{LastErrorCode=0; LastUpdateTime=132763421436980943}\r\nID : E00062A5-251B-439C-961C-EBF9A996229E\r\nIPv6 : False\r\nInterfaceConstraint : @{InterfaceGuid=00000000-0000-0000-0000-000000000000}\r\nLayeredOn : 63D6936C-7220-4B82-9249-7118A7885D20\r\nMacPools : {@{EndMacAddress=00-15-5D-3B-5F-FF; StartMacAddress=00-15-5D-3B-50-00}}\r\nManagementIP : 10.0.49.147\r\nMaxConcurrentEndpoints : 0\r\nName : BaseOVNKubernetesHybridOverlayNetwork\r\nPolicies : {}\r\nResources : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; Health=; \r\n ID=72D40D15-7BC8-4DE9-9F7C-E78089663511; PortOperationTime=0; State=1; SwitchOperationTime=0; \r\n VfpOperationTime=0; parentId=D3CD0E64-F8A8-4781-B509-934EBE2BE4B9}\r\nState : 1\r\nSubnets : {@{AdditionalParams=; AddressPrefix=100.64.0.0/30; GatewayAddress=100.64.0.2; Health=; \r\n ID=C8086622-2F63-4F08-91AB-A464AF1D1DA2; ObjectType=5; Policies=System.Object[]; State=0}}\r\nTotalEndpoints : 0\r\nType : Overlay\r\nVersion : 38654705669\r\n\r\nActivityId : 1E11416B-1C7F-48A3-81CF-B06F27851BBE\r\nAdditionalParams : \r\nAutomaticDNS : True\r\nCurrentEndpointCount : 0\r\nDNSServerCompartment : 4\r\nDrMacAddress : 00-15-5D-87-45-3C\r\nExtensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering \r\n Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=True; Name=Microsoft Azure \r\n VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; \r\n Name=Microsoft NDIS Capture}}\r\nFlags : 8\r\nHealth : @{LastErrorCode=0; LastUpdateTime=132763421550987382}\r\nID : AD6BD317-DE2A-4F8C-8E5D-41970C3BFA48\r\nIPv6 : False\r\nInterfaceConstraint : @{InterfaceGuid=00000000-0000-0000-0000-000000000000}\r\nLayeredOn : 63D6936C-7220-4B82-9249-7118A7885D20\r\nMacPools : {@{EndMacAddress=00-15-5D-01-5F-FF; StartMacAddress=00-15-5D-01-50-00}}\r\nManagementIP : 10.0.49.147\r\nMaxConcurrentEndpoints : 0\r\nName : OVNKubernetesHybridOverlayNetwork\r\nPolicies : {@{Type=HostRoute}, @{DestinationPrefix=10.129.2.0/23; \r\n DistributedRouterMacAddress=0a:58:0a:81:02:03; IsolationId=4097; ProviderAddress=10.0.50.120; \r\n Type=RemoteSubnetRoute}, @{DestinationPrefix=10.128.0.0/23; \r\n DistributedRouterMacAddress=0a:58:0a:80:00:03; IsolationId=4097; ProviderAddress=10.0.57.112; \r\n Type=RemoteSubnetRoute}, @{DestinationPrefix=10.130.0.0/23; \r\n DistributedRouterMacAddress=0a:58:0a:82:00:03; IsolationId=4097; ProviderAddress=10.0.61.202; \r\n Type=RemoteSubnetRoute}...}\r\nResources : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; Health=; \r\n ID=1E11416B-1C7F-48A3-81CF-B06F27851BBE; PortOperationTime=0; State=1; SwitchOperationTime=0; \r\n VfpOperationTime=0; parentId=D3CD0E64-F8A8-4781-B509-934EBE2BE4B9}\r\nState : 1\r\nSubnets : {@{AdditionalParams=; AddressPrefix=10.132.0.0/24; GatewayAddress=10.132.0.1; Health=; \r\n ID=8E1A0F3C-69E6-4EC7-B2A4-64EBD67E30A5; ObjectType=5; Policies=System.Object[]; State=0}}\r\nTotalEndpoints : 0\r\nType : Overlay\r\nVersion : 38654705669\r\n\r\n\r\n\r\n"} 2021-09-17T08:51:04.556Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured {"service": "hybrid-overlay-node", "args": "--node sgao-win1 --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log\" depend= kubelet"} 2021-09-17T08:51:19.853Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\config\\\\cni.conf", "out": "False\r\n"} 2021-09-17T08:51:19.853Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal copy {"local file": "/tmp/cni092218268/cni.conf", "remote dir": "C:\\k\\cni\\config\\"} 2021-09-17T08:51:51.748Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass C:\\k\\wmcb.exe configure-cni --cni-dir=\"C:\\k\\cni\\ --cni-config=\"C:\\k\\cni\\config\\cni.conf", "out": "CNI configuration completed successfully"} 2021-09-17T08:51:51.748Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured kubelet for CNI {"cmd": "C:\\k\\wmcb.exe configure-cni --cni-dir=\"C:\\k\\cni\\ --cni-config=\"C:\\k\\cni\\config\\cni.conf", "output": "CNI configuration completed successfully"} 2021-09-17T08:51:57.131Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Import-Module -DisableNameChecking C:\\Temp\\hns.psm1; $net = (Get-HnsNetwork | where { $_.Name -eq 'OVNKubernetesHybridOverlayNetwork' }); $endpoint = New-HnsEndpoint -NetworkId $net.ID -Name VIPEndpoint; Attach-HNSHostEndpoint -EndpointID $endpoint.ID -CompartmentID 1; (Get-NetIPConfiguration -AllCompartments -All -Detailed | where { $_.NetAdapter.LinkLayerAddress -eq $endpoint.MacAddress }).IPV4Address.IPAddress.Trim()\"", "out": "10.132.0.14\r\n"} 2021-09-17T08:51:57.182Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe create kube-proxy binPath=\"C:\\k\\kube-proxy.exe --windows-service --v=4 --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=sgao-win1 --kubeconfig=c:\\k\\kubeconfig --cluster-cidr=10.132.0.0/24 --log-dir=C:\\var\\log\\kube-proxy\\ --logtostderr=false --network-name=OVNKubernetesHybridOverlayNetwork --source-vip=10.132.0.14 --enable-dsr=false\" depend= hybrid-overlay-node start=auto", "out": "[SC] CreateService SUCCESS\r\n"} 2021-09-17T08:51:57.205Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query kube-proxy", "out": "\r\nSERVICE_NAME: kube-proxy \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 1077 (0x435)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:51:57.746Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe start kube-proxy", "out": "\r\nSERVICE_NAME: kube-proxy \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 2 START_PENDING \r\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x7d0\r\n PID : 3876\r\n FLAGS : \r\n"} 2021-09-17T08:51:57.746Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured {"service": "kube-proxy", "args": "--windows-service --v=4 --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=sgao-win1 --kubeconfig=c:\\k\\kubeconfig --cluster-cidr=10.132.0.0/24 --log-dir=C:\\var\\log\\kube-proxy\\ --logtostderr=false --network-name=OVNKubernetesHybridOverlayNetwork --source-vip=10.132.0.14 --enable-dsr=false\" depend= hybrid-overlay-node"} 2021-09-17T08:52:12.778Z INFO nc ip-10-0-49-147.us-east-2.compute.internal instance has been configured as a worker node {"version": "3.1.0+8ffe65a"} 2021-09-17T08:52:12.779Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"openshift-windows-machine-config-operator","name":"windows-instances","apiVersion":"v1"}, "reason": "InstanceSetup", "message": "Configured instance with address ip-10-0-49-147.us-east-2.compute.internal as a worker node"} 2021-09-17T08:52:12.810Z INFO metrics Prometheus configured {"endpoints": "windows-exporter", "port": 9182, "name": "metrics"} 2021-09-17T08:52:12.812Z INFO controllers.configmap processing {"instances in": "windows-instances"} 2021-09-17T08:52:12.814Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal initializing SSH connection 2021-09-17T08:52:22.992Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configuring 2021-09-17T08:52:23.019Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe qc windows_exporter", "out": "[SC] QueryServiceConfig SUCCESS\r\n\r\nSERVICE_NAME: windows_exporter\r\n TYPE : 10 WIN32_OWN_PROCESS \r\n START_TYPE : 2 AUTO_START\r\n ERROR_CONTROL : 1 NORMAL\r\n BINARY_PATH_NAME : C:\\k\\windows_exporter.exe --collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info\r\n LOAD_ORDER_GROUP : \r\n TAG : 0\r\n DISPLAY_NAME : windows_exporter\r\n DEPENDENCIES : \r\n SERVICE_START_NAME : LocalSystem\r\n"} 2021-09-17T08:52:23.048Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query windows_exporter", "out": "\r\nSERVICE_NAME: windows_exporter \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 4 RUNNING \r\n (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:52:23.075Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe stop windows_exporter", "out": "\r\nSERVICE_NAME: windows_exporter \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:52:38.100Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query windows_exporter", "out": "\r\nSERVICE_NAME: windows_exporter \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:52:38.124Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe qc kube-proxy", "out": "[SC] QueryServiceConfig SUCCESS\r\n\r\nSERVICE_NAME: kube-proxy\r\n TYPE : 10 WIN32_OWN_PROCESS \r\n START_TYPE : 2 AUTO_START\r\n ERROR_CONTROL : 1 NORMAL\r\n BINARY_PATH_NAME : C:\\k\\kube-proxy.exe --windows-service --v=4 --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=sgao-win1 --kubeconfig=c:\\k\\kubeconfig --cluster-cidr=10.132.0.0/24 --log-dir=C:\\var\\log\\kube-proxy\\ --logtostderr=false --network-name=OVNKubernetesHybridOverlayNetwork --source-vip=10.132.0.14 --enable-dsr=false\r\n LOAD_ORDER_GROUP : \r\n TAG : 0\r\n DISPLAY_NAME : kube-proxy\r\n DEPENDENCIES : hybrid-overlay-node\r\n SERVICE_START_NAME : LocalSystem\r\n"} 2021-09-17T08:52:38.147Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query kube-proxy", "out": "\r\nSERVICE_NAME: kube-proxy \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 4 RUNNING \r\n (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:52:38.172Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe stop kube-proxy", "out": "\r\nSERVICE_NAME: kube-proxy \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 3 STOP_PENDING \r\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:52:53.199Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query kube-proxy", "out": "\r\nSERVICE_NAME: kube-proxy \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:52:53.223Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe qc hybrid-overlay-node", "out": "[SC] QueryServiceConfig SUCCESS\r\n\r\nSERVICE_NAME: hybrid-overlay-node\r\n TYPE : 10 WIN32_OWN_PROCESS \r\n START_TYPE : 2 AUTO_START\r\n ERROR_CONTROL : 1 NORMAL\r\n BINARY_PATH_NAME : C:\\k\\hybrid-overlay-node.exe --node sgao-win1 --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log\r\n LOAD_ORDER_GROUP : \r\n TAG : 0\r\n DISPLAY_NAME : hybrid-overlay-node\r\n DEPENDENCIES : kubelet\r\n SERVICE_START_NAME : LocalSystem\r\n"} 2021-09-17T08:52:53.246Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query hybrid-overlay-node", "out": "\r\nSERVICE_NAME: hybrid-overlay-node \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 4 RUNNING \r\n (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:52:53.270Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe stop hybrid-overlay-node", "out": "\r\nSERVICE_NAME: hybrid-overlay-node \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 3 STOP_PENDING \r\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:53:08.295Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query hybrid-overlay-node", "out": "\r\nSERVICE_NAME: hybrid-overlay-node \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:53:08.319Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe qc kubelet", "out": "[SC] QueryServiceConfig SUCCESS\r\n\r\nSERVICE_NAME: kubelet\r\n TYPE : 10 WIN32_OWN_PROCESS \r\n START_TYPE : 2 AUTO_START\r\n ERROR_CONTROL : 1 NORMAL\r\n BINARY_PATH_NAME : c:\\k\\kubelet.exe --windows-service --kubeconfig=c:\\k\\kubeconfig --register-with-taints=os=Windows:NoSchedule --v=3 --pod-infra-container-image=mcr.microsoft.com/oss/kubernetes/pause:3.4.1 --image-pull-progress-deadline=30m --cni-bin-dir=c:\\k\\cni --cni-conf-dir=c:\\k\\cni\\config --config=c:\\k\\kubelet.conf --node-labels=node.openshift.io/os_id=Windows --cloud-provider= --resolv-conf=\"\" --node-ip=10.0.49.147 --network-plugin=cni --bootstrap-kubeconfig=c:\\k\\bootstrap-kubeconfig --cert-dir=c:\\var\\lib\\kubelet\\pki\\ --logtostderr=false --log-file=C:\\var\\log\\kubelet\\kubelet.log\r\n LOAD_ORDER_GROUP : \r\n TAG : 0\r\n DISPLAY_NAME : kubelet\r\n DEPENDENCIES : docker\r\n SERVICE_START_NAME : LocalSystem\r\n"} 2021-09-17T08:53:08.342Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query kubelet", "out": "\r\nSERVICE_NAME: kubelet \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 4 RUNNING \r\n (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:53:08.371Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe stop kubelet", "out": "\r\nSERVICE_NAME: kubelet \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 3 STOP_PENDING \r\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:53:23.397Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query kubelet", "out": "\r\nSERVICE_NAME: kubelet \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:53:23.409Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\k\\ mkdir C:\\k\\", "out": ""} 2021-09-17T08:53:23.421Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\Temp\\ mkdir C:\\Temp\\", "out": ""} 2021-09-17T08:53:23.433Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\k\\cni\\ mkdir C:\\k\\cni\\", "out": ""} 2021-09-17T08:53:23.445Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\k\\cni\\config\\ mkdir C:\\k\\cni\\config\\", "out": ""} 2021-09-17T08:53:23.457Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\var\\log\\ mkdir C:\\var\\log\\", "out": ""} 2021-09-17T08:53:23.470Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\var\\log\\kube-proxy\\ mkdir C:\\var\\log\\kube-proxy\\", "out": ""} 2021-09-17T08:53:23.482Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "if not exist C:\\var\\log\\hybrid-overlay\\ mkdir C:\\var\\log\\hybrid-overlay\\", "out": ""} 2021-09-17T08:53:23.482Z INFO wc ip-10-0-49-147.us-east-2.compute.internal transferring files 2021-09-17T08:53:23.765Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\kubelet.exe", "out": "True\r\n"} 2021-09-17T08:53:24.392Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\\\kubelet.exe -Algorithm SHA256; $out.Hash", "out": "2D345F88C05082B7D69F67F21A7FBBE4F47C197911FEC26BAC0C6627DEFDD6A1\r\n"} 2021-09-17T08:53:24.392Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload//kube-node/kubelet.exe"} 2021-09-17T08:53:24.665Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\hybrid-overlay-node.exe", "out": "True\r\n"} 2021-09-17T08:53:25.160Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\\\hybrid-overlay-node.exe -Algorithm SHA256; $out.Hash", "out": "671384101209280A648CC66E759A2669C40FBC064AD1B3F78E8A75B5AACAA930\r\n"} 2021-09-17T08:53:25.160Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload/hybrid-overlay-node.exe"} 2021-09-17T08:53:25.441Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\wmcb.exe", "out": "True\r\n"} 2021-09-17T08:53:25.882Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\\\wmcb.exe -Algorithm SHA256; $out.Hash", "out": "FCA7D2D0A8E630F2E86CE363E048C34D715008C2B4CBAC6C21F0FFC60BD8418C\r\n"} 2021-09-17T08:53:25.882Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload/wmcb.exe"} 2021-09-17T08:53:26.141Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\windows_exporter.exe", "out": "True\r\n"} 2021-09-17T08:53:26.593Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\\\windows_exporter.exe -Algorithm SHA256; $out.Hash", "out": "74C906D66F8109402497FD0A1689F1B96D43FA6893CA7ADACA24F49C24FA4FA1\r\n"} 2021-09-17T08:53:26.593Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload/windows_exporter.exe"} 2021-09-17T08:53:26.848Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\\\win-overlay.exe", "out": "True\r\n"} 2021-09-17T08:53:27.272Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\cni\\\\win-overlay.exe -Algorithm SHA256; $out.Hash", "out": "75728E246536777E5BE769F46F48945DB1DA2A4E6991260687B7E449058BF989\r\n"} 2021-09-17T08:53:27.273Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload//cni/win-overlay.exe"} 2021-09-17T08:53:27.543Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\\\kube-proxy.exe", "out": "True\r\n"} 2021-09-17T08:53:28.032Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\\\kube-proxy.exe -Algorithm SHA256; $out.Hash", "out": "0723D83C610BBAF81CFAD883D5E794C68DEFDE80D87E8711058A691C3A92C3A5\r\n"} 2021-09-17T08:53:28.032Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload//kube-node/kube-proxy.exe"} 2021-09-17T08:53:28.300Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\Temp\\\\wget-ignore-cert.ps1", "out": "True\r\n"} 2021-09-17T08:53:28.719Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\Temp\\\\wget-ignore-cert.ps1 -Algorithm SHA256; $out.Hash", "out": "E17025E45E722614D4A62A425A07886143FE0000A8C22CD4D50F76C1F9690893\r\n"} 2021-09-17T08:53:28.719Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload//powershell/wget-ignore-cert.ps1"} 2021-09-17T08:53:28.987Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\\\win-bridge.exe", "out": "True\r\n"} 2021-09-17T08:53:29.414Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\cni\\\\win-bridge.exe -Algorithm SHA256; $out.Hash", "out": "7927052219FD6EFE27B50FEEFB1F56DEF1B207E0588CB57FFD97BAABF68BC4A0\r\n"} 2021-09-17T08:53:29.414Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload//cni/win-bridge.exe"} 2021-09-17T08:53:29.672Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\Temp\\\\hns.psm1", "out": "True\r\n"} 2021-09-17T08:53:30.087Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\Temp\\\\hns.psm1 -Algorithm SHA256; $out.Hash", "out": "F8EE141F43B66539E4809418EB5D57A31932FCF8A2853636C31C52DAD030419B\r\n"} 2021-09-17T08:53:30.087Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload//powershell/hns.psm1"} 2021-09-17T08:53:30.353Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\\\flannel.exe", "out": "True\r\n"} 2021-09-17T08:53:30.774Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\cni\\\\flannel.exe -Algorithm SHA256; $out.Hash", "out": "ADD50339E3CFA59F4367D0E71B111120A62644062798794D894C816B47483E15\r\n"} 2021-09-17T08:53:30.774Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload//cni/flannel.exe"} 2021-09-17T08:53:31.035Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\\\host-local.exe", "out": "True\r\n"} 2021-09-17T08:53:31.463Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\cni\\\\host-local.exe -Algorithm SHA256; $out.Hash", "out": "C78D3E5C5731C19BA18BE4B87E7AEA15E5A672A562D6BFDB690A37F79AA265EF\r\n"} 2021-09-17T08:53:31.463Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/payload//cni/host-local.exe"} 2021-09-17T08:53:31.486Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe qc windows_exporter", "out": "[SC] QueryServiceConfig SUCCESS\r\n\r\nSERVICE_NAME: windows_exporter\r\n TYPE : 10 WIN32_OWN_PROCESS \r\n START_TYPE : 2 AUTO_START\r\n ERROR_CONTROL : 1 NORMAL\r\n BINARY_PATH_NAME : C:\\k\\windows_exporter.exe --collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info\r\n LOAD_ORDER_GROUP : \r\n TAG : 0\r\n DISPLAY_NAME : windows_exporter\r\n DEPENDENCIES : \r\n SERVICE_START_NAME : LocalSystem\r\n"} 2021-09-17T08:53:31.509Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query windows_exporter", "out": "\r\nSERVICE_NAME: windows_exporter \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:53:32.378Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe start windows_exporter", "out": "\r\nSERVICE_NAME: windows_exporter \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 2 START_PENDING \r\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x7d0\r\n PID : 1060\r\n FLAGS : \r\n"} 2021-09-17T08:53:32.378Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured {"service": "windows_exporter", "args": "--collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info\""} 2021-09-17T08:53:33.044Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass C:\\Temp\\wget-ignore-cert.ps1 -server https://api-int.sgao-b49.qe.devcluster.openshift.com:22623/config/worker -output C:\\Windows\\Temp\\worker.ign -acceptHeader application/vnd.coreos.ignition+json`;version=3.1.0", "out": ""} 2021-09-17T08:54:04.610Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass C:\\k\\\\wmcb.exe initialize-kubelet --ignition-file C:\\Windows\\Temp\\worker.ign --kubelet-path C:\\k\\kubelet.exe --node-ip=10.0.49.147", "out": "Bootstrapping completed successfully"} 2021-09-17T08:54:04.610Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured kubelet {"cmd": "C:\\k\\\\wmcb.exe initialize-kubelet --ignition-file C:\\Windows\\Temp\\worker.ign --kubelet-path C:\\k\\kubelet.exe --node-ip=10.0.49.147", "output": "Bootstrapping completed successfully"} 2021-09-17T08:54:34.628Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configure {"service": "hybrid-overlay-node", "args": "--node sgao-win1 --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log\" depend= kubelet"} 2021-09-17T08:54:34.654Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe qc hybrid-overlay-node", "out": "[SC] QueryServiceConfig SUCCESS\r\n\r\nSERVICE_NAME: hybrid-overlay-node\r\n TYPE : 10 WIN32_OWN_PROCESS \r\n START_TYPE : 2 AUTO_START\r\n ERROR_CONTROL : 1 NORMAL\r\n BINARY_PATH_NAME : C:\\k\\hybrid-overlay-node.exe --node sgao-win1 --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log\r\n LOAD_ORDER_GROUP : \r\n TAG : 0\r\n DISPLAY_NAME : hybrid-overlay-node\r\n DEPENDENCIES : kubelet\r\n SERVICE_START_NAME : LocalSystem\r\n"} 2021-09-17T08:54:34.677Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query hybrid-overlay-node", "out": "\r\nSERVICE_NAME: hybrid-overlay-node \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 4 RUNNING \r\n (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:54:34.700Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query hybrid-overlay-node", "out": "\r\nSERVICE_NAME: hybrid-overlay-node \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 4 RUNNING \r\n (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:56:35.899Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Get-HnsNetwork", "out": "\r\n\r\nActivityId : 54B65D47-6BE9-48C4-B145-AA86BBADBCC2\r\nAdditionalParams : \r\nCurrentEndpointCount : 0\r\nExtensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering \r\n Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=False; Name=Microsoft Azure \r\n VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; \r\n Name=Microsoft NDIS Capture}}\r\nFlags : 0\r\nHealth : @{AddressNotificationMissedCount=0; AddressNotificationSequenceNumber=0; \r\n InterfaceNotificationMissedCount=0; InterfaceNotificationSequenceNumber=0; LastErrorCode=0; \r\n LastUpdateTime=132763419822202348; RouteNotificationMissedCount=0; \r\n RouteNotificationSequenceNumber=0}\r\nID : 784735AF-170C-429D-81C9-AD155465183A\r\nIPv6 : False\r\nLayeredOn : 2DE3312D-C462-4485-BABD-C540871A24F1\r\nMacPools : {@{EndMacAddress=00-15-5D-3D-3F-FF; StartMacAddress=00-15-5D-3D-30-00}}\r\nMaxConcurrentEndpoints : 0\r\nName : nat\r\nNatName : ICSA73B71D1-2056-49FE-87D6-A192B359062C\r\nPolicies : {}\r\nResources : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=; \r\n ID=54B65D47-6BE9-48C4-B145-AA86BBADBCC2; PortOperationTime=0; State=1; SwitchOperationTime=0; \r\n VfpOperationTime=0; parentId=6819C11A-B854-4192-9334-B768B63C42AA}\r\nState : 1\r\nSubnets : {@{AdditionalParams=; AddressPrefix=172.17.48.0/20; GatewayAddress=172.17.48.1; Health=; \r\n ID=E61F9295-6575-4186-AD97-C15772C891B0; Policies=System.Object[]; State=0}}\r\nTotalEndpoints : 0\r\nType : nat\r\nVersion : 38654705669\r\n\r\nActivityId : 72D40D15-7BC8-4DE9-9F7C-E78089663511\r\nAdditionalParams : \r\nCurrentEndpointCount : 0\r\nDNSServerCompartment : 3\r\nDrMacAddress : 00-15-5D-87-45-3C\r\nExtensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering \r\n Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=True; Name=Microsoft Azure \r\n VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; \r\n Name=Microsoft NDIS Capture}}\r\nFlags : 0\r\nHealth : @{LastErrorCode=0; LastUpdateTime=132763421436980943}\r\nID : E00062A5-251B-439C-961C-EBF9A996229E\r\nIPv6 : False\r\nInterfaceConstraint : @{InterfaceGuid=00000000-0000-0000-0000-000000000000}\r\nLayeredOn : 63D6936C-7220-4B82-9249-7118A7885D20\r\nMacPools : {@{EndMacAddress=00-15-5D-3B-5F-FF; StartMacAddress=00-15-5D-3B-50-00}}\r\nManagementIP : 10.0.49.147\r\nMaxConcurrentEndpoints : 0\r\nName : BaseOVNKubernetesHybridOverlayNetwork\r\nPolicies : {}\r\nResources : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; Health=; \r\n ID=72D40D15-7BC8-4DE9-9F7C-E78089663511; PortOperationTime=0; State=1; SwitchOperationTime=0; \r\n VfpOperationTime=0; parentId=D3CD0E64-F8A8-4781-B509-934EBE2BE4B9}\r\nState : 1\r\nSubnets : {@{AdditionalParams=; AddressPrefix=100.64.0.0/30; GatewayAddress=100.64.0.2; Health=; \r\n ID=C8086622-2F63-4F08-91AB-A464AF1D1DA2; ObjectType=5; Policies=System.Object[]; State=0}}\r\nTotalEndpoints : 0\r\nType : Overlay\r\nVersion : 38654705669\r\n\r\nActivityId : 1E11416B-1C7F-48A3-81CF-B06F27851BBE\r\nAdditionalParams : \r\nAutomaticDNS : True\r\nCurrentEndpointCount : 0\r\nDNSServerCompartment : 4\r\nDrMacAddress : 00-15-5D-87-45-3C\r\nExtensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering \r\n Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=True; Name=Microsoft Azure \r\n VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; \r\n Name=Microsoft NDIS Capture}}\r\nFlags : 8\r\nHealth : @{LastErrorCode=0; LastUpdateTime=132763421550987382}\r\nID : AD6BD317-DE2A-4F8C-8E5D-41970C3BFA48\r\nIPv6 : False\r\nInterfaceConstraint : @{InterfaceGuid=00000000-0000-0000-0000-000000000000}\r\nLayeredOn : 63D6936C-7220-4B82-9249-7118A7885D20\r\nMacPools : {@{EndMacAddress=00-15-5D-01-5F-FF; StartMacAddress=00-15-5D-01-50-00}}\r\nManagementIP : 10.0.49.147\r\nMaxConcurrentEndpoints : 1\r\nName : OVNKubernetesHybridOverlayNetwork\r\nPolicies : {@{Type=HostRoute}, @{DestinationPrefix=10.129.2.0/23; \r\n DistributedRouterMacAddress=0a:58:0a:81:02:03; IsolationId=4097; ProviderAddress=10.0.50.120; \r\n Type=RemoteSubnetRoute}, @{DestinationPrefix=10.128.0.0/23; \r\n DistributedRouterMacAddress=0a:58:0a:80:00:03; IsolationId=4097; ProviderAddress=10.0.57.112; \r\n Type=RemoteSubnetRoute}, @{DestinationPrefix=10.130.0.0/23; \r\n DistributedRouterMacAddress=0a:58:0a:82:00:03; IsolationId=4097; ProviderAddress=10.0.61.202; \r\n Type=RemoteSubnetRoute}...}\r\nResources : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; Health=; \r\n ID=1E11416B-1C7F-48A3-81CF-B06F27851BBE; PortOperationTime=0; State=1; SwitchOperationTime=0; \r\n VfpOperationTime=0; parentId=D3CD0E64-F8A8-4781-B509-934EBE2BE4B9}\r\nState : 1\r\nSubnets : {@{AdditionalParams=; AddressPrefix=10.132.0.0/24; GatewayAddress=10.132.0.1; Health=; \r\n ID=8E1A0F3C-69E6-4EC7-B2A4-64EBD67E30A5; ObjectType=5; Policies=System.Object[]; State=0}}\r\nTotalEndpoints : 1\r\nType : Overlay\r\nVersion : 38654705669\r\n\r\n\r\n\r\n"} 2021-09-17T08:56:35.899Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured {"service": "hybrid-overlay-node", "args": "--node sgao-win1 --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log\" depend= kubelet"} 2021-09-17T08:56:52.359Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass Test-Path C:\\k\\cni\\config\\\\cni.conf", "out": "True\r\n"} 2021-09-17T08:56:52.970Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass $out = Get-FileHash C:\\k\\cni\\config\\\\cni.conf -Algorithm SHA256; $out.Hash", "out": "53013B58C52277DA648CD12F6A0FB132A2D8097670DBD3AC820D35A7C2C6270F\r\n"} 2021-09-17T08:56:52.971Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal file already exists on VM with expected content {"file": "/tmp/cni790255179/cni.conf"} 2021-09-17T08:57:26.994Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass C:\\k\\wmcb.exe configure-cni --cni-dir=\"C:\\k\\cni\\ --cni-config=\"C:\\k\\cni\\config\\cni.conf", "out": "CNI configuration completed successfully"} 2021-09-17T08:57:26.994Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured kubelet for CNI {"cmd": "C:\\k\\wmcb.exe configure-cni --cni-dir=\"C:\\k\\cni\\ --cni-config=\"C:\\k\\cni\\config\\cni.conf", "output": "CNI configuration completed successfully"} 2021-09-17T08:57:39.406Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Import-Module -DisableNameChecking C:\\Temp\\hns.psm1; $net = (Get-HnsNetwork | where { $_.Name -eq 'OVNKubernetesHybridOverlayNetwork' }); $endpoint = New-HnsEndpoint -NetworkId $net.ID -Name VIPEndpoint; Attach-HNSHostEndpoint -EndpointID $endpoint.ID -CompartmentID 1; (Get-NetIPConfiguration -AllCompartments -All -Detailed | where { $_.NetAdapter.LinkLayerAddress -eq $endpoint.MacAddress }).IPV4Address.IPAddress.Trim()\"", "out": "10.132.0.94\r\n"} 2021-09-17T08:57:39.448Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe qc kube-proxy", "out": "[SC] QueryServiceConfig SUCCESS\r\n\r\nSERVICE_NAME: kube-proxy\r\n TYPE : 10 WIN32_OWN_PROCESS \r\n START_TYPE : 2 AUTO_START\r\n ERROR_CONTROL : 1 NORMAL\r\n BINARY_PATH_NAME : C:\\k\\kube-proxy.exe --windows-service --v=4 --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=sgao-win1 --kubeconfig=c:\\k\\kubeconfig --cluster-cidr=10.132.0.0/24 --log-dir=C:\\var\\log\\kube-proxy\\ --logtostderr=false --network-name=OVNKubernetesHybridOverlayNetwork --source-vip=10.132.0.14 --enable-dsr=false\r\n LOAD_ORDER_GROUP : \r\n TAG : 0\r\n DISPLAY_NAME : kube-proxy\r\n DEPENDENCIES : hybrid-overlay-node\r\n SERVICE_START_NAME : LocalSystem\r\n"} 2021-09-17T08:57:39.494Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe query kube-proxy", "out": "\r\nSERVICE_NAME: kube-proxy \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 1 STOPPED \r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x0\r\n"} 2021-09-17T08:57:40.249Z DEBUG wc ip-10-0-49-147.us-east-2.compute.internal run {"cmd": "sc.exe start kube-proxy", "out": "\r\nSERVICE_NAME: kube-proxy \r\n TYPE : 10 WIN32_OWN_PROCESS \r\n STATE : 2 START_PENDING \r\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\r\n WIN32_EXIT_CODE : 0 (0x0)\r\n SERVICE_EXIT_CODE : 0 (0x0)\r\n CHECKPOINT : 0x0\r\n WAIT_HINT : 0x7d0\r\n PID : 4320\r\n FLAGS : \r\n"} 2021-09-17T08:57:40.249Z INFO wc ip-10-0-49-147.us-east-2.compute.internal configured {"service": "kube-proxy", "args": "--windows-service --v=4 --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=sgao-win1 --kubeconfig=c:\\k\\kubeconfig --cluster-cidr=10.132.0.0/24 --log-dir=C:\\var\\log\\kube-proxy\\ --logtostderr=false --network-name=OVNKubernetesHybridOverlayNetwork --source-vip=10.132.0.94 --enable-dsr=false\" depend= hybrid-overlay-node"} 2021-09-17T08:57:55.269Z INFO nc ip-10-0-49-147.us-east-2.compute.internal instance has been configured as a worker node {"version": "3.1.0+8ffe65a"} 2021-09-17T08:57:55.269Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"openshift-windows-machine-config-operator","name":"windows-instances","apiVersion":"v1"}, "reason": "InstanceSetup", "message": "Configured instance with address ip-10-0-49-147.us-east-2.compute.internal as a worker node"} 2021-09-17T08:57:55.280Z INFO metrics Prometheus configured {"endpoints": "windows-exporter", "port": 9182, "name": "metrics"}
Since the initial bug has been fixed, I'll mark this bug as VERIFIED and opened a new BZ 2005360 to track issue in Comment 5, thanks.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Windows Container Support for Red Hat OpenShift 5.0.0 [security update]), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0577