Bug 2070892 - [WMCO] BYOH instance fails to remove when deleting config-map in Azure
Summary: [WMCO] BYOH instance fails to remove when deleting config-map in Azure
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Windows Containers
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.11.0
Assignee: jvaldes
QA Contact: Jose Luis Franco
URL:
Whiteboard:
Depends On:
Blocks: 2081825
TreeView+ depends on / blocked
 
Reported: 2022-04-01 10:06 UTC by Jose Luis Franco
Modified: 2022-08-23 06:32 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Node external IP addresses without reverse lookup records were used to associate the Windows intance. Consequence: Windows instance de-configuration failed, when Node external IP is present without PTR record Fix: Do not fail if PTR record is not present for the first node address, keep looking among all the node addresses until a reverse lookup record is found. Result: Windows instance de-configuration succeed when node external IP is present without PTR record.
Clone Of:
: 2081825 (view as bug list)
Environment:
Last Closed: 2022-08-23 06:32:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift windows-machine-config-operator pull 1045 0 None Merged Bug 2070892: Skip node addresses without PTR record during instance de-configuration 2022-05-04 13:42:52 UTC
Red Hat Product Errata RHBA-2022:6129 0 None None None 2022-08-23 06:32:48 UTC

Description Jose Luis Franco 2022-04-01 10:06:51 UTC
Must gather logs:

1. Issue: 
The removal of a BYOH instance fails when removing the windows-instances config-map. After removing the config-map, the node is still present in the "oc get nodes" output.
2. WMCO & OpenShift Version
[cloud-user@preserve-jfrancoa ~]$ oc get csv -n openshift-windows-machine-config-operator
NAME                                     DISPLAY                            VERSION     REPLACES   PHASE
elasticsearch-operator.5.4.0-128         OpenShift Elasticsearch Operator   5.4.0-128              Succeeded
windows-machine-config-operator.v5.0.0   Windows Machine Config Operator    5.0.0                  Succeeded
[cloud-user@preserve-jfrancoa ~]$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-03-29-163038   True        False         3h9m    Cluster version is 4.10.0-0.nightly-2022-03-29-163038

3. Platform - Azure (Couldn't confirm on others yet)
4. If the platform is vSphere, what is the VMware tools version?
5. Is it a new test case or an old test case?
   if it is the old test case, is it regression or first-time tested? 
   Is it platform-specific or consistent across all platforms?
This is an old-test which is being reformatted (as it was relying on a machine-set to create the BYOH instance). So, far the test has been performed manually, creating the instance by hand, configuring ssh/ports/docker in the Windows host via PowerShell script, adding the internal IP in the config-map and applying it. But the next step is to automate all of it.
6. Steps to Reproduce
  1. Create an IPI cluster on azure
  2. Install WMCO 5.0.0
  3. Create a machineset for windows instances (2 replicas) and apply it
  4. State at this point:
[cloud-user@preserve-jfrancoa ~]$ oc get machine -n openshift-machine-api                                                        
NAME                                      PHASE     TYPE              REGION   ZONE   AGE                     
jfrancoa-0104-dc8k4-master-0              Running   Standard_D8s_v3   westus          126m                                              
jfrancoa-0104-dc8k4-master-1              Running   Standard_D8s_v3   westus          126m                 
jfrancoa-0104-dc8k4-master-2              Running   Standard_D8s_v3   westus          126m                                         
jfrancoa-0104-dc8k4-worker-westus-5hxg5   Running   Standard_D4s_v3   westus          118m                                         
jfrancoa-0104-dc8k4-worker-westus-h5862   Running   Standard_D4s_v3   westus          118m                                                                         
jfrancoa-0104-dc8k4-worker-westus-rnzb5   Running   Standard_D4s_v3   westus          118m                                                                                                                                                       
windows-hk67r                             Running   Standard_D2s_v3   westus          88m
windows-v22fh                             Running   Standard_D2s_v3   westus          88m
  5. Create a new Windows Server 2019 VM in the same resource group, add to the same network as the existing cluster.
  6. Configure SSH, Firewall-Rules and install docker with this script:
$authorizedKeyConf = "$env:ProgramData\ssh\administrators_authorized_keys"

Write-Output "PUB_SSH_KEYS(FIELD OMMITED)" | Out-File -FilePath $authorizedKeyConf -Encoding ascii

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
# set service startup type
Set-Service -Name ssh-agent -StartupType 'Automatic'
Set-Service -Name sshd -StartupType 'Automatic'
# start service
Start-Service ssh-agent
Start-Service sshd
# configure key based-authentication
$sshdConfigFilePath = "$env:ProgramData\ssh\sshd_config"
$pubKeyConf = (Get-Content -path $sshdConfigFilePath) -replace '#PubkeyAuthentication yes','PubkeyAuthentication yes'
$pubKeyConf | Set-Content -Path $sshdConfigFilePath
$passwordConf = (Get-Content -path $sshdConfigFilePath) -replace '#PasswordAuthentication yes','PasswordAuthentication yes'
$passwordConf | Set-Content -Path $sshdConfigFilePath
# create key file in configuration

$acl = Get-Acl $authorizedKeyConf
# disable inheritance
$acl.SetAccessRuleProtection($true, $false)
# set full control for Administrators
$administratorsRule = New-Object system.security.accesscontrol.filesystemaccessrule("Administrators","FullControl","Allow")
$acl.SetAccessRule($administratorsRule)
# set full control for SYSTEM
$systemRule = New-Object system.security.accesscontrol.filesystemaccessrule("SYSTEM","FullControl","Allow")
$acl.SetAccessRule($systemRule)
# apply file acl
$acl | Set-Acl
# restart service
Restart-Service sshd
# success

Get-Service -Name sshd
Get-Service -Name ssh-agent

New-NetFirewallRule -DisplayName "ContainerLogsPort" -LocalPort 10250 -Enabled True -Direction Inbound -Protocol TCP -Action Allow -EdgeTraversalPolicy Allow

Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
# configure repository policy
Set-PSRepository PSGallery -InstallationPolicy Trusted
# install module with provider
Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
# install docker package
Install-Package -Name docker -ProviderName DockerMsftProvider -Force

Get-Service -Name docker
exit 0
  7. Restart VM for docker engine configuration
  8. Get internal VM's IP from web-console
  9. Create config-map with IP:
[cloud-user@preserve-jfrancoa ~]$ cat config-map.yml 
kind: ConfigMap
apiVersion: v1
metadata:
  name: windows-instances
  namespace: openshift-windows-machine-config-operator
data:
 10.0.0.5: |- 
      username=capi
  10. Apply config-map and see the instance (windows-byoh) being present in the nodes output:
[cloud-user@preserve-jfrancoa ~]$ oc get nodes                                                                                                                                                                                                                                                                        
NAME                                      STATUS   ROLES    AGE     VERSION
jfrancoa-0104-dc8k4-master-0              Ready    master   3h27m   v1.23.5+1f952b3
jfrancoa-0104-dc8k4-master-1              Ready    master   3h27m   v1.23.5+1f952b3
jfrancoa-0104-dc8k4-master-2              Ready    master   3h27m   v1.23.5+1f952b3
jfrancoa-0104-dc8k4-worker-westus-5hxg5   Ready    worker   3h11m   v1.23.5+1f952b3
jfrancoa-0104-dc8k4-worker-westus-h5862   Ready    worker   3h12m   v1.23.5+1f952b3
jfrancoa-0104-dc8k4-worker-westus-rnzb5   Ready    worker   3h12m   v1.23.5+1f952b3
windows-byoh                              Ready    worker   10m     v1.23.3-2003+e419edff267ffa
windows-hk67r                             Ready    worker   166m    v1.23.3-2003+e419edff267ffa
windows-v22fh                             Ready    worker   159m    v1.23.3-2003+e419edff267ffa

  11. Delete config-map
7. Actual Result and Expected Result
Actual:
windows-byoh instance is still associated to the cluster and is visible in the "oc get nodes" output
Expected:
The byoh instance stops being controlled by the cluster and it is not displayed in the "oc get nodes" output.
8. A possible workaround has been tried? Is there a way to recover from the issue being tried out?
oc delete node windows-byoh
9. Logs
       Must-gather-windows-node-logs(https://github.com/openshift/must-gather/blob/master/collection-scripts/gather_windows_node_logs#L24)
           oc get network.operator cluster -o yaml
           oc logs -f deployment/windows-machine-config-operator -n openshift-windows-machine-config-operator
       Windows MachineSet yaml or windows-instances ConfigMap
           oc get machineset <windows_machineSet_name> -n openshift-machine-api -o yaml
           oc get configmaps <windows_configmap_name> -n <namespace_name> -o yaml

WMCO logs:

{"level":"info","ts":1648805792.5267334,"logger":"controllers.configmap","msg":"processing","instances in":"windows-instances"}
{"level":"info","ts":1648805822.8400867,"logger":"wc 10.0.0.5","msg":"configuring"}
{"level":"info","ts":1648805883.592028,"logger":"wc 10.0.0.5","msg":"transferring files"}
{"level":"info","ts":1648805896.4712582,"logger":"wc 10.0.0.5","msg":"configured","service":"windows_exporter","args":"--collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info"}
{"level":"info","ts":1648805930.1672964,"logger":"wc 10.0.0.5","msg":"configured kubelet","cmd":"C:\\k\\\\wmcb.exe initialize-kubelet --ignition-file C:\\Windows\\Temp\\worker.ign --kubelet-path C:\\k\\kubelet.exe --cluster-dns 172.30.0.10 --platform-type=Azure","output":"Bootstrapping completed successfully"}
{"level":"info","ts":1648805960.2310863,"logger":"wc 10.0.0.5","msg":"configure","service":"hybrid-overlay-node","args":"--node windows-byoh --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log"}
{"level":"info","ts":1648806082.2507439,"logger":"wc 10.0.0.5","msg":"configured","service":"hybrid-overlay-node","args":"--node windows-byoh --k8s-kubeconfig c:\\k\\kubeconfig --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log"}
{"level":"info","ts":1648806130.4519472,"logger":"wc 10.0.0.5","msg":"configured kubelet for CNI","cmd":"C:\\k\\wmcb.exe configure-cni --cni-dir=\"C:\\k\\cni\\ --cni-config=\"C:\\k\\cni\\config\\cni.conf","output":"CNI configuration completed successfully"}
{"level":"info","ts":1648806137.0610793,"logger":"wc 10.0.0.5","msg":"configured","service":"kube-proxy","args":"--windows-service --v=4 --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=windows-byoh --kubeconfig=c:\\k\\kubeconfig --cluster-cidr=10.132.3.0/24 --log-dir=C:\\var\\log\\kube-proxy\\ --logtostderr=false --network-name=OVNKubernetesHybridOverlayNetwork --source-vip=10.132.3.112 --enable-dsr=false"}
{"level":"info","ts":1648806152.1051683,"logger":"nc 10.0.0.5","msg":"instance has been configured as a worker node","version":"5.0.0+3a4029fc"}
{"level":"info","ts":1648806152.1409457,"logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
{"level":"info","ts":1648806182.1820877,"logger":"controllers.configmap","msg":"processing","instances in":"windows-instances"}
{"level":"info","ts":1648806182.1824377,"logger":"controllers.configmap","msg":"instance is up to date","node":"windows-byoh","version":"5.0.0+3a4029fc"}
{"level":"info","ts":1648806182.1987617,"logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
{"level":"info","ts":1648806511.5985012,"logger":"controllers.configmap","msg":"processing","instances in":"windows-instances"}
{"level":"error","ts":1648806511.625849,"logger":"controller.configmap","msg":"Reconciler error","reconciler group":"","reconciler kind":"ConfigMap","name":"windows-instances","namespace":"openshift-windows-machine-config-operator","error":"error removing undesired nodes from cluster: unable to find instance associated with node windows-byoh: failed to lookup DNS for IP 52.234.24.62: lookup 62.24.234.52.in-addr.arpa. on 168.63.129.16:53: no such host","errorVerbose":"lookup 62.24.234.52.in-addr.arpa. on 168.63.129.16:53: no such host\nfailed to lookup DNS for IP 52.234.24.62\ngithub.com/openshift/windows-machine-config-operator/controllers.hasAssociatedInstance\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:245\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).deconfigureInstances\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:213\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).reconcileNodes\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:162\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:136\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1581\nunable to find instance associated with node windows-byoh\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).deconfigureInstances\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:215\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).reconcileNodes\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:162\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:136\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1581\nerror removing undesired nodes from cluster\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).reconcileNodes\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:163\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:136\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1581","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1648806511.631428,"logger":"controllers.configmap","msg":"processing","instances in":"windows-instances"}
<<<<<<<<<<< ERROR LOG KEEPS APPEARING OVER AND OVER >>>>>>>>>>>>>>>>>

 Optional logs:
    Anything that can be useful to debug the issue.

Comment 4 Jose Luis Franco 2022-04-29 11:50:13 UTC
Thanks for confirming @Jose Valdes if you need a reproducer, let me know.

Comment 6 Jose Luis Franco 2022-05-06 08:51:45 UTC
Using self-built wmco-index image using release_4.11 (rebased on f409122):

[cloud-user@preserve-jfrancoa ~]$ oc get configmaps -n openshift-windows-machine-config-operator
NAME                                   DATA   AGE
kube-root-ca.crt                       1      80m
openshift-service-ca.crt               1      80m
windows-instances                      1      6m45s
windows-machine-config-operator-lock   0      80m
windows-services-5.0.0-f409122         2      80m


BYOH node added successfully with external IP:

[cloud-user@preserve-jfrancoa ~]$ oc get nodes -o wide
NAME                                                     STATUS   ROLES    AGE     VERSION                       INTERNAL-IP   EXTERNAL-IP      OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
jfrancoa-0605-azure-q7nrj-master-0                       Ready    master   119m    v1.23.3+d464c70               10.0.0.8      <none>           Red Hat Enterprise Linux CoreOS 411.85.202205040357-0 (Ootpa)   4.18.0-348.23.1.el8_5.x86_64   cri-o://1.24.0-32.rhaos4.11.gitc17baa0.el8
jfrancoa-0605-azure-q7nrj-master-1                       Ready    master   119m    v1.23.3+d464c70               10.0.0.7      <none>           Red Hat Enterprise Linux CoreOS 411.85.202205040357-0 (Ootpa)   4.18.0-348.23.1.el8_5.x86_64   cri-o://1.24.0-32.rhaos4.11.gitc17baa0.el8
jfrancoa-0605-azure-q7nrj-master-2                       Ready    master   120m    v1.23.3+d464c70               10.0.0.6      <none>           Red Hat Enterprise Linux CoreOS 411.85.202205040357-0 (Ootpa)   4.18.0-348.23.1.el8_5.x86_64   cri-o://1.24.0-32.rhaos4.11.gitc17baa0.el8
jfrancoa-0605-azure-q7nrj-worker-southcentralus1-clf65   Ready    worker   108m    v1.23.3+d464c70               10.0.128.6    <none>           Red Hat Enterprise Linux CoreOS 411.85.202205040357-0 (Ootpa)   4.18.0-348.23.1.el8_5.x86_64   cri-o://1.24.0-32.rhaos4.11.gitc17baa0.el8
jfrancoa-0605-azure-q7nrj-worker-southcentralus2-jfqfx   Ready    worker   108m    v1.23.3+d464c70               10.0.128.4    <none>           Red Hat Enterprise Linux CoreOS 411.85.202205040357-0 (Ootpa)   4.18.0-348.23.1.el8_5.x86_64   cri-o://1.24.0-32.rhaos4.11.gitc17baa0.el8
jfrancoa-0605-azure-q7nrj-worker-southcentralus3-b2cwq   Ready    worker   108m    v1.23.3+d464c70               10.0.128.5    <none>           Red Hat Enterprise Linux CoreOS 411.85.202205040357-0 (Ootpa)   4.18.0-348.23.1.el8_5.x86_64   cri-o://1.24.0-32.rhaos4.11.gitc17baa0.el8
windows-6lnz7                                            Ready    worker   78m     v1.23.3-2034+eccb3856381a4e   10.0.128.7    <none>           Windows Server 2019 Datacenter                                  10.0.17763.2803                containerd://1.17.5
windows-byoh                                             Ready    worker   8m13s   v1.23.3-2034+eccb3856381a4e   10.0.128.9    20.225.107.168   Windows Server 2019 Datacenter                                  10.0.17763.2803                containerd://1.17.5
windows-x2zq2                                            Ready    worker   72m     v1.23.3-2034+eccb3856381a4e   10.0.128.8    <none>           Windows Server 2019 Datacenter                                  10.0.17763.2803                containerd://1.17.5

Deleting the windows-instance starts up the deconfiguring of the node after few minutes:
[cloud-user@preserve-jfrancoa ~]$ oc delete configmaps windows-instances -n openshift-windows-machine-config-operator                                    
configmap "windows-instances" deleted

[cloud-user@preserve-jfrancoa ~]$ oc get nodes
NAME                                                     STATUS                        ROLES    AGE    VERSION
jfrancoa-0605-azure-q7nrj-master-0                       Ready                         master   123m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-master-1                       Ready                         master   122m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-master-2                       Ready                         master   123m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-worker-southcentralus1-clf65   Ready                         worker   111m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-worker-southcentralus2-jfqfx   Ready                         worker   111m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-worker-southcentralus3-b2cwq   Ready                         worker   111m   v1.23.3+d464c70
windows-6lnz7                                            Ready                         worker   81m    v1.23.3-2034+eccb3856381a4e
windows-byoh                                             NotReady,SchedulingDisabled   worker   11m    v1.23.3-2034+eccb3856381a4e
windows-x2zq2                                            Ready                         worker   75m    v1.23.3-2034+eccb3856381a4e

Being the node deconfigured after few more minutes:

NAME                                                     STATUS   ROLES    AGE    VERSION
jfrancoa-0605-azure-q7nrj-master-0                       Ready    master   128m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-master-1                       Ready    master   128m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-master-2                       Ready    master   128m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-worker-southcentralus1-clf65   Ready    worker   116m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-worker-southcentralus2-jfqfx   Ready    worker   116m   v1.23.3+d464c70
jfrancoa-0605-azure-q7nrj-worker-southcentralus3-b2cwq   Ready    worker   116m   v1.23.3+d464c70
windows-6lnz7                                            Ready    worker   86m    v1.23.3-2034+eccb3856381a4e
windows-x2zq2                                            Ready    worker   80m    v1.23.3-2034+eccb3856381a4e

wmco logs:

1.6518263053147686e+09  INFO    controllers.configmap   processing      {"instances in": "windows-instances"}
1.6518263053151248e+09  INFO    controllers.configmap   instance is up to date  {"node": "windows-byoh", "version": "5.0.0-f409122"}
1.6518263053152518e+09  DEBUG   events  Normal  {"object": {"kind":"ConfigMap","namespace":"openshift-windows-machine-config-operator","name":"windows-ins
tances","apiVersion":"v1"}, "reason": "InstanceSetup", "message": "Configured instance with address 10.0.128.9 as a worker node"}
1.651826305333492e+09   INFO    metrics Prometheus configured   {"endpoints": "windows-exporter", "port": 9182, "name": "metrics"}
1.651826569685518e+09   DEBUG   controllers.configmap   Reconciling     {"ConfigMap": "openshift-windows-machine-config-operator/windows-instances"}
1.6518265696856031e+09  INFO    controllers.configmap   processing      {"instances in": "windows-instances"}
1.6518265697676702e+09  DEBUG   wc 10.0.128.9   initializing SSH connection
1.6518265802792642e+09  INFO    wc 10.0.128.9   deconfiguring
1.651826580314752e+09   DEBUG   wc 10.0.128.9   run     {"cmd": "sc.exe qc windows_exporter", "out": "[SC] QueryServiceConfig SUCCESS\r\n\r\nSERVICE_NAME:
 windows_exporter\r\n        TYPE               : 10  WIN32_OWN_PROCESS \r\n        START_TYPE         : 2   AUTO_START\r\n        ERROR_CONTROL      : 1 
  NORMAL\r\n        BINARY_PATH_NAME   : C:\\k\\windows_exporter.exe --collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,mem
ory,cpu_info\r\n        LOAD_ORDER_GROUP   : \r\n        TAG                : 0\r\n        DISPLAY_NAME       : windows_exporter\r\n        DEPENDENCIES  
     : \r\n        SERVICE_START_NAME : LocalSystem\r\n"}

....................................................................................
1.6518267515357893e+09  DEBUG   wc 10.0.128.9   run     {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Get-HnsNetwork | where { $_.Name -eq 'BaseOVNKubernetesHybridOverlayNetwork'}\"", "out": ""}
1.651826921407263e+09   DEBUG   wc 10.0.128.9   run     {"cmd": "powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Get-HnsNetwork | where { $_.Name -eq 'OVNKubernetesHybridOverlayNetwork'}\"", "out": ""}
1.6518269214251475e+09  INFO    nc 10.0.128.9   instance has been deconfigured  {"node": "windows-byoh"}
1.651826921434048e+09   DEBUG   events  Normal  {"object": {"kind":"ConfigMap","namespace":"openshift-windows-machine-config-operator","name":"windows-instances","apiVersion":"v1"}, "reason": "InstanceTeardown", "message": "Deconfigured node with addresses [{Hostname windows-byoh} {InternalIP 10.0.128.9} {ExternalIP 20.225.107.168}]"}

Comment 10 errata-xmlrpc 2022-08-23 06:32:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift support for Windows Containers 6.0.0 product release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6129


Note You need to log in before you can comment on or make changes to this bug.