Bug 1823579 - WMCO won't start as OVN hybrid naming has changed
Summary: WMCO won't start as OVN hybrid naming has changed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Windows Containers
Version: 4.5
Hardware: All
OS: All
unspecified
urgent
Target Milestone: ---
: 4.5.0
Assignee: ravig
QA Contact: gaoshang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-13 23:04 UTC by ravig
Modified: 2020-07-13 17:27 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1823919 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:27:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift windows-machine-config-operator pull 38 0 None closed Bug 1823579: [ci] React to hybrid overlay changes upstream 2020-06-23 10:01:19 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:27:45 UTC

Description ravig 2020-04-13 23:04:09 UTC
Description of problem:

Because of this commit https://github.com/openshift/ovn-kubernetes/commit/2563e6b82827726cd7850ed2824346a6ae5ba5eb#diff-86eeef0b956d450bd2a787067b7443fdR26 in onv-kubernetes repo, the hybrid-overlay binary is no longer called hybrid-overlay.exe but it is called hybrid-overlay-node.exe. We've a hardcoding to hybrid-overlay.exe in the following locations:

    Dockerfile
    Dockefile.ci
    wellknownlocations

And there are other places where it is mentioned in code comments.

The name of this binary needs to be changed for the operator to start working. Otherwise the CI will keep failing.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 gaoshang 2020-04-17 08:36:03 UTC
This bug has been verified and passed, thanks.

Versions:
4.5.0-0.nightly-2020-04-16-233800
windows-machine-config-operator commit 6becd7c1ac616f14c6d91b0475113877fea510d7

Steps to Reproduce:
1, build wmco operator image succeeded
# operator-sdk build quay.io/sgaoshang/wmco:latest
...
INFO[0124] Operator build complete.

2, check payload folder in image, hybrid-overlay.exe change to hybrid-overlay-node.exe
# tree /payload/
/payload/
├── cni-plugins
│   ├── flannel.exe
│   ├── host-local.exe
│   ├── win-bridge.exe
│   └── win-overlay.exe
├── hybrid-overlay-node.exe
├── kube-node
│   ├── kubelet.exe
│   └── kube-proxy.exe
├── powershell
│   └── wget-ignore-cert.ps1
└── wmcb.exe

3 directories, 9 files

3, run operator locally, check hybrid-overlay set up correctly

# operator-sdk run --local --namespace=windows-machine-config-operator
INFO[0000] Running the operator locally in namespace windows-machine-config-operator. 
{"level":"info","ts":1587110720.6337183,"logger":"cmd","msg":"operator","version":"0.0.1"}
{"level":"info","ts":1587110720.6338465,"logger":"cmd","msg":"go","version":"go1.13.6","os":"linux","arch":"amd64"}
{"level":"info","ts":1587110720.6338556,"logger":"cmd","msg":"operator-sdk","version":"v0.15.2"}
{"level":"info","ts":1587110720.6371543,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1587110720.6371765,"logger":"leader","msg":"Skipping leader election; not running in a cluster."}
{"level":"info","ts":1587110722.88065,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1587110722.8814504,"logger":"cmd","msg":"registering Components."}
{"level":"info","ts":1587110722.8819356,"logger":"cmd","msg":"skipping CR metrics server creation; not running in a cluster."}
{"level":"info","ts":1587110722.8819458,"logger":"cmd","msg":"starting the Cmd."}
{"level":"info","ts":1587110722.8822331,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1587110722.882379,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"windowsmachineconfig-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587110722.982805,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"windowsmachineconfig-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587110723.0831695,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"windowsmachineconfig-controller"}
{"level":"info","ts":1587110723.0832243,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"windowsmachineconfig-controller","worker count":1}
{"level":"info","ts":1587110745.6924896,"logger":"controller_windowsmachineconfig","msg":"reconciling WindowsMachineConfig","Request.Namespace":"windows-machine-config-operator","Request.Name":"instance"}
2020/04/17 04:05:45 kubeconfig source: /root/ocp/aws/cluster/auth/kubeconfig
2020/04/17 04:05:47 Added rule with port 5986 to the security groups of your local IP 
2020/04/17 04:05:47 Added rule with port 22 to the security groups of your local IP 
2020/04/17 04:05:47 Added rule with port 3389 to the security groups of your local IP 
2020/04/17 04:05:47 Using existing Security Group: sg-0ce401d793900c43a
{"level":"info","ts":1587111046.828141,"logger":"windows","msg":"bootstrapper initialization failed","stderr":"{\"level\":\"info\",\"ts\":1587111050.9821312,\"logger\":\"wmcb\",\"msg\":\"Bootstrapping completed successfully\"}\n"}
2020/04/17 04:16:51 error closing ssh client connection: close tcp 10.0.149.126:45066->3.17.13.220:22: use of closed network connection
{"level":"info","ts":1587111427.0358968,"logger":"nodeconfig","msg":"VM has been configured as a worker node","VM ID":"i-0afe4abc9ad5565a3"}


# oc create -f deploy/crds/wmc.openshift.io_v1alpha1_windowsmachineconfig_cr.yaml

# oc get nodes
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-0-140-165.us-east-2.compute.internal   Ready    master   3h58m   v1.18.0-rc.1
ip-10-0-141-223.us-east-2.compute.internal   Ready    worker   3h48m   v1.18.0-rc.1
ip-10-0-156-251.us-east-2.compute.internal   Ready    master   3h58m   v1.18.0-rc.1
ip-10-0-157-12.us-east-2.compute.internal    Ready    worker   3h48m   v1.18.0-rc.1
ip-10-0-167-187.us-east-2.compute.internal   Ready    worker   3h48m   v1.18.0-rc.1
ip-10-0-17-164.us-east-2.compute.internal    Ready    worker   15m     v1.17.3
ip-10-0-175-88.us-east-2.compute.internal    Ready    master   3h58m   v1.18.0-rc.1


# oc describe node ip-10-0-17-164.us-east-2.compute.internal
...
Annotations:        k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 00-15-5D-E5-73-CA
                    k8s.ovn.org/hybrid-overlay-node-subnet: 10.132.0.0/24
                    volumes.kubernetes.io/controller-managed-attach-detach: true

Check in windows node, check overlay network is present

PS C:\Users\Administrator> Get-HnsNetwork

ActivityId             : F3AB77C1-35F2-47BB-8685-0002C145B575
AdditionalParams       :
CurrentEndpointCount   : 0
Extensions             : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering
                         Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=False; Name=Microsoft Azure
                         VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True;
                         Name=Microsoft NDIS Capture}}
Flags                  : 0
Health                 : @{AddressNotificationMissedCount=0; AddressNotificationSequenceNumber=0;
                         InterfaceNotificationMissedCount=0; InterfaceNotificationSequenceNumber=0; LastErrorCode=0;
                         LastUpdateTime=132315844238130860; RouteNotificationMissedCount=0;
                         RouteNotificationSequenceNumber=0}
ID                     : 8C30D5D0-A1F9-43A0-AB61-8041526F0B90
IPv6                   : False
LayeredOn              : A1CF2BA2-921E-4D33-83CE-68A3F72C0638
MacPools               : {@{EndMacAddress=00-15-5D-F7-EF-FF; StartMacAddress=00-15-5D-F7-E0-00}}
MaxConcurrentEndpoints : 0
Name                   : nat
NatName                : ICSBE05EF6A-CB5C-4F4A-8FDC-2A35EBC0ED2F
Policies               : {}
Resources              : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=;
                         ID=F3AB77C1-35F2-47BB-8685-0002C145B575; PortOperationTime=0; State=1; SwitchOperationTime=0;
                         VfpOperationTime=0; parentId=055531EB-EDE4-441E-BBB9-BF16EBB3F2FC}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=172.17.32.0/20; GatewayAddress=172.17.32.1; Health=;
                         ID=0B5243B4-E79E-4293-BF34-F3272B7C8F30; Policies=System.Object[]; State=0}}
TotalEndpoints         : 0
Type                   : nat
Version                : 38654705666

ActivityId             : 3E7FA185-6EDB-4594-97E8-74204FB8779C
AdditionalParams       :
AutomaticDNS           : True
CurrentEndpointCount   : 0
DNSServerCompartment   : 4
DrMacAddress           : 00-15-5D-E5-73-CA
Extensions             : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering
                         Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=True; Name=Microsoft Azure
                         VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True;
                         Name=Microsoft NDIS Capture}}
Flags                  : 8
Health                 : @{LastErrorCode=0; LastUpdateTime=132315847258690514}
ID                     : BBA80D89-39A9-413A-BB03-57EB2E5A24FA
IPv6                   : False
InterfaceConstraint    : @{InterfaceGuid=00000000-0000-0000-0000-000000000000}
LayeredOn              : 5F8A8EC6-4DDF-4428-A158-C630E97DA95D
MacPools               : {@{EndMacAddress=00-15-5D-5E-1F-FF; StartMacAddress=00-15-5D-5E-10-00}}
ManagementIP           : 10.0.17.164
MaxConcurrentEndpoints : 0
Name                   : OVNKubernetesHybridOverlayNetwork
Policies               : {@{Type=HostRoute}, @{DestinationPrefix=10.130.0.0/23;
                         DistributedRouterMacAddress=0a:58:0a:82:00:03; IsolationId=4097;
                         ProviderAddress=10.0.140.165; Type=RemoteSubnetRoute}, @{DestinationPrefix=10.128.2.0/23;
                         DistributedRouterMacAddress=0a:58:0a:80:02:03; IsolationId=4097;
                         ProviderAddress=10.0.141.223; Type=RemoteSubnetRoute}, @{DestinationPrefix=10.128.0.0/23;
                         DistributedRouterMacAddress=0a:58:0a:80:00:03; IsolationId=4097;
                         ProviderAddress=10.0.156.251; Type=RemoteSubnetRoute}...}
Resources              : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; Health=;
                         ID=3E7FA185-6EDB-4594-97E8-74204FB8779C; PortOperationTime=0; State=1; SwitchOperationTime=0;
                         VfpOperationTime=0; parentId=3CEF1B40-1A84-4B34-AA77-69B54A5A5D62}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=10.132.0.0/24; GatewayAddress=10.132.0.1; Health=;
                         ID=789ADCBB-C791-4EE8-8640-FB01FBF3E1A3; ObjectType=5; Policies=System.Object[]; State=0}}
TotalEndpoints         : 0
Type                   : Overlay
Version                : 38654705666

ActivityId             : 3EAD3401-F487-4B54-93FE-6D3A24616C49
AdditionalParams       :
CurrentEndpointCount   : 0
DNSServerCompartment   : 3
DrMacAddress           : 00-15-5D-E5-73-CA
Extensions             : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering
                         Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=True; Name=Microsoft Azure
                         VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True;
                         Name=Microsoft NDIS Capture}}
Flags                  : 0
Health                 : @{LastErrorCode=0; LastUpdateTime=132315847104294515}
ID                     : 8007F101-3461-4E0B-A7FD-D06AF1D7DBBA
IPv6                   : False
InterfaceConstraint    : @{InterfaceGuid=00000000-0000-0000-0000-000000000000}
LayeredOn              : 5F8A8EC6-4DDF-4428-A158-C630E97DA95D
MacPools               : {@{EndMacAddress=00-15-5D-10-5F-FF; StartMacAddress=00-15-5D-10-50-00}}
ManagementIP           : 10.0.17.164
MaxConcurrentEndpoints : 0
Name                   : BaseOVNKubernetesHybridOverlayNetwork
Policies               : {}
Resources              : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; Health=;
                         ID=3EAD3401-F487-4B54-93FE-6D3A24616C49; PortOperationTime=0; State=1; SwitchOperationTime=0;
                         VfpOperationTime=0; parentId=3CEF1B40-1A84-4B34-AA77-69B54A5A5D62}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=100.64.0.0/30; GatewayAddress=100.64.0.2; Health=;
                         ID=25F007BB-D11F-4D41-A44E-EB217474BC19; ObjectType=5; Policies=System.Object[]; State=0}}
TotalEndpoints         : 0
Type                   : Overlay
Version                : 38654705666

Comment 4 errata-xmlrpc 2020-07-13 17:27:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.