Another solution is adding 'nodeIP: xxx' in the node-config.yml so that the node is functional.

Created attachment 1263960 [details]
The ip of tun0 is displayed in the first place of VM ip addresses.

I'm not really sure waht the bug is here.  You need to configure the node so that it has its own addresses.  There are several ways to do that (depending upon how you did the install).  The ansible installer has an argument, 'openshift start master' takes the --network-cidr argument.  etc.

@Ben

The problem here is, when we first time setup the env, the SDN will assign network to the node. Let's say 10.128.0.0/23 here, and the tun0 ip is 10.128.0.1

And after that if we restart the node service, the SDN node start up will fail due to the tun0 IP which assigned by SDN is included in the cluster CIDR.

I am not sure how did it happen, but the node address list looks abnormal to me.

status:
  addresses:
  - address: 10.128.0.1
    type: ExternalIP
  - address: fe80::6083:bfff:febf:db75
    type: ExternalIP
  - address: 10.66.146.29
    type: InternalIP
  - address: 2620:52:0:4292:250:56ff:fe9f:b058
    type: InternalIP
  - address: fe80::250:56ff:fe9f:b058
    type: InternalIP
  - address: 172.17.0.1
    type: ExternalIP
  - address: fe80::42:2eff:fe71:6bcd
    type: ExternalIP
  - address: master.c.vsphere.local
    type: Hostname

So... the node is not actually using 10.128.0.1?  It looks to me like that was allocated to the machine.

Can you get me the full output from 'ip a' please?

@bbennett here is the output
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:a0:61:dd brd ff:ff:ff:ff:ff:ff
    inet 10.66.146.232/22 brd 10.66.147.255 scope global dynamic ens192
       valid_lft 13746sec preferred_lft 13746sec
    inet6 2620:52:0:4292:250:56ff:fea0:61dd/64 scope global noprefixroute dynamic 
       valid_lft 2591891sec preferred_lft 604691sec
    inet6 fe80::250:56ff:fea0:61dd/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:4b:a0:c4:0b brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 46:f3:8f:9b:55:71 brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN qlen 1000
    link/ether ea:1d:7b:cb:6a:47 brd ff:ff:ff:ff:ff:ff
32: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master ovs-system state UNKNOWN qlen 1000
    link/ether f6:8e:05:be:52:c2 brd ff:ff:ff:ff:ff:ff
33: tun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN qlen 1000
    link/ether 4e:67:ad:c3:cf:4e brd ff:ff:ff:ff:ff:ff
    inet 10.128.0.1/23 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::4c67:adff:fec3:cf4e/64 scope link 
       valid_lft forever preferred_lft forever

(In reply to Meng Bo from comment #4)
> @Ben
> 
> The problem here is, when we first time setup the env, the SDN will assign
> network to the node. Let's say 10.128.0.0/23 here, and the tun0 ip is
> 10.128.0.1
> 
> And after that if we restart the node service, the SDN node start up will
> fail due to the tun0 IP which assigned by SDN is included in the cluster
> CIDR.
> 
> I am not sure how did it happen, but the node address list looks abnormal to
> me.
> 
> status:
>   addresses:
>   - address: 10.128.0.1
>     type: ExternalIP
>   - address: fe80::6083:bfff:febf:db75
>     type: ExternalIP
>   - address: 10.66.146.29
>     type: InternalIP
>   - address: 2620:52:0:4292:250:56ff:fe9f:b058
>     type: InternalIP
>   - address: fe80::250:56ff:fe9f:b058
>     type: InternalIP
>   - address: 172.17.0.1
>     type: ExternalIP
>   - address: fe80::42:2eff:fe71:6bcd
>     type: ExternalIP
>   - address: master.c.vsphere.local
>     type: Hostname

The vSphere cloud provider code is completely wrong here, IMO.  It should not be returning all IP addresses of all Linux network interfaces on the system, since some of those can reach the outside and others cannot.  vSphere has no idea which ones are which, only the guest OS and it's network setup does.

Furthermore, since nothing in Kubernetes/OpenShift currently supports IPv6, it's quite odd to be sending IPv6 addresess.

Even more, sending *link-local* IPv6 addresses is unlikely to do what people expect, even if Kubernetes supported IPv6.

The relevant code is in the vSphere cloud provider in kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:

	var mvm mo.VirtualMachine
	err = getVirtualMachineManagedObjectReference(ctx, c, vm, "guest.net", &mvm)

	// retrieve VM's ip(s)
	for _, v := range mvm.Guest.Net {
		var addressType api.NodeAddressType
		if i.cfg.Network.PublicNetwork == v.Network {
			addressType = api.NodeExternalIP
		} else {
			addressType = api.NodeInternalIP
		}
		for _, ip := range v.IpAddress {
			api.AddToNodeAddresses(&addrs,
				api.NodeAddress{
					Type:    addressType,
					Address: ip,
				},
			)
		}
	}
	return addrs, nil

It looks like the vSphere API's "guest.net" object is an alias for the "GuestInfo.net" property described here:

http://pubs.vmware.com/vsphere-50/index.jsp#com.vmware.wssdk.apiref.doc_50/vim.vm.GuestInfo.html#field_detail

and that appears to list all NICs and IP addresses on the guest, regardless of whether they got their address from VMWare infrastructure or not.  The GCE and AWS cloud providers don't have this problem, because they know exactly what IP address the VM was provisioned with by the GCE/AWS infrastructure and report that, rather than scraping the machine for them.

Perhaps vSphere cloud provider should be using "guest.ipaddress" instead, which is documented in that same link as "Primary IP address assigned to the guest operating system, if known."?

Can you paste in your cloud-config file that you pass to openshift-node?

What was the value of the 'public-network' key (if any) and did that ever change over the course of your efforts here?

Filed an upstream issue for the vSphere stuff:

https://github.com/kubernetes/kubernetes/issues/44075

Beyond that, openshift-sdn should probably:

1) not pick IPv6 addresses from Node.Status.Addresses

2) do something more intelligent than picking the first IP address from Node.Status.Addresses; but this depends on cloud providers being consistent in what they call ExternalIP and InternalIP, and vSphere currently torpedoes that consistency.

This is a vSphere bug, Dan has opened the issue there.  Any further improvements we make are just minor things that don't fix the main problem.

Dropping the severity so we track the other potential changes.

Part of this is supposedly fixed upstream in kube-1.7 due to https://github.com/kubernetes/kubernetes/pull/43545.  There are apparently other fixes required as well, which supposedly VMWare is working on.  c7a22a588f112dd34ae8dce9b4014ba5510e5575 might be the that second part.

We could potentially backport some of these fixes if PM thinks it's a high enough priority.  The 43545 pull is part of kube 1.7, while "second part" I reference above will be part of kube 1.8.  So both would need backports to OSE 3.6.

OpenShift has now been rebased to Kube 1.7 and that pulls in some upstream VSphere fixes.  If somebody can test OSE 3.7 nightlies and see if that fixes any of these issues, that would be great.

Verified this is fixed in
oc v3.7.0-0.125.0
kubernetes v1.7.0+695f48a16f
features: Basic-Auth GSSAPI Kerberos SPNEGO

Now I have completed removed nodeIP from node-config.yml, removed the node, enabled vSphere cloud provider. After atomic-openshift-node service is restarted, the node can be added to the cluster.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188