Bug 1393083

Description of problem:
Can not ping node tunnel’s ip address after updating hostSubnetLength

Version-Release number of selected component (if applicable):
oc v3.4.0.23+24b1a58
kubernetes v1.4.0+776c994
Server https://dhcp-41-65.bos.redhat.com:8443
openshift v3.4.0.23+24b1a58
kubernetes v1.4.0+776c994


How reproducible:
Most time.

Steps to Reproduce:
1. Cluster start up with below configuration:
clusterNetworkCIDR: 10.1.0.0/16
hostSubnetLength: 8

2. Check both tunnel’s ip address in both master and node
master: 10.1.1.1/24
node: 10.1.0.1/24

3. From master ping node tunnel’s ip address passed:
[root@dhcp-41-65 master]# ping 10.1.0.1
PING 10.1.0.1 (10.1.0.1) 56(84) bytes of data.
64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=0.992 ms
64 bytes from 10.1.0.1: icmp_seq=2 ttl=64 time=0.370 ms
[root@dhcp-41-65 master]# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
10.1.0.1                 ether   06:56:71:ed:44:96   C                     tun0
dhcp-41-64.bos.redhat.c  ether   52:54:00:c5:bc:5d   C                     eth0
gateway                  ether   40:b4:f0:c4:e0:81   C                     eth0
dhcp-41-151.bos.redhat.  ether   50:7b:9d:27:b4:65   C                     eth0

4. Follow steps from https://github.com/openshift/openshift-docs/pull/3112 to change clusterNetworkCIDR and hostsubnetlength.
clusterNetworkCIDR: 10.1.0.0/16
hostSubnetLength: 6

5. Check both tunnel’s ip address in both master and node
master: 10.1.0.1/26
node: 10.1.1.1/26

6. From master ping node tunnel’s ip address failed:
[root@dhcp-41-65 master]# ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
From 10.1.0.1 icmp_seq=1 Destination Host Unreachable
From 10.1.0.1 icmp_seq=2 Destination Host Unreachable
From 10.1.0.1 icmp_seq=3 Destination Host Unreachable
From 10.1.0.1 icmp_seq=4 Destination Host Unreachable
From 10.1.0.1 icmp_seq=5 Destination Host Unreachable
From 10.1.0.1 icmp_seq=6 Destination Host Unreachable
[root@dhcp-41-65 master]# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
10.1.1.1                         (incomplete)                              tun0
dhcp-41-64.bos.redhat.c  ether   52:54:00:c5:bc:5d   C                     eth0
gateway                  ether   40:b4:f0:c4:e0:81   C                     eth0
dhcp-41-151.bos.redhat.  ether   50:7b:9d:27:b4:65   C                     eth0
[root@dhcp-41-65 master]# 

7. Rebooting both master and node will recover ping and arp table.
[root@dhcp-41-65 master]# reboot
PolicyKit daemon disconnected from the bus.
We are no longer a registered authentication agent.
Connection to 10.18.41.65 closed by remote host.
Connection to 10.18.41.65 closed.
[weliang@dhcp-41-24 ~]$ ssh root.41.65
root.41.65's password: 
Last login: Tue Nov  8 14:59:48 2016 from dhcp-41-151.bos.redhat.com
***************************
***************************
** weliang-rhel72-vm1
***************************
***************************
[root@dhcp-41-65 ~]# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
gateway                  ether   40:b4:f0:c4:e0:81   C                     eth0
dhcp-41-151.bos.redhat.  ether   50:7b:9d:27:b4:65   C                     eth0
dhcp-41-64.bos.redhat.c  ether   52:54:00:c5:bc:5d   C                     eth0
[root@dhcp-41-65 ~]# ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=1.08 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=64 time=0.219 ms
64 bytes from 10.1.1.1: icmp_seq=3 ttl=64 time=0.319 ms
64 bytes from 10.1.1.1: icmp_seq=4 ttl=64 time=0.382 ms
^C
--- 10.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.219/0.501/1.084/0.341 ms
[root@dhcp-41-65 ~]# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
gateway                  ether   40:b4:f0:c4:e0:81   C                     eth0
10.1.1.1                 ether   da:c2:03:50:47:6b   C                     tun0
dhcp-41-151.bos.redhat.  ether   50:7b:9d:27:b4:65   C                     eth0
dhcp-41-64.bos.redhat.c  ether   52:54:00:c5:bc:5d   C                     eth0
[root@dhcp-41-65 ~]# 

Actual results:
Can not ping node tunnel’s ip address after updating hostSubnetLength

Expected results:
Still can ping node tunnel’s ip address after updating hostSubnetLength

Additional info: