Description of problem:
Compute instances on GCP are not passing load balancer health checks because the route table in the VM is not getting modified properly

Version-Release number of selected component (if applicable):
GCP image: rhcos-420-8-20190611-0


How reproducible:
Always

Steps to Reproduce:
1. Create a compute instance on GCP that listens on a specific port (6443 in the case of the bootstrap machine). 
2. Create a load balancer that points to the instance with a TCP health check for that port.
3. Wait for the load balancer to report the instance as healthy. 

Actual results:
The instance never reports as healthy

Expected results:
The instance reports as healthy because the port is open.

Additional info:

On the machine, this is the status of the gcp-routes systemd unit:

# systemctl status gcp-routes
● gcp-routes.service - Update GCP routes for forwarded IPs.
   Loaded: loaded (/usr/lib/systemd/system/gcp-routes.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-07-12 16:29:38 UTC; 11min ago
 Main PID: 1215 (bash)
    Tasks: 2 (limit: 26213)
   Memory: 3.5M
   CGroup: /system.slice/gcp-routes.service
           ├─1215 /bin/bash /sbin/gcp-routes.sh
           └─6697 sleep 30

Jul 12 16:39:10 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Error: any valid prefix is expected rather than "10.0.0.634.74.167.161".
Jul 12 16:39:40 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 10.0.0.6
Jul 12 16:39:40 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 34.74.167.161
Jul 12 16:39:40 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Error: any valid prefix is expected rather than "10.0.0.634.74.167.161".
Jul 12 16:40:10 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 10.0.0.6
Jul 12 16:40:10 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 34.74.167.161
Jul 12 16:40:10 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Error: any valid prefix is expected rather than "10.0.0.634.74.167.161".
Jul 12 16:40:40 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 10.0.0.6
Jul 12 16:40:40 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 34.74.167.161
Jul 12 16:40:40 cewong-n2lhl-bootstrap.c.openshift-dev-installer.internal bash[1215]: Error: any valid prefix is expected rather than "10.0.0.634.74.167.161".


The route table does not show the external ip:

# ip route list table local
local 10.0.0.5 dev ens4 proto kernel scope host src 10.0.0.5
broadcast 10.0.0.5 dev ens4 proto kernel scope link src 10.0.0.5
broadcast 10.88.0.0 dev cni0 proto kernel scope link src 10.88.0.1 linkdown
local 10.88.0.1 dev cni0 proto kernel scope host src 10.88.0.1
broadcast 10.88.255.255 dev cni0 proto kernel scope link src 10.88.0.1 linkdown
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1

This is a run of /sbin/gcp-routes.sh with bash debugging:
# bash -x /sbin/gcp-routes.sh
+ declare -A routes
+ :
+ run
+ net_path=network-interfaces/
++ curler network-interfaces/
++ curl --silent -L -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/
+ for vif in $(curler ${net_path})
++ curler network-interfaces/0/mac
++ curl --silent -L -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/mac
+ hw_addr=42:01:0a:00:00:05
+ fwip_path=network-interfaces/0/forwarded-ips/
++ get_ifname 42:01:0a:00:00:05
++ sysfs_path=/sys/class/net
+++ find /sys/class/net -maxdepth 1 -mindepth 1
++ for dev in $(find ${sysfs_path} -maxdepth 1  -mindepth 1)
++ local mac=42:01:0a:00:00:05
+++ basename /sys/class/net/ens4
++ local name=ens4
++ '[' 42:01:0a:00:00:05 == 42:01:0a:00:00:05 ']'
++ echo ens4
++ return
+ dev_name=ens4
++ curler network-interfaces/0/forwarded-ips/
++ curl --silent -L -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/forwarded-ips/
+ for level in $(curler ${fwip_path})
++ curler network-interfaces/0/forwarded-ips//0
++ curl --silent -L -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/forwarded-ips//0
+ for fwip in $(curler ${fwip_path}/${level})
+ echo 'Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 10.0.0.6'
Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 10.0.0.6
+ routes[$dev_name]+=10.0.0.6
+ for level in $(curler ${fwip_path})
++ curler network-interfaces/0/forwarded-ips//1
++ curl --silent -L -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/forwarded-ips//1
+ for fwip in $(curler ${fwip_path}/${level})
+ echo 'Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 34.74.167.161'
Processing route for NIC 0/42:01:0a:00:00:05 as ens4 for 34.74.167.161
+ routes[$dev_name]+=34.74.167.161
+ set_routes ens4
+ local dev=ens4
+ read -a dev_routes
++ ip route show dev ens4 table local proto 66
++ awk '{print$2}'
+ for route in ${dev_routes[@]}
+ ip route replace to local 10.0.0.634.74.167.161 dev ens4 proto 66
Error: any valid prefix is expected rather than "10.0.0.634.74.167.161".
+ unset dev_routes
+ routes[$dev_name]=
+ unset hw_addr
+ unset fwip_path
+ unset dev_name
+ sleep 30