Description of problem:

In current configuration, the keepalived is configured to add the ingress-vip on any node which runs any ingress controller.
When using the router sharding, it causes that the keepalived will not configure the ingress-vip to the node which runs the "default" router, but randomly chooses any node running any (sharding) ingress controllers.
Customer has multiple ingress controller instances running in the cluster for route sharding. 

In the vsphere configuration, the keepalived is running on each host as a static pod. 
The static pod manifest is coming from the machine config.

For the worker machine config, the template of the keepalived configures that the ingress vip should run on the node where the health check script finds router listening on port 1936.

// keepalived.conf.tmpl

~~~
# TODO: Improve this check. The port is assumed to be alive.
# Need to assess what is the ramification if the port is not there.
vrrp_script chk_ingress {
    script "/usr/bin/timeout 0.9 /usr/bin/curl -o /dev/null -Lfs http://localhost:1936/healthz/ready"
    interval 1
    weight 50
}

{{$nonVirtualIP := .NonVirtualIP}}

vrrp_instance {{ .Cluster.Name }}_INGRESS {
    state BACKUP
    interface {{ .VRRPInterface }}
    virtual_router_id {{ .Cluster.IngressVirtualRouterID }}
    priority 40
    advert_int 1
    {{if .EnableUnicast}}
    unicast_src_ip {{.NonVirtualIP}}
    unicast_peer {
        {{range .IngressConfig.Peers}}
        {{if ne $nonVirtualIP .}}{{.}}{{end}}
        {{end}}
    }
    {{end}}
    authentication {
        auth_type PASS
        auth_pass {{ .Cluster.Name }}_ingress_vip
    }
    virtual_ipaddress {
        {{ .Cluster.IngressVIP }}/{{ .Cluster.VIPNetmask }}
    }
    track_script {
        chk_ingress
    }
}
~~~

The script has major flaw in case you use multiple ingress controllers (not only, if you have application running listening port 1936 on hostNetwork under path "/healthz/ready" the keepalived will think it is router).

//Workarounds

The immediate workaround is to remove the other ingress shards so keep-alive runs on the infra nodes hosting the default ingress.

The short workaround is to apply new machine config that removes the keepalived component from machines that do not host the default ingress.

Long term solution would be most likely change in the behavior.


Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.7
Vsphere IPI


How reproducible:
- create cluster with 4 worker nodes on vsphere IPI with keepalived
- create two ingress controllers - default and for sharding (schedule the pods to 2 workers each)
- check on which node the keepalived IP is configured - and confirm if this is the right node
- disable the default ingress controller and check if the IP bounces to the other worker nodes


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: