Bug 1755073

Description of problem:
UPI fails when installing to a cloud provider.

Version-Release number of the following components:
$ bin/openshift-install version
bin/openshift-install unreleased-master-1780-gf96afb99f1ce4f8976ce62f7df44acb24d2062d6-dirty
built from commit f96afb99f1ce4f8976ce62f7df44acb24d2062d6
release image registry.svc.ci.openshift.org/origin/release:4.2

How reproducible:
Always
Steps to Reproduce:
1. Follow UPI install docs (either repo or official). When creating the install-config, you must select a cloud provider. In my case, I select aws.
2. Watch the install timeout.
3. Check clusteroperator status. You will see auth and console operators fail.

Actual results:
$ oc get clusteroperator/authentication
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication             Unknown     Unknown       True       61m

Expected results:
$ oc get clusteroperator/authentication
NAME             VERSION                         AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication   4.2.0-0.okd-2019-09-23-171417   True        False         False      54m

Additional info:
When a cluster is installed on a cloud platform, the ingress operator creates a service of type loadBalancer to front-end ingress controller pods. These pods can not run on master nodes, b/c Kubernetes excludes masters from target LB pools.

[1] in the UPI docs causes masters to get provisioned with `master,worker` roles. When masters include the worker role, the scheduler will schedule ingress controller pods to masters, causing the ingress controller service to NOT load balance ingress traffic to the ingress controller pods.

The authentication operator uses the oauth route as a health check as part of its bootstrap. The associated ingress has nowhere to forward the traffic because the ingress controller pods are running on master nodes, which are not included in the target LB pool, effectively making ingress useless. The console operator does not start properly b/c it depends on a functional auth-operator.

[1] https://github.com/openshift/installer/blob/master/docs/user/aws/install_upi.md#empty-compute-pools