Description of problem:

The router pod should, like all CVO managed components without a special exception, tolerate masters, but it does not.

Version-Release number of selected component (if applicable):

$ KUBECONFIG=wking/auth/kubeconfig oc adm release info --commits | grep ingress
  cluster-ingress-operator                      https://github.com/openshift/cluster-ingress-operator                      9478e28af89922fa4d54389b1ae8ae6fafb2662b

How reproducible:

Every time.

Steps to Reproduce:

1. Break your Machine API provider, e.g. by running libvirt with a non-standard volume pool before [1] lands.
2. Launch a cluster.
3. Wait for things to stabilize.

Then:

$ oc get pods --all-namespaces | grep Pending
openshift-ingress                            router-default-7688479d99-nbnj8                            0/1       Pending     0          31m
openshift-monitoring                         prometheus-operator-647d84b5c6-rsplb                       0/1       Pending     0          31m
openshift-operator-lifecycle-manager         olm-operators-sf5sm                                        0/1       Pending     0          36m
$ oc get pod -o "jsonpath={.status.conditions}{'\n'}" -n openshift-ingress router-default-7688479d99-nbnj8
[map[type:PodScheduled status:False lastProbeTime:<nil> lastTransitionTime:2019-01-30T20:00:04Z reason:Unschedulable message:0/1 nodes are available: 1 node(s) didn't match node selector.]]
$ oc get -o yaml deployment -n openshift-ingress router-default
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: 2019-01-30T20:00:04Z
  generation: 1
  labels:
    app: router
  name: router-default
  namespace: openshift-ingress
  resourceVersion: "12646"
  selfLink: /apis/extensions/v1beta1/namespaces/openshift-ingress/deployments/router-default
  uid: a2a9a529-24c9-11e9-8d1a-52fdfc072182
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: router
      router: router-default
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: router
        router: router-default
    spec:
      containers:
      - env:
        - name: STATS_PORT
          value: "1936"
        - name: ROUTER_SERVICE_NAMESPACE
          value: openshift-ingress
        - name: DEFAULT_CERTIFICATE_DIR
          value: /etc/pki/tls/private
        - name: ROUTER_SERVICE_NAME
          value: default
        - name: ROUTER_CANONICAL_HOSTNAME
          value: apps.wking.installer.testing
        image: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-01-30-150036@sha256:6991fb24697317cb8a1b8a4cfd129d77d05a199f382a4c5ba7eae7ad55bb386b
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            host: localhost
            path: /healthz
            port: 1936
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: router
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          name: https
          protocol: TCP
        - containerPort: 1936
          hostPort: 1936
          name: stats
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            host: localhost
            path: /healthz
            port: 1936
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/pki/tls/private
          name: default-certificate
          readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      nodeSelector:
        node-role.kubernetes.io/worker: ""
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: router
      serviceAccountName: router
      terminationGracePeriodSeconds: 30
      volumes:
      - name: default-certificate
        secret:
          defaultMode: 420
          secretName: router-certs-default
status:
  conditions:
  - lastTransitionTime: 2019-01-30T20:00:04Z
    lastUpdateTime: 2019-01-30T20:00:04Z
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: 2019-01-30T20:10:05Z
    lastUpdateTime: 2019-01-30T20:10:05Z
    message: ReplicaSet "router-default-7688479d99" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

Actual results:

Pending pod with "0/1 nodes are available: 1 node(s) didn't match node selector."

Expected results:

A running pod.

Additional info:

"high" severity is based on Clayton's request [2].

[1]: https://github.com/openshift/cluster-api-provider-libvirt/pull/45
[2]: https://github.com/openshift/installer/pull/1146#issuecomment-459037176