Description of problem:
-----------------------
OVN migration to second interface failed due to ovnkube-node pod failing

oc get po -n openshift-ovn-kubernetes -o wide | awk /openshift-master-1/
ovnkube-master-5sw2m   6/6     Running   12               109m   10.1.208.21   openshift-master-1   <none>           <none>
ovnkube-node-b6lmg     3/4     Running   13 (3m57s ago)   109m   10.1.208.21   openshift-master-1   <none>           <none>

oc get po -n openshift-ovn-kubernetes -o wide | awk /openshift-master-1/
ovnkube-master-5sw2m   6/6     Running   12               109m   10.1.208.21   openshift-master-1   <none>           <none>
ovnkube-node-b6lmg     3/4     Running   13 (3m57s ago)   109m   10.1.208.21   openshift-master-1   <none>           <none>
[kni ~]$ oc describe po ovnkube-node-b6lmg -n openshift-ovn-kubernetes
Name:                 ovnkube-node-b6lmg
Namespace:            openshift-ovn-kubernetes
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 openshift-master-1/10.1.208.21
Start Time:           Thu, 02 Dec 2021 06:46:47 -0500
Labels:               app=ovnkube-node
                      component=network
                      controller-revision-hash=5895968979
                      kubernetes.io/os=linux
                      openshift.io/component=network
                      pod-template-generation=1
                      type=infra
Annotations:          networkoperator.openshift.io/ip-family-mode: single-stack
Status:               Running
IP:                   10.1.208.21
IPs:
  IP:           10.1.208.21
Controlled By:  DaemonSet/ovnkube-node
Containers:
  ovn-controller:
    Container ID:  cri-o://68889fc9266ce191823569e745c5001e37d9c7929726d6723422532764bf2c31
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
      set -e
      if [[ -f "/env/${K8S_NODE}" ]]; then
        set -o allexport
        source "/env/${K8S_NODE}"
        set +o allexport
      fi

      echo "$(date -Iseconds) - starting ovn-controller"
      exec ovn-controller unix:/var/run/openvswitch/db.sock -vfile:off \
        --no-chdir --pidfile=/var/run/ovn/ovn-controller.pid \
        --syslog-method="null" \
        --log-file=/var/log/ovn/acl-audit-log.log \
        -vFACILITY:"local0" \
        -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt \
        -vconsole:"${OVN_LOG_LEVEL}" -vconsole:"acl_log:off" \
        -vPATTERN:console:"%D{%Y-%m-%dT%H:%M:%S.###Z}|%05N|%c%T|%p|%m" \
        -vsyslog:"acl_log:info" \
        -vfile:"acl_log:info"

    State:          Running
      Started:      Thu, 02 Dec 2021 07:36:17 -0500
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:     10m
      memory:  300Mi
    Environment:
      OVN_LOG_LEVEL:  info
      K8S_NODE:        (v1:spec.nodeName)
    Mounts:
      /dev/log from log-socket (rw)
      /env from env-overrides (rw)
      /etc/openvswitch from etc-openvswitch (rw)
      /etc/ovn/ from etc-openvswitch (rw)
      /ovn-ca from ovn-ca (rw)
      /ovn-cert from ovn-cert (rw)
      /run/openvswitch from run-openvswitch (rw)
      /run/ovn/ from run-ovn (rw)
      /var/lib/openvswitch from var-lib-openvswitch (rw)
      /var/log/ovn from node-log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vk2gn (ro)
  ovn-acl-logging:
    Container ID:  cri-o://16092f57da4eb0329c0978a6a8ec905243e38d0622aae9c397ba1b2d026b16f7
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
      set -euo pipefail

      # Rotate audit log files when then get to max size (in bytes)
      MAXFILESIZE=$(( "50"*1000000 ))
      LOGFILE=/var/log/ovn/acl-audit-log.log
      CONTROLLERPID=$(cat /run/ovn/ovn-controller.pid)

      # Redirect err to null so no messages are shown upon rotation
      tail -F ${LOGFILE} 2> /dev/null &

      while true
      do
        # Make sure ovn-controller's logfile exists, and get current size in bytes
        if [ -f "$LOGFILE" ]; then
          file_size=`du -b ${LOGFILE} | tr -s '\t' ' ' | cut -d' ' -f1`
        else
          ovs-appctl -t /var/run/ovn/ovn-controller.${CONTROLLERPID}.ctl vlog/reopen
          file_size=`du -b ${LOGFILE} | tr -s '\t' ' ' | cut -d' ' -f1`
        fi

        if [ $file_size -gt $MAXFILESIZE ];then
          echo "Rotating OVN ACL Log File"
          timestamp=`date '+%Y-%m-%dT%H-%M-%S'`
          mv ${LOGFILE} /var/log/ovn/acl-audit-log.$timestamp.log
          ovs-appctl -t /run/ovn/ovn-controller.${CONTROLLERPID}.ctl vlog/reopen
          CONTROLLERPID=$(cat /run/ovn/ovn-controller.pid)
        fi

        # sleep for 30 seconds to avoid wasting CPU
        sleep 30
      done
    State:          Running
      Started:      Thu, 02 Dec 2021 07:36:17 -0500
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:        10m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /run/ovn/ from run-ovn (rw)
      /var/log/ovn from node-log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vk2gn (ro)
  kube-rbac-proxy:
    Container ID:  cri-o://df3da787138c488d1dfa74c998d1102a62da28d25a4de93a55246dc3cf4e27a7
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2149a0469fb553d83418d6ff5f4430fafab71c594deaf789376167cfeed9cc
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2149a0469fb553d83418d6ff5f4430fafab71c594deaf789376167cfeed9cc
    Port:          9103/TCP
    Host Port:     9103/TCP
    Command:
      /bin/bash
      -c
      #!/bin/bash
      set -euo pipefail
      TLS_PK=/etc/pki/tls/metrics-cert/tls.key
      TLS_CERT=/etc/pki/tls/metrics-cert/tls.crt
      # As the secret mount is optional we must wait for the files to be present.
      # The service is created in monitor.yaml and this is created in sdn.yaml.
      # If it isn't created there is probably an issue so we want to crashloop.
      retries=0
      TS=$(date +%s)
      WARN_TS=$(( ${TS} + $(( 20 * 60)) ))
      HAS_LOGGED_INFO=0

      log_missing_certs(){
          CUR_TS=$(date +%s)
          if [[ "${CUR_TS}" -gt "WARN_TS"  ]]; then
            echo $(date -Iseconds) WARN: ovn-node-metrics-cert not mounted after 20 minutes.
          elif [[ "${HAS_LOGGED_INFO}" -eq 0 ]] ; then
            echo $(date -Iseconds) INFO: ovn-node-metrics-cert not mounted. Waiting one hour.
            HAS_LOGGED_INFO=1
          fi
      }
      while [[ ! -f "${TLS_PK}" ||  ! -f "${TLS_CERT}" ]] ; do
        log_missing_certs
        sleep 5
      done

      echo $(date -Iseconds) INFO: ovn-node-metrics-certs mounted, starting kube-rbac-proxy
      exec /usr/bin/kube-rbac-proxy \
        --logtostderr \
        --secure-listen-address=:9103 \
        --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 \
        --upstream=http://127.0.0.1:29103/ \
        --tls-private-key-file=${TLS_PK} \
        --tls-cert-file=${TLS_CERT}
    State:          Running
      Started:      Thu, 02 Dec 2021 07:36:17 -0500
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:        10m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /etc/pki/tls/metrics-cert from ovn-node-metrics-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vk2gn (ro)
  ovnkube-node:
    Container ID:  cri-o://32cbd581af2f60dabb7cad4f8b46b26f3bd31c7ad1200ea9ddeb44b400f9d613
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e
    Port:          29103/TCP
    Host Port:     29103/TCP
    Command:
      /bin/bash
      -c
      set -xe
      if [[ -f "/env/${K8S_NODE}" ]]; then
        set -o allexport
        source "/env/${K8S_NODE}"
        set +o allexport
      fi
      echo "I$(date "+%m%d %H:%M:%S.%N") - waiting for db_ip addresses"
      cp -f /usr/libexec/cni/ovn-k8s-cni-overlay /cni-bin-dir/
      ovn_config_namespace=openshift-ovn-kubernetes
      echo "I$(date "+%m%d %H:%M:%S.%N") - disable conntrack on geneve port"
      iptables -t raw -A PREROUTING -p udp --dport 6081 -j NOTRACK
      iptables -t raw -A OUTPUT -p udp --dport 6081 -j NOTRACK
      ip6tables -t raw -A PREROUTING -p udp --dport 6081 -j NOTRACK
      ip6tables -t raw -A OUTPUT -p udp --dport 6081 -j NOTRACK
      retries=0
      while true; do
        # TODO: change to use '--request-timeout=30s', if https://github.com/kubernetes/kubernetes/issues/49343 is fixed.
        db_ip=$(timeout 30 kubectl get ep -n ${ovn_config_namespace} ovnkube-db -o jsonpath='{.subsets[0].addresses[0].ip}')
        if [[ -n "${db_ip}" ]]; then
          break
        fi
        (( retries += 1 ))
        if [[ "${retries}" -gt 40 ]]; then
          echo "E$(date "+%m%d %H:%M:%S.%N") - db endpoint never came up"
          exit 1
        fi
        echo "I$(date "+%m%d %H:%M:%S.%N") - waiting for db endpoint"
        sleep 5
      done

      echo "I$(date "+%m%d %H:%M:%S.%N") - starting ovnkube-node db_ip ${db_ip}"
      if [ "shared" == "shared" ]; then
        gateway_mode_flags="--gateway-mode shared --gateway-interface br-ex"
      elif [ "shared" == "local" ]; then
        gateway_mode_flags="--gateway-mode local --gateway-interface br-ex"
      else
        echo "Invalid OVN_GATEWAY_MODE: \"shared\". Must be \"local\" or \"shared\"."
        exit 1
      fi

      export_network_flows_flags=
      if [[ -n "${NETFLOW_COLLECTORS}" ]] ; then
        export_network_flows_flags="--netflow-targets ${NETFLOW_COLLECTORS}"
      fi
      if [[ -n "${SFLOW_COLLECTORS}" ]] ; then
        export_network_flows_flags="$export_network_flows_flags --sflow-targets ${SFLOW_COLLECTORS}"
      fi
      if [[ -n "${IPFIX_COLLECTORS}" ]] ; then
        export_network_flows_flags="$export_network_flows_flags --ipfix-targets ${IPFIX_COLLECTORS}"
      fi
      gw_interface_flag=
      # if br-ex1 is configured on the node, we want to use it for external gateway traffic
      if [ -d /sys/class/net/br-ex1 ]; then
        gw_interface_flag="--exgw-interface=br-ex1"
      fi
      exec /usr/bin/ovnkube --init-node "${K8S_NODE}" \
        --nb-address "ssl:10.1.208.20:9641,ssl:10.1.208.21:9641,ssl:10.1.208.22:9641" \
        --sb-address "ssl:10.1.208.20:9642,ssl:10.1.208.21:9642,ssl:10.1.208.22:9642" \
        --nb-client-privkey /ovn-cert/tls.key \
        --nb-client-cert /ovn-cert/tls.crt \
        --nb-client-cacert /ovn-ca/ca-bundle.crt \
        --nb-cert-common-name "ovn" \
        --sb-client-privkey /ovn-cert/tls.key \
        --sb-client-cert /ovn-cert/tls.crt \
        --sb-client-cacert /ovn-ca/ca-bundle.crt \
        --sb-cert-common-name "ovn" \
        --config-file=/run/ovnkube-config/ovnkube.conf \
        --loglevel "${OVN_KUBE_LOG_LEVEL}" \
        --inactivity-probe="${OVN_CONTROLLER_INACTIVITY_PROBE}" \
        ${gateway_mode_flags} \
        --metrics-bind-address "127.0.0.1:29103" \
        --metrics-enable-pprof \
        ${export_network_flows_flags} \
        ${gw_interface_flag}

    State:       Running
      Started:   Thu, 02 Dec 2021 08:32:53 -0500
    Last State:  Terminated
      Reason:    Error
      Message:    stderr: ""
I1202 13:32:30.332571  162643 ovs.go:206] exec(3036): /usr/bin/ovs-vsctl --timeout=15 --if-exists get interface patch-br-ex1_openshift-master-1-to-br-int ofport
I1202 13:32:30.335837  162643 ovs.go:209] exec(3036): stdout: "2\n"
I1202 13:32:30.335844  162643 ovs.go:210] exec(3036): stderr: ""
I1202 13:32:30.335848  162643 gateway.go:264] Gateway is ready
I1202 13:32:30.335866  162643 ovs.go:206] exec(3037): /usr/bin/ovs-appctl --timeout=15 -t /var/run/ovn/ovn-controller.9999.ctl connection-status
I1202 13:32:30.338144  162643 ovs.go:209] exec(3037): stdout: "connected\n"
I1202 13:32:30.338151  162643 ovs.go:210] exec(3037): stderr: ""
I1202 13:32:30.338154  162643 node.go:239] Node connection status = connected
I1202 13:32:30.338158  162643 ovs.go:206] exec(3038): /usr/bin/ovs-vsctl --timeout=15 -- br-exists br-int
I1202 13:32:30.341647  162643 ovs.go:209] exec(3038): stdout: ""
I1202 13:32:30.341654  162643 ovs.go:210] exec(3038): stderr: ""
I1202 13:32:30.341657  162643 ovs.go:206] exec(3039): /usr/bin/ovs-ofctl dump-aggregate br-int
I1202 13:32:30.344256  162643 ovs.go:209] exec(3039): stdout: "NXST_AGGREGATE reply (xid=0x4): packet_count=3056957 byte_count=815706824 flow_count=1929\n"
I1202 13:32:30.344263  162643 ovs.go:210] exec(3039): stderr: ""
I1202 13:32:30.344272  162643 ovs.go:206] exec(3040): /usr/bin/ovs-vsctl --timeout=15 --if-exists get interface patch-br-ex_openshift-master-1-to-br-int ofport
I1202 13:32:30.347453  162643 ovs.go:209] exec(3040): stdout: ""
I1202 13:32:30.347459  162643 ovs.go:210] exec(3040): stderr: ""
I1202 13:32:30.347464  162643 ovs.go:206] exec(3041): /usr/bin/ovs-vsctl --timeout=15 --if-exists get interface patch-br-ex1_openshift-master-1-to-br-int ofport
I1202 13:32:30.350651  162643 ovs.go:209] exec(3041): stdout: "2\n"
I1202 13:32:30.350659  162643 ovs.go:210] exec(3041): stderr: ""
I1202 13:32:30.350661  162643 gateway.go:264] Gateway is ready
F1202 13:32:30.350680  162643 ovnkube.go:132] error waiting for node readiness: timed out waiting for the condition

      Exit Code:    1
      Started:      Thu, 02 Dec 2021 08:27:29 -0500
      Finished:     Thu, 02 Dec 2021 08:32:30 -0500
    Ready:          False
    Restart Count:  10
    Requests:
      cpu:      10m
      memory:   300Mi
    Readiness:  exec [test -f /etc/cni/net.d/10-ovn-kubernetes.conf] delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:
      KUBERNETES_SERVICE_PORT:          6443
      KUBERNETES_SERVICE_HOST:          api-int.kni-qe-4.lab.eng.rdu2.redhat.com
      OVN_CONTROLLER_INACTIVITY_PROBE:  180000
      OVN_KUBE_LOG_LEVEL:               4
      K8S_NODE:                          (v1:spec.nodeName)
    Mounts:
      /cni-bin-dir from host-cni-bin (rw)
      /env from env-overrides (rw)
      /etc/cni/net.d from host-cni-netd (rw)
      /etc/openvswitch from etc-openvswitch (rw)
      /etc/ovn/ from etc-openvswitch (rw)
      /etc/systemd/system from systemd-units (ro)
      /host from host-slash (ro)
      /ovn-ca from ovn-ca (rw)
      /ovn-cert from ovn-cert (rw)
      /run/netns from host-run-netns (ro)
      /run/openvswitch from run-openvswitch (rw)
      /run/ovn-kubernetes/ from host-run-ovn-kubernetes (rw)
      /run/ovn/ from run-ovn (rw)
      /run/ovnkube-config/ from ovnkube-config (rw)
      /var/lib/cni/networks/ovn-k8s-cni-overlay from host-var-lib-cni-networks-ovn-kubernetes (rw)
      /var/lib/openvswitch from var-lib-openvswitch (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vk2gn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  systemd-units:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/systemd/system
    HostPathType:
  host-slash:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:
  host-run-netns:
    Type:          HostPath (bare host directory volume)
    Path:          /run/netns
    HostPathType:
  var-lib-openvswitch:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/openvswitch/data
    HostPathType:
  etc-openvswitch:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/openvswitch
    HostPathType:
  run-openvswitch:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/openvswitch
    HostPathType:
  run-ovn:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/ovn
    HostPathType:
  node-log:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/ovn
    HostPathType:
  log-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/log
    HostPathType:
  host-run-ovn-kubernetes:
    Type:          HostPath (bare host directory volume)
    Path:          /run/ovn-kubernetes
    HostPathType:
  host-cni-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/bin
    HostPathType:
  host-cni-netd:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/multus/cni/net.d
    HostPathType:
  host-var-lib-cni-networks-ovn-kubernetes:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks/ovn-k8s-cni-overlay
    HostPathType:
  ovnkube-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ovnkube-config
    Optional:  false
  env-overrides:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      env-overrides
    Optional:  true
  ovn-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ovn-ca
    Optional:  false
  ovn-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ovn-cert
    Optional:    false
  ovn-node-metrics-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ovn-node-metrics-cert
    Optional:    true
  kube-api-access-vk2gn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Burstable
Node-Selectors:              beta.kubernetes.io/os=linux
Tolerations:                 op=Exists
Events:
  Type     Reason        Age                  From               Message
  ----     ------        ----                 ----               -------
  Normal   Scheduled     109m                 default-scheduler  Successfully assigned openshift-ovn-kubernetes/ovnkube-node-b6lmg to openshift-master-1
  Warning  FailedMount   109m                 kubelet            MountVolume.SetUp failed for volume "ovn-cert" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount   109m                 kubelet            MountVolume.SetUp failed for volume "ovn-ca" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount   109m                 kubelet            MountVolume.SetUp failed for volume "env-overrides" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulling       109m                 kubelet            Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e"
  Normal   Pulled        109m                 kubelet            Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e" in 7.48203
1219s
  Normal   Pulled        109m                 kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e" already present on m
achine
  Normal   Started       109m                 kubelet            Started container ovn-controller
  Normal   Created       109m                 kubelet            Created container ovn-controller
  Normal   Created       109m                 kubelet            Created container ovn-acl-logging
  Normal   Started       109m                 kubelet            Started container ovn-acl-logging
  Normal   Pulling       109m                 kubelet            Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2149a0469fb553d83418d6ff5f4430fafab71c594deaf789376167cfeed9cc"
  Normal   Pulled        109m                 kubelet            Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2149a0469fb553d83418d6ff5f4430fafab71c594deaf789376167cfeed9cc" in 519.808
15ms
  Normal   Created       109m                 kubelet            Created container kube-rbac-proxy
  Normal   Started       109m                 kubelet            Started container kube-rbac-proxy
  Normal   Pulled        109m                 kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e" already present on m
achine
  Normal   Created       109m                 kubelet            Created container ovnkube-node
  Normal   Started       109m                 kubelet            Started container ovnkube-node
  Warning  Unhealthy     108m (x8 over 109m)  kubelet            Readiness probe failed:
  Warning  NodeNotReady  64m                  node-controller    Node is not ready
  Normal   Pulled        60m                  kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e" already present on m
achine
  Normal   Started       60m                  kubelet            Started container kube-rbac-proxy
  Normal   Created       60m                  kubelet            Created container ovn-controller
  Normal   Started       60m                  kubelet            Started container ovn-controller
  Normal   Pulled        60m                  kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e" already present on m
achine
  Normal   Created       60m                  kubelet            Created container ovn-acl-logging
  Normal   Started       60m                  kubelet            Started container ovn-acl-logging
  Normal   Created       60m                  kubelet            Created container kube-rbac-proxy
  Normal   Pulled        60m                  kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2149a0469fb553d83418d6ff5f4430fafab71c594deaf789376167cfeed9cc" already present on m
achine
  Normal   Created       60m                  kubelet            Created container ovnkube-node
  Normal   Started       60m                  kubelet            Started container ovnkube-node
  Normal   Pulled        50m (x3 over 60m)    kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1537ce54b4edf652c6b210d6e9e6344161a77088833074f0356ef5ebbf5f1e" already present on m
achine
  Warning  Unhealthy     45m                  kubelet            Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of fa319ba7878442f588697436d695599a6435e231f3
213623611f99a1b9200e22 is running failed: container process not found
  Warning  BackOff       15m (x47 over 50m)   kubelet            Back-off restarting failed container
  Warning  Unhealthy     1s (x616 over 60m)   kubelet            Readiness probe failed:
 

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
4.10.0-0.nightly-2021-12-02-033910


Steps to Reproduce:
-------------------
1. Deploy BM IPI cluster
2. Start OVN migration to 2nd interface
3. Observe status of ovnkube pods in openshift-ovn-kubernetes

Actual results:
---------------
OVN migration fails. From machine-config-daemon pods on machine:

I1202 12:36:19.033534   10130 daemon.go:872] systemctl --failed:
NetworkManager-wait-online.service loaded failed failed Network Manager Wait Online
I1202 12:36:19.033546   10130 daemon.go:608] Starting MachineConfigDaemon
I1202 12:36:19.033624   10130 daemon.go:615] Enabling Kubelet Healthz Monitor
I1202 12:36:48.954555   10130 trace.go:205] Trace[1002495192]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (02-Dec-2021 12:36:18.953) (total time: 30000ms):
Trace[1002495192]: [30.000891916s] [30.000891916s] END
E1202 12:36:48.954596   10130 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://172.30.0.1:443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp>
I1202 12:36:48.954846   10130 trace.go:205] Trace[2118766229]: "Reflector ListAndWatch" name:github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101 (02-Dec-2021 12:36:18.953) >
Trace[2118766229]: [30.001136739s] [30.001136739s] END



Expected results:
-----------------
OVN migration succeeds

Additional info:
----------------
Baremetal cluster: 3masters + 4 workers - baremetal network IPv4 - provisioning network IPv6