Created attachment 1751320 [details]
MachineSet configuration

Description of problem:

Environment is RHOSP 16.1, OCP 4.6.12 with IPI deployment. 

When adding a custom MachineSet with a secondary network attached, we would see inconsistent interface naming, and kuryr-cni would pick the wrong interface to bind to.

kuryr.conf would always contain
link_iface = ens3

also when our interfaces were swapped and the secondary network was bound ens3 while the primary network was bound to ens4.

Version-Release number of selected component (if applicable):

OCP 4.6.12

How reproducible:

Sometimes

Steps to Reproduce:
1.
Adding a machineset with the below network config in the providerSpec:
$ yq e '.spec.template.spec.providerSpec.value.networks' ocp-machineset.yaml
- filter: {}
  subnets:
    - filter:
        name: shiftstack-kbw6f-nodes
        tags: openshiftClusterID=shiftstack-kbw6f
    - filter:
        name: data-subnet

We observed inconsistent interface ordering, and kuryr-cni binds always to the first interface (ens3) also when this would be bound to our secondary network.

Actual results:
kuryr-cni would always select ens3 for its link_iface.

Expected results:

kuryr-cni should select the interface bound to the primary network used by the kubelet.

Additional info:

Changing our network configuration in the MachineSet to the following value would give us consistent interface ordering with the secondary network always bound to the secondary interface ens4 and we have correct operation from kuryr-cni.

$ yq e '.spec.template.spec.providerSpec.value.networks' ocp-machineset-consistent.yaml
- filter: {}
  subnets:
    - filter:
        name: shiftstack-kbw6f-nodes
        tags: openshiftClusterID=shiftstack-kbw6f
- filter: {}
  subnets:
    - filter:
        name: data-subnet