Bug 1978481

Created attachment 1796985 [details]
config and troubleshooting info

Description of problem:

Bonding with static IP on AI+SNO after configure-ovs.sh script runs makes node unreachable.
While in AI live ISO, the bond with static IP works but after first MCS boot when configure-ovs.sh runs the node is not reachable anymore. It seems the OVS configuration is correct, br-ex is getting the IP of the bond and ovs-ports looks correct but I cannot reach the node.
Please note I am using a self-hosted AI not the one available on OCM and the "baremetal" node is a VM on kvm.


Version-Release number of selected component (if applicable):


https://mirror.openshift.com/pub/openshift-v4/amd64/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-4.8.0-rc.1-x86_64-live.x86_64.iso
quay.io/coreos/coreos-installer:latest
quay.io/ocpmetal/ocp-metal-ui:latest
quay.io/ocpmetal/postgresql-12-centos7
quay.io/ocpmetal/assisted-service:latest


How reproducible:

Using bonding and static IP (see attachment ovn.txt)
For testing I tried with a single interface and the SNO cluster is installed successfully. (see in ovn.txt the corresponding nmstate yaml)


Steps to Reproduce:
1.Create a cluster on AI
2.Update install-config to use OVNKubernetes calling /install-config endpoint

curl -v -X PATCH -H 'Content-Type: application/json' -d '"{\"networking\": {\"networkType\":\"OVNKubernetes\"}}"' ${ASSISTED_SERVICE_URL}/api/assisted-install/v1/clusters/$CLUSTER_ID/install-config


3.Generate ISO with custom static_network_config calling /download/image endpoint

Example:

jq -n --arg SSH_KEY "$NODE_SSH_KEY" --arg NMSTATE_YAML1 "$(cat static_ip_bond.yaml)" \
'{
  "ssh_public_key": $SSH_KEY,
  "image_type": "full-iso",
  "static_network_config": [
    {
      "network_yaml": $NMSTATE_YAML1,
      "mac_interface_map": [{"mac_address": "52:54:00:25:81:a9", "logical_nic_name": "enp1s0"}, {"mac_address": "52:54:00:25:81:b9", "logical_nic_name": "enp2s0"}]
    }
  ]
}' >> $request_body

curl -v -H "Content-Type: application/json" -d @$request_body ${ASSISTED_SERVICE_URL}/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/image

4.Download discovery ISO using API (do not click "Generate ISO on the UI or the static_network_config will be replaced)

curl -o /var/lib/libvirt/images/sno_image_ocpt2.iso $ASSISTED_URL/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/image


5.Boot the ISO wait for node to be discovered
6.Start cluster installation
7.Wait first reboot after coreos is installed on the disk


Actual results:

ovs-configure.sh will execute successfully but the node with the static IP will be unreachable.

Expected results:

Node reachable through the network and installation progresses.

Additional info:

I can boot in emergency mode and it seems the nmconnection files are configured correctly. Any ideas on what to check or troubleshoot is welcome.

See attached file ovn.txt for additional details including configure-ovs output