@chanphil.com Could you help verifying it?
Making Comment 3 un-private as Phil is a PE and cannot see private comments. Hi Phil, please see request above.
Hi All, Specifically, which OCP release and RHCOS build must this be verified on that contains this update/fix?
@chanphil.com 4.10.20 would work.
I have tested standing up the OCP cluster with two interfaces per node, while defining two default gateways on each using the following builds: OCP 4.10.22 and RHCOS 4.10.16 OCP 4.10.23 and RHCOS 4.10.16 OCP 4.10.24 and RHCOS 410.84.202207140708-0 I have been unsuccessful in installing the cluster using network configuration with the new OCP 4.10.x builds. As described by the fix under https://github.com/openshift/ovn-kubernetes/pull/1131,, I should be able to provide a hint file containing the variable KUBELET_NODEIP_HINT through machine-config through the installation so that this allows the br-ex interface to select the correction IP address range. I exchanged emails with Tim Rozet for these steps as detailed here https://access.redhat.com/articles/6956852. The bootstrap and control plane nodes start up, however, when taking a closer look at the routing on a master: [core@master-01 ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 9.12.23.1 0.0.0.0 UG 48 0 0 br-ex 0.0.0.0 192.168.79.1 0.0.0.0 UG 101 0 0 enc2 9.12.23.0 0.0.0.0 255.255.255.0 U 48 0 0 br-ex 10.128.0.0 10.129.0.1 255.252.0.0 UG 0 0 0 ovn-k8s-mp0 10.129.0.0 0.0.0.0 255.255.254.0 U 0 0 0 ovn-k8s-mp0 169.254.169.0 9.12.23.1 255.255.255.252 UG 0 0 0 br-ex 169.254.169.3 10.129.0.1 255.255.255.255 UGH 0 0 0 ovn-k8s-mp0 172.30.0.0 9.12.23.1 255.255.0.0 UG 0 0 0 br-ex 192.168.79.0 0.0.0.0 255.255.255.0 U 101 0 0 enc2 [core@master-01 ~]$ ip route show default via 9.12.23.1 dev br-ex proto static metric 48 default via 192.168.79.1 dev enc2 proto dhcp metric 101 9.12.23.0/24 dev br-ex proto kernel scope link src 9.12.23.56 metric 48 10.128.0.0/14 via 10.129.0.1 dev ovn-k8s-mp0 10.129.0.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.0.2 169.254.169.0/30 via 9.12.23.1 dev br-ex 169.254.169.3 via 10.129.0.1 dev ovn-k8s-mp0 172.30.0.0/16 via 9.12.23.1 dev br-ex mtu 1400 192.168.79.0/24 dev enc2 proto kernel scope link src 192.168.79.22 metric 101 The br-ex interface chooses to use 9.12.23.0 for its default routes. Which is incorrect, as I want the 192.168.79.0 range to be selected for the br-ex as configured by the nodeip-configuration service that is properly created on the node upon installation: [core@master-01 ~]$ cat /etc/default/nodeip-configuration KUBELET_NODEIP_HINT=192.168.79.1 I’ve attached a must-gather log and full journalctl from a control plane node. Please let me know if you require additional information. Thank you, Phil
Created attachment 1898477 [details] must-gather log
Created attachment 1898478 [details] master-01 journalctl log
Hi Phil, The KUBELET_NODEIP_HINT will not work to indicate to configure-ovs which NIC to use in 4.10. That only works in 4.12. For 4.10 you have 2 options: 1. You create the br-ex network manager keyfiles manually and load them with ignition. 2. You create a special file with ignition on the host at "/var/lib/ovnk/iface_default_hint". This file should simply have the name of the interface inside of it that you wish to use for br-ex bridge. Option 2 is obviously the easiest, but I know it does not provide a great deal of flexibility as you have to know the NIC name on the host. Could you please try this out and see if it works?
Hi Tim, I followed option #2 using OCP build 4.10.42. Using the same network topology as previous test environments where my 9.12.23.x network interface is `enc1` and 192.168.79.x network interface is `enc2`. I want to use 192.168.79.x for br-ex and placed `enc2` inside the hint file /var/lib/ovnk/iface_default_hint, passing that into the ignition files. All nodes(bootstrap, master and worker) start up and the hint file is generated on the master nodes: [core@master-00 ~]$ cat /var/lib/ovnk/iface_default_hint enc2 But the br-ex bridge never comes up on any of master or worker nodes: [core@master-00 ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 9.12.23.1 0.0.0.0 UG 100 0 0 enc1 0.0.0.0 192.168.79.1 0.0.0.0 UG 101 0 0 enc2 9.12.23.0 0.0.0.0 255.255.255.0 U 100 0 0 enc1 10.128.0.0 0.0.0.0 255.252.0.0 U 0 0 0 tun0 172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 tun0 192.168.79.0 0.0.0.0 255.255.255.0 U 101 0 0 enc2 [core@master-00 ~]$ ip route show default via 9.12.23.1 dev enc1 proto static metric 100 default via 192.168.79.1 dev enc2 proto dhcp metric 101 9.12.23.0/24 dev enc1 proto kernel scope link src 9.12.23.54 metric 100 10.128.0.0/14 dev tun0 scope link 172.30.0.0/16 dev tun0 192.168.79.0/24 dev enc2 proto kernel scope link src 192.168.79.21 metric 101 Cluster installation does not complete. I’ll attach a copy of the journal logs from both master and worker nodes, please let me know if you need additional info or logs. Thanks, Phil
Created attachment 1926220 [details] worker-0.journalctl.log
Created attachment 1926221 [details] master-0.journalctl.log
Hi, As noted above, I used ‘enc2’ inside the "/var/lib/ovnk/iface_default_hint" file under ignition for the cluster nodes on first boot. This did not succeed for OCP 4.10.42. The br-ex bridge interface did not get assigned on the control plane or worker nodes. We were left with the following routing: [core@master-00 ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 9.12.23.1 0.0.0.0 UG 100 0 0 enc1 0.0.0.0 192.168.79.1 0.0.0.0 UG 101 0 0 enc2 9.12.23.0 0.0.0.0 255.255.255.0 U 100 0 0 enc1 10.128.0.0 0.0.0.0 255.252.0.0 U 0 0 0 tun0 172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 tun0 192.168.79.0 0.0.0.0 255.255.255.0 U 101 0 0 enc2 I have now tested this hint option on two additional OCP builds using the same iface_default_hint under ignition: OCP 4.10.44 OCP 4.10.0-0.nightly-s390x-2022-11-30-111235 The cluster installation still does not succeed - one or both workers do not join. However, the routing table on the master nodes look much better. The br-ex bridge get properly assigned to the correct interface (enc2) under "/var/lib/ovnk/iface_default_hint": [core@master-00 ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.79.1 0.0.0.0 UG 48 0 0 br-ex 0.0.0.0 9.12.23.1 0.0.0.0 UG 100 0 0 enc1 9.12.23.0 0.0.0.0 255.255.255.0 U 100 0 0 enc1 10.128.0.0 10.129.0.1 255.252.0.0 UG 0 0 0 ovn-k8s-mp0 10.129.0.0 0.0.0.0 255.255.254.0 U 0 0 0 ovn-k8s-mp0 169.254.169.0 192.168.79.1 255.255.255.252 UG 0 0 0 br-ex 169.254.169.3 10.129.0.1 255.255.255.255 UGH 0 0 0 ovn-k8s-mp0 172.30.0.0 192.168.79.1 255.255.0.0 UG 0 0 0 br-ex 192.168.79.0 0.0.0.0 255.255.255.0 U 48 0 0 br-ex [core@master-00 ~]$ cat /var/lib/ovnk/iface_default_hint enc2 On one of the worker nodes, I captured the following routing on first boot: [core@worker-01 ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 9.12.23.1 0.0.0.0 UG 100 0 0 enc1 0.0.0.0 192.168.79.1 0.0.0.0 UG 101 0 0 enc2 9.12.23.0 0.0.0.0 255.255.255.0 U 100 0 0 enc1 192.168.79.0 0.0.0.0 255.255.255.0 U 101 0 0 enc2 After it self restarts, the routing table assigns the br-ex bridge to the incorrect interface (enc1). [core@worker-01 ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 9.12.23.1 0.0.0.0 UG 48 0 0 br-ex 0.0.0.0 192.168.79.1 0.0.0.0 UG 101 0 0 enc2 9.12.23.0 0.0.0.0 255.255.255.0 U 48 0 0 br-ex 10.128.0.0 10.131.0.1 255.252.0.0 UG 0 0 0 ovn-k8s-mp0 10.131.0.0 0.0.0.0 255.255.254.0 U 0 0 0 ovn-k8s-mp0 169.254.169.0 9.12.23.1 255.255.255.252 UG 0 0 0 br-ex 169.254.169.3 10.131.0.1 255.255.255.255 UGH 0 0 0 ovn-k8s-mp0 172.30.0.0 9.12.23.1 255.255.0.0 UG 0 0 0 br-ex 192.168.79.0 0.0.0.0 255.255.255.0 U 101 0 0 enc2 When checking the hint file residing on this same worker node, the file shows ‘enc1’ and not what I originally defined in the ignition which was ‘enc2’. [core@worker-01 ~]$ cat /var/lib/ovnk/iface_default_hint enc1 I’m not sure why it is changing this parameter on the worker node(s) after first boot, but this hint seems to work on the master nodes. I’m attaching a must-gather log from the worker node for further debug. Please let me know if there is anything else you may need. Thank you, Phil
Created attachment 1929455 [details] Must-gather from worker node running OCP 4.10.44
Changing the bug to Assigned state based on https://bugzilla.redhat.com/show_bug.cgi?id=2092501#c34
I repeated the same test configuration using option iface_default_hint with OCP 4.10.45. The cluster installation still failed. The behavior seems to have regressed prior to the test results above with 4.10.44 - the br-ex bridge is not assigned on the interface on the control plane nodes. The worker nodes do not complete their bootstrap. I’ve attached the full journalctl logs from both bootstrap-0 and master-0 nodes. Please let me know if there's anything else you may need. Thank you, Phil
Created attachment 1931213 [details] bootstrap-0.journalctl.12082022.log
Created attachment 1931214 [details] master-0.journalctl.12082022.log
@chanphil.com I think that the proper way to solve this is to deploy keyfiles for the gateway interfaces with a proper explicit metric so that the preferred gateway has always the lowest metric and is always the preferred one. Anyway, back to the hint file... About message #39, I went to check the attached logs in message #40 and he doesn't seem to be deploying with ovn-k correctly, there are absolutely no logs from configure-ovs at all... I traced back to message #34 and the corresponding attached logs on message #35, and I do see configure-ovs logs. But there it looks like the hint file is not being correctly deployed on the worker nodes as configure-ovs does not find it: Dec 02 15:41:50 worker-01.pok-60.ocptest.pok.stglabs.ibm.com configure-ovs.sh[1330]: ++ get_iface_default_hint /var/lib/ovnk/iface_default_hint Dec 02 15:41:50 worker-01.pok-60.ocptest.pok.stglabs.ibm.com configure-ovs.sh[1330]: ++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint Dec 02 15:41:50 worker-01.pok-60.ocptest.pok.stglabs.ibm.com configure-ovs.sh[1330]: ++ '[' -f /var/lib/ovnk/iface_default_hint ']' Dec 02 15:41:50 worker-01.pok-60.ocptest.pok.stglabs.ibm.com configure-ovs.sh[1330]: ++ echo '' Dec 02 15:41:50 worker-01.pok-60.ocptest.pok.stglabs.ibm.com configure-ovs.sh[1330]: + iface_default_hint= So basically configure-ovs does not find a /var/lib/ovnk/iface_default_hint and uses a different criteria to choose the interface, and then it creates /var/lib/ovnk/iface_default_hint itself. I suspect you might not be deploying two MachineConfigs, one for the master node pool and another one for the worker node pool.
Hi All, I retested the latest OCP 4.10.48 using the iface_default_hint file with two machineconfigs. One for machine nodes and another for worker nodes. The right interface (enc2) is chosen for br-ex bridge this time on all the master and worker nodes. Installation of the OCP 4.10.48 cluster succeeds. I have also verified this on OCP 4.11.23 works as expected. Based on the results, using the iface_default_hint file works in determine the br-ex bridge. Thank you all for your help. -Phil
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.10.51 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:0561