The issue here is that the default gateway route is changing to another interface other than br-ex before ovnkube comes up. This coupled with 4.10.18 which is missing the fix: https://github.com/ovn-org/ovn-kubernetes/pull/2782 causes an invalid route to be added via the wrong interface. With 4.10.20 and later the fix is present, however that would have also resulted in error because there is no default route available via br-ex. There are however, other non-default routes available via br-ex. In Local gateway mode all of the traffic egresses via the kernel routing stack, so egress traffic out another non-br-ex interface would work. Due to this behavior, there really is no logical reason that we *have* to use a default route next hop via br-ex for masquerade/service routes. Any next hop would do. Therefore to fix this issue we can add code to also consider non-default routes legitimate to identify next hops for these routes. I believe this should only be supported for local gateway mode. In shared gateway mode traffic egresses the br-ex bridge directly, and therefore if there is no default route via br-ex we cannot send external pod traffic. I don't really see this as a valid use case.
We have come up with a better fix that requires no routes at all or a valid next hop external the cluster. This has been posted upstream and will be backportable to 4.10.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399