This is deliberate. It makes the process of adding and removing networks easier from a programming perspective (adding and removing networks is always the same operation - add a default gateway, remove one - as opposed to needing to examine the routing table, determine if a default gateway is required, etc). We have no intention of changing this.

However, we are discussing a method of specifying a preferred default gateway by adding an option to set route precedence on connecting to a network. Would this be acceptable?

With the current implementation, we tried to reorder default routes which didn't work. The precedence you are talking about - will it work differently?

@mheon@mheon please see the previous comment.

Route order does not matter in the kernel; if there are several routes to the same target, the kernel will (by default) round-robin traffic across all of them.

Route precedence makes one of the routes more desirable, causing the kernel to route all traffic to the given target across it.

sounds good then. Having precedence implemented might help.

Precedence implemented upstream via https://github.com/containers/netavark/pull/474

Matt, does that mean only netavark change is needed and there is nothing to fix on podman's side?

There is also a c/common change, https://github.com/containers/common/pull/1217 - that is required to allow Podman to set the new config option. I don't know exactly when a version of the library with that patch was merged into Podman, but it is definitely present now.

Thanks Matt, I don't see https://github.com/containers/common/pull/1217 vendored into podman-4.3.1 - assuming it's only in the main branch for now?

Yep, that's correct.

It should release with 4.4.0 - I don't know if we're planning on backports past that.

Thanks Matt, I will keep this one in POST waiting for the next release.

Test with podman-4.4.0-1.module+el8.8.0+18060+3f21f2cc.x86_64 and netavark-1.5.0-4.module+el8.8.0+18060+3f21f2cc.x86_64. The metric works as expected:
# podman network create -o=metric=100 test100
test100
# podman network create -o=metric=200 test200
test200
# podman run -dt --name test100_test200 --network test100 --network test200 quay.io/libpod/busybox top
Trying to pull quay.io/libpod/busybox:latest...
Getting image source signatures
Copying blob 9758c28807f2 done  
Copying config f0b02e9d09 done  
Writing manifest to image destination
Storing signatures
4b49f11460f8e06c9ffc7847a0093afa4e00d48e95981cfcc4fc334ac2d140dd
# podman run -dt --name test200_test100 --network test200 --network test100 quay.io/libpod/busybox top
3779cc196be444e2f8946ad23d226c42382f43d245ad6403f2956a8b9fde205f
#  nsenter -t $(podman inspect --format '{{ .State.Pid }}' test200_test100) -n ip r s
default via 10.89.0.1 dev eth0 proto static metric 100 
default via 10.89.1.1 dev eth1 proto static metric 200 
10.89.0.0/24 dev eth0 proto kernel scope link src 10.89.0.3 
10.89.1.0/24 dev eth1 proto kernel scope link src 10.89.1.3 
#  nsenter -t $(podman inspect --format '{{ .State.Pid }}' test100_test200) -n ip r s
default via 10.89.0.1 dev eth0 proto static metric 100 
default via 10.89.1.1 dev eth1 proto static metric 200 
10.89.0.0/24 dev eth0 proto kernel scope link src 10.89.0.2 
10.89.1.0/24 dev eth1 proto kernel scope link src 10.89.1.2 
# nsenter -t $(podman inspect --format '{{ .State.Pid }}' test200_test100) -n ip  r get 1.1.1.1
1.1.1.1 via 10.89.0.1 dev eth0 src 10.89.0.3 uid 0 
    cache 
# nsenter -t $(podman inspect --format '{{ .State.Pid }}' test100_test200) -n ip  r get 1.1.1.1
1.1.1.1 via 10.89.0.1 dev eth0 src 10.89.0.2 uid 0 
    cache 

No matter the order of the network attached to the container. The route works as the metric setup.

So move it to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758