Bug 2121841

Summary: Multiple default gateways are created inside container if there are multiple interfaces
Product: Red Hat Enterprise Linux 8 Reporter: navabharathi.gorantl
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Joy Pu <ypu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.7CC: bbaude, dwalsh, jligon, jnovy, lsm5, mheon, pthomas, tsweeney, umohnani, ypu
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-4.4.0-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-16 08:20:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description navabharathi.gorantl 2022-08-26 19:20:42 UTC
Description of problem:
When there are multiple interfaces configured on container, podman adds all of them as default gateways instead of just the default one

Version-Release number of selected component (if applicable):
podman-4.0.0-3.module+el8.6.0+14305+6b14f34e.x86_64


How reproducible:
Easily reproducible

Steps to Reproduce:
1. Create a container and configure multiple network interfaces
2. Check ip routes table inside container

bash-4.2# ip r
default via 10.255.6.1 dev eth1 
default via 10.85.24.1 dev eth0 
10.85.24.0/21 dev eth0 proto kernel scope link src 10.85.24.54 
10.255.6.0/24 dev eth1 proto kernel scope link src 10.255.6.2 

3. All interfaces are added to default gateways

Actual results:
All network interfaces are added to default gateways

Expected results:
Only the one gateway should be added as default in the main table.

Additional info:

Comment 1 Matthew Heon 2022-08-26 20:31:55 UTC
This is deliberate. It makes the process of adding and removing networks easier from a programming perspective (adding and removing networks is always the same operation - add a default gateway, remove one - as opposed to needing to examine the routing table, determine if a default gateway is required, etc). We have no intention of changing this.

However, we are discussing a method of specifying a preferred default gateway by adding an option to set route precedence on connecting to a network. Would this be acceptable?

Comment 2 navabharathi.gorantl 2022-10-05 16:55:10 UTC
With the current implementation, we tried to reorder default routes which didn't work. The precedence you are talking about - will it work differently?

Comment 3 Tom Sweeney 2022-10-05 19:24:28 UTC
@mheon@mheon please see the previous comment.

Comment 4 Matthew Heon 2022-10-05 20:23:14 UTC
Route order does not matter in the kernel; if there are several routes to the same target, the kernel will (by default) round-robin traffic across all of them.

Route precedence makes one of the routes more desirable, causing the kernel to route all traffic to the given target across it.

Comment 5 navabharathi.gorantl 2022-10-05 21:03:53 UTC
sounds good then. Having precedence implemented might help.

Comment 6 Matthew Heon 2022-11-30 20:06:15 UTC
Precedence implemented upstream via https://github.com/containers/netavark/pull/474

Comment 7 Jindrich Novy 2022-12-01 09:15:08 UTC
Matt, does that mean only netavark change is needed and there is nothing to fix on podman's side?

Comment 8 Matthew Heon 2022-12-01 14:15:08 UTC
There is also a c/common change, https://github.com/containers/common/pull/1217 - that is required to allow Podman to set the new config option. I don't know exactly when a version of the library with that patch was merged into Podman, but it is definitely present now.

Comment 10 Jindrich Novy 2022-12-02 08:45:12 UTC
Thanks Matt, I don't see https://github.com/containers/common/pull/1217 vendored into podman-4.3.1 - assuming it's only in the main branch for now?

Comment 11 Matthew Heon 2022-12-02 14:18:50 UTC
Yep, that's correct.

Comment 12 Matthew Heon 2022-12-02 14:19:50 UTC
It should release with 4.4.0 - I don't know if we're planning on backports past that.

Comment 13 Jindrich Novy 2022-12-02 14:21:56 UTC
Thanks Matt, I will keep this one in POST waiting for the next release.

Comment 18 Joy Pu 2023-02-13 07:28:13 UTC
Test with podman-4.4.0-1.module+el8.8.0+18060+3f21f2cc.x86_64 and netavark-1.5.0-4.module+el8.8.0+18060+3f21f2cc.x86_64. The metric works as expected:
# podman network create -o=metric=100 test100
test100
# podman network create -o=metric=200 test200
test200
# podman run -dt --name test100_test200 --network test100 --network test200 quay.io/libpod/busybox top
Trying to pull quay.io/libpod/busybox:latest...
Getting image source signatures
Copying blob 9758c28807f2 done  
Copying config f0b02e9d09 done  
Writing manifest to image destination
Storing signatures
4b49f11460f8e06c9ffc7847a0093afa4e00d48e95981cfcc4fc334ac2d140dd
# podman run -dt --name test200_test100 --network test200 --network test100 quay.io/libpod/busybox top
3779cc196be444e2f8946ad23d226c42382f43d245ad6403f2956a8b9fde205f
#  nsenter -t $(podman inspect --format '{{ .State.Pid }}' test200_test100) -n ip r s
default via 10.89.0.1 dev eth0 proto static metric 100 
default via 10.89.1.1 dev eth1 proto static metric 200 
10.89.0.0/24 dev eth0 proto kernel scope link src 10.89.0.3 
10.89.1.0/24 dev eth1 proto kernel scope link src 10.89.1.3 
#  nsenter -t $(podman inspect --format '{{ .State.Pid }}' test100_test200) -n ip r s
default via 10.89.0.1 dev eth0 proto static metric 100 
default via 10.89.1.1 dev eth1 proto static metric 200 
10.89.0.0/24 dev eth0 proto kernel scope link src 10.89.0.2 
10.89.1.0/24 dev eth1 proto kernel scope link src 10.89.1.2 
# nsenter -t $(podman inspect --format '{{ .State.Pid }}' test200_test100) -n ip  r get 1.1.1.1
1.1.1.1 via 10.89.0.1 dev eth0 src 10.89.0.3 uid 0 
    cache 
# nsenter -t $(podman inspect --format '{{ .State.Pid }}' test100_test200) -n ip  r get 1.1.1.1
1.1.1.1 via 10.89.0.1 dev eth0 src 10.89.0.2 uid 0 
    cache 

No matter the order of the network attached to the container. The route works as the metric setup.

So move it to verified.

Comment 20 errata-xmlrpc 2023-05-16 08:20:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758