RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2091840 - Failing to build images due to higher mtu on podman0 bridge
Summary: Failing to build images due to higher mtu on podman0 bridge
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: buildah
Version: 9.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Paul Holzinger
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-31 07:27 UTC by Sandeep Yadav
Modified: 2022-06-08 10:58 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-08 10:58:18 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-123712 0 None None None 2022-05-31 07:54:39 UTC

Description Sandeep Yadav 2022-05-31 07:27:00 UTC
Description of problem:

After buildah moved from host to private network namespace we faced bug[1] because mtu on `cni-podman0` bridge was higher the local interface.
In bug[1] we were told to set correct mtu in the config file, We were able to solve that issue after setting mtu in  /etc/cni/net.d/87-podman-bridge.conflist.

Now, With netavark introduction we are again facing same issue during container image build(reported in bug[2]) - mtu on `podman0` bridge is higher than interface.

~~~
$ ip link | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
4: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
11: podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
12: veth49b24621@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman0 state UP mode DEFAULT group default qlen 1000
~~~

Need help with the below queries:-

* Which is default and recommended - netavark or CNI in RHEL9.0?
* How to set mtu in netavark case - Which configuration file(for cni case we were setting mtu in  /etc/cni/net.d/87-podman-bridge.conflist)?


[1] https://bugzilla.redhat.com/show_bug.cgi?id=2060932
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2091816

Version-Release number of selected component (if applicable):

RHEL-9

Version of podman/buildah and netavark:-
~~~
podman-catatonit-4.0.2-7.el9_0.x86_64
podman-4.0.2-7.el9_0.x86_64
buildah-1.24.2-4.el9_0.x86_64
netavark-1.0.1-34.el9_0.x86_64
~~~


How reproducible:

Everytime


Steps to Reproduce:
1) Run an instance in Openstack environment where mtu on interface is set lower than 1500. (It was 1450 in our case)
2) Build container using buildah and try Pull content from outside / Ping outside network with mtu 1500

Actual results:

* dnf install during image build is failing with timeout - Unable to curl the repo metadata:-
~~~  
masked-repo                         2.7 kB/s | 365 kB     02:16    
Errors during downloading metadata for repository 'masked-repo':
  - Curl error (28): Timeout was reached for <url masked for sanity> [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
~~~

Expected results:

Image build to pass


Additional info:


If we change to cni and set correct mtu in 87-podman-bridge.conflist - things works.

# diff /usr/share/containers/containers.conf /usr/share/containers/containers.conf.backup 
265c265
< network_backend = "cni"
---
> #network_backend = ""


/etc/cni/net.d/87-podman-bridge.conflist
~~~
"mtu": 1450
~~~

Comment 1 Tom Sweeney 2022-05-31 21:11:11 UTC
Paul, PTAL

Comment 2 Paul Holzinger 2022-06-01 12:11:46 UTC
You can set the mtu in the config file for netavark as well.

By default the network config is stored in memory but you can create a config file at /etc/containers/networks/podman.json (default path, can be change din containers.conf).
Podman will use this file over the built in default.

You should be able to use this config:

{
  "name": "podman",
  "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
  "driver": "bridge",
  "network_interface": "podman0",
  "created": "2022-06-01T14:07:03.536116064+02:00",
  "subnets": [
    {
      "subnet": "10.88.0.0/16",
      "gateway": "10.88.0.1"
    }
  ],
  "ipv6_enabled": false,
  "internal": false,
  "dns_enabled": false,
  "ipam_options": {
    "driver": "host-local"
  },
  "options": {
    "mtu": "1400"
  }
}


Basically just run `podman network inspect podman | jq .[]` and then add the options block with the mtu to it and write it to the config file.

Comment 3 Sandeep Yadav 2022-06-03 14:06:13 UTC
Hello Paul,

Thanks for answering my query on high to set mtu.

* Could some please confirm which is default and recommended - netavark or CNI in RHEL9.0?

Comment 4 Paul Holzinger 2022-06-03 14:20:55 UTC
Netavark should be the default starting with RHEL9 and we recommend to use it. If you encounter any problems with it please let us know.

Comment 7 Sandeep Yadav 2022-06-08 10:58:18 UTC
Thank you so much Paul,

I don't have any further queries on this, closing this bug.


Note You need to log in before you can comment on or make changes to this bug.