RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2060932 - Failing to build images due to mtu issue on cni-podman0 bridge
Summary: Failing to build images due to mtu issue on cni-podman0 bridge
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: buildah
Version: 9.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Paul Holzinger
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-04 15:38 UTC by Sandeep Yadav
Modified: 2023-09-15 01:22 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-08 19:13:47 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-114545 0 None None None 2022-03-04 16:30:11 UTC

Description Sandeep Yadav 2022-03-04 15:38:15 UTC
Description of problem:-

Failing to build images due to mtu issue on cni-podman bridge in latest version of buildah(buildah-1.24.1-1.el9.x86_64), Higher mtu set on cni-podman0 than the interface on instance by default.

* dnf install during image build is failing with timeout - Unable to curl the repo metadata:-
~~~  
masked-repo                         2.7 kB/s | 365 kB     02:16    
Errors during downloading metadata for repository 'masked-repo':
  - Curl error (28): Timeout was reached for <url masked for sanity> [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
~~~

During our checks, we found in commit[A] - buildah have moved from using Host Namespace to using private Namespace. Now buildah reads config from /usr/share/containers/containers.conf which defaults to private .

[A] https://github.com/containers/buildah/commit/05d5d22dc6b76efa4fd1b1c1eccec2fbbb2f404a

buildah inspect output (when using older buildah-1.24.0-0.15.el9.x86_64)
~~~
         {
             "Name": "network",
             "Host": true,   >>>>>>>>>>>
             "Path": ""
        }
~~~   

buildah inspect output (when using newer buildah-1.24.1-1.el9.x86_64)

~~~
        {
             "Name": "network",
             "Host": false,    >>>>>>>>
             "Path": ""
        }
~~~      

Actual results:

* Our instance where we run build images for Openstack is hosted in an internal Openstack cloud which uses private network for connectivity (where mtu is 1450).
* With new buildah, When we try to build images as root user, a cni-podman0 bridge is created which have mtu of 1500. In older version cni-podman0 was not used to create and host networking was used instead.

~~~
# ip link | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 >>>> 1450 on actual interface
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
4: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
48: cni-podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 >>>> 1500 mtu on cni-podman0
54: veth9646aed9@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni-podman0 state UP mode DEFAULT group default 
~~~
* We could not curl the repo metadata url because of higher mtu set on cni-podman bridge / ping to outside network with mtu > 1422 was also failing.



Expected results:-

* Buildah should set correct mtu on cni-podman based on interface on the machine


Version-Release number of selected component (if applicable):

RHEL-9 Beta 

Last good known version:-
buildah-1.24.0-0.15.el9.x86_64

Affected version:-
buildah-1.24.1-1.el9.x86_64

How reproducible:

Everytime

Steps to Reproduce:
1) Run an instance in Openstack environment where mtu on interface is set lower than 1500. (It was 1450 in our case)
2) Install buildah-1.24.0-0.15.el9.x86_64
3) Build container using buildah and try Pull content from outside / Ping outside network with mtu 1500



Additional info:

Workaround:-

1) When we set lower mtu(1400/1450) on cni-podman0 - we were able to successfully curl/ pull metadata for repos and image build was successful.

buidah provides way to configuring mtu on bridge via setting mtu in  /etc/cni/net.d/87-podman-bridge.conflist

2) Downgrade to older version buildah-1.24.0-0.15.el9.x86_64
3) Use --network=host when running buildah bud

Comment 3 Paul Holzinger 2022-03-07 17:38:09 UTC
The change from host to private network namespace is expected.

For cni we use a default config which does not contain the mtu key. This means that the mtu is chosen by the kernel.
I don't think we can choose the correct mtu. The system has multiple interfaces it is impossible to know where the network packages are send.

IMO you should just set the correct mtu in the config file. I don't think we should try to fix this cni or netavark.

Comment 6 Red Hat Bugzilla 2023-09-15 01:22:36 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.