Bug 2009047

Summary: Networking failure when running Fedora 35 container
Product: Red Hat Enterprise Linux 8 Reporter: Chris Evich <cevich>
Component: runcAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Alex Jia <ajia>
Severity: medium Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, jnovy, jwboyer, pasik, skitt, tsweeney, vala.martin, ypu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-10 13:27:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chris Evich 2021-09-29 18:53:25 UTC
Description of problem:

Found by attempting to do a `dnf update` inside a Fedora 35 container on a fully updated CentOS Stream 8 host using podman.

Version-Release number of selected component (if applicable):

runc-1.0.0-74.rc95.module_el8.4.0+886+c9a8d9ad.x86_64

How reproducible:
100% every time


Steps to Reproduce:
1. podman run -it --rm registry.fedoraproject.org/fedora:35 dnf update -y

Actual results:

# dnf update -y
Fedora 35 - x86_64                                       0.0  B/s |   0  B     00:00   
Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]


Expected results:

# dnf update                                                     
Fedora 35 - x86_64                                        20 MB/s |  61 MB     00:03
...cut...
Dependencies resolved.


Additional info:

I'm told there was a big discussion related to this for another distribution, and it boiled down to runc needing to return ENOSYS when syscall is not supported, but returns EPERM instead.  The solution was to update runc in that case, so might be the right thing to do here as well.

Comment 1 Chris Evich 2021-09-29 19:06:11 UTC
In case it matters, the system I reproduced on was using:

podman 3.2.3-0.11.module_el8.4.0+942+d25aada8
conmon 2:2.0.29-1.module_el8.4.0+886+c9a8d9ad
container-selinux 2:2.167.0-1.module_el8.4.0+942+d25aada8
containernetworking-plugins 0.9.1-1.module_el8.4.0+781+acf4c33b
containers-common 1:1.3.1-5.module_el8.4.0+886+c9a8d9ad
runc 1.0.0-74.rc95.module_el8.4.0+886+c9a8d9ad.x86_64

Comment 2 Jindrich Novy 2021-09-30 08:48:04 UTC
Do you happen to have a link to upstream PR Chris?

Comment 3 Chris Evich 2021-10-04 16:38:14 UTC
(In reply to Jindrich Novy from comment #2)
> Do you happen to have a link to upstream PR Chris?

I found this in https://github.com/containers/automation_images/pull/93 but the CI-details have long since been overwritten.  I added a workaround commit (--security-opt seccomp=unconfined) that's perfectly acceptable in this use-case, but not likely for others.  If you're simply after a VM to reproduce this in, ping me on IRC and I can make the arrangements.  But the CentOS image in GCP is fairly stock, I'm pretty sure none of the extra GCP-support packages/services would affect this bug.  In other words, it /should/ reproduce in a vanilla QEMU VM as well.  Anyway, poke me if you need help.

Comment 4 Martin Vala 2021-10-21 04:29:40 UTC
I have same problem  when trying to update fc35 in docker


docker run -it --rm --name test registry.fedoraproject.org/fedora-toolbox:35 /bin/bash

[root@b4f3b027157e /]# dnf update
Fedora 35 - x86_64                                                                                                                                                                                             0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]

When i add --security-opt seccomp=unconfined then it is fine.

docker run -it --rm --name test2 --security-opt seccomp=unconfined registry.fedoraproject.org/fedora-toolbox:35 /bin/bash

Is there any other workaround this because i would like to use image registry.fedoraproject.org/fedora-toolbox:35 when doing docker build and i have same problem there: https://gitlab.com/ndmspc/user/-/jobs/1698245870

Thanks for help

Comment 5 Chris Evich 2021-10-25 20:14:59 UTC
(In reply to Martin Vala from comment #4)
> I have same problem  when trying to update fc35 in docker

I'm not the expert on the underlying issue, but I know that docker also uses runc so it seems likely to be the same problem.  Other than an actual fix, I don't think any other workaround (besides maybe a custom policy) is possible.  But I have no idea how to write a workaround-policy for this, and it could very-well open up other problems if I tried :S

Jindrich, any news on getting a new runc and/or if it fixes this?

Comment 6 Chris Evich 2021-10-25 20:18:39 UTC
Oh! Hang on a sec., I just noticed runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 is available as an update on CentOS Stream-8...okay, just tried and with that it does seem to resolve the issue.

Comment 7 Jindrich Novy 2021-10-26 07:54:27 UTC
Thanks Chris! I will target this at RHEL8.6 to increase awareness this is fixed there. Can we get qa ack please?

Comment 10 Alex Jia 2021-10-27 10:37:39 UTC
[root@kvm-06-guest11 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)

[root@kvm-06-guest11 ~]# rpm -q podman runc kernel
podman-3.4.1-3.module+el8.6.0+12998+730dddcd.x86_64
runc-1.0.2-1.module+el8.6.0+12698+b6644727.x86_64
kernel-4.18.0-348.3.el8.x86_64

[root@kvm-06-guest11 ~]# podman run -it --rm registry.fedoraproject.org/fedora:35 /bin/bash
[root@7f49d7fcae44 /]# dnf update -y
Fedora 35 - x86_64                                                                                                                             17 MB/s |  61 MB     00:03
Fedora 35 openh264 (From Cisco) - x86_64                                                                                                      4.0 kB/s | 2.5 kB     00:00
Fedora Modular 35 - x86_64                                                                                                                    1.4 MB/s | 2.6 MB     00:01
Fedora 35 - x86_64 - Updates                                                                                                                  386  B/s | 257  B     00:00
Fedora Modular 35 - x86_64 - Updates                                                                                                          252  B/s | 257  B     00:01
Dependencies resolved.
Nothing to do.
Complete!

Comment 11 Chris Evich 2021-10-27 17:58:11 UTC
Perfect.  Thanks!

Comment 12 Martin Vala 2021-11-01 14:25:41 UTC
I can confirm that is is also working for me now. Thanks for support

Comment 13 Stephen Kitt 2022-03-16 09:37:31 UTC
*** Bug 2025899 has been marked as a duplicate of this bug. ***

Comment 15 errata-xmlrpc 2022-05-10 13:27:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1762