Created attachment 1809223 [details] `dnf makecache` raw output Description of problem: I am attempting to build containers based on registry.fedoraproject.org/fedora:35 for both `Amd64` and `Arm64` using Docker Buildx on Github Actions. Until recently, it was working fine, but now dnf is not working and it is impossible to install any package. Here[1] is a link to the build logs from Github Actions. It looks like it stopped working sometime between Sunday (Jul 25) and Monday (Jul 26). Version-Release number of selected component (if applicable): N/A How reproducible: This issue is reproducible using Docker itself and Docker Buildx (both on my local Fedora 33 machine and Github Actions' Ubuntu 20.04 runner), but not Podman. Steps to Reproduce: 1. `docker run --name f35 -ti registry.fedoraproject.org/fedora:35 bash` 2. Attempt to use dnf to install a package. 3. `dnf makecache` fails. Actual results: (Output is slightly reformatted for easier readability. See attachment for raw output.) ``` Fedora rawhide openh264 (From Cisco) - x86_64 Errors during downloading metadata for repository 'fedora-cisco-openh264': - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-rawhide&arch=x86_64 [getaddrinfo() thread failed to start] Error: Failed to download metadata for repo 'fedora-cisco-openh264': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-rawhide&arch=x86_64 [getaddrinfo() thread failed to start] Fedora - Rawhide - Developmental packages for the next Fedora release Errors during downloading metadata for repository 'rawhide': - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64 [getaddrinfo() thread failed to start] Error: Failed to download metadata for repo 'rawhide': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64 [getaddrinfo() thread failed to start] ``` Expected results: For dnf to properly download the repository metadata and work as expected. Additional info: [1]: https://github.com/gotmax23/Containerfiles/actions/workflows/systemd-Fedora.35-ci.yml
This bug appears to have been reported against 'rawhide' during the Fedora 35 development cycle. Changing version to 35.
A little while ago, I created an issue on Github[1], because I wasn't sure if I selected the correct Bugzilla component and nobody responsed here. I just marked this bug as urgent because it makes the Fedora 35 Docker image unusable. [1]: https://github.com/fedora-cloud/docker-brew-fedora/issues/97 Thanks, Maxwell
The problem is with docker seccomp not letting colne3() execute which is needed by glibc. I managed to make it work by downloading the default seccomp (wget https://raw.githubusercontent.com/moby/moby/master/profiles/seccomp/default.json -O seccomp.def.orig.json) and applying this patch: --- seccomp.def.orig.json 2021-08-17 08:33:50.362018930 +0300 +++ seccomp.json 2021-08-12 12:19:48.959299075 +0300 @@ -74,6 +74,7 @@ "clock_gettime64", "clock_nanosleep", "clock_nanosleep_time64", + "clone3", "close", "close_range", "connect", @@ -626,18 +627,6 @@ "excludes": { "caps": [ "CAP_SYS_ADMIN" - ] - } - }, - { - "names": [ - "clone3" - ], - "action": "SCMP_ACT_ERRNO", - "errnoRet": 38, - "excludes": { - "caps": [ - "CAP_SYS_ADMIN" ] } }, and than adding { ... "seccomp-profile": "/etc/docker/seccomp.json" } to /etc/docker/daemon.json
Is this happening using docker on Fedora ? is so we should move this bz to the moby-engine component to patch the seccomp.
I am using the docker-ce repo for Fedora, but I don't think is Fedora related because I have patched the default seccomp profile from moby project on github (https://github.com/moby/moby/blob/master/profiles/seccomp/default.json)
(In reply to Razvan from comment #5) > I am using the docker-ce repo for Fedora, but I don't think is Fedora > related because I have patched the default seccomp profile from moby project > on github > (https://github.com/moby/moby/blob/master/profiles/seccomp/default.json) I am also using the docker-ce repository instead of the `moby-engine` package. I will see if I can reproduce this bug with Fedora's `moby-engine` and test Razvan's change, later. (In reply to Clement Verna from comment #4) > Is this happening using docker on Fedora ? is so we should move this bz to > the moby-engine component to patch the seccomp. I selected the `fedora-container-image` Bugzilla component because this issue only occurs with the `registry.fedoraproject.org/fedora:35` container image. I build my own Docker images based on almost every mainstream Linux distro for testing Ansible content; I've only had this problem with the Fedora 35 image. With Docker images based on earlier versions of Fedora and EL distros (RHEL, Almalinux, and CentOS), `dnf` works properly.
I can reproduce this using the docker.io package in Debian bullseye. I am in a similar situation to the original reporter in that I am creating Fedora 35 based containers for a (currently private) project on github, although I am also using self-hosted runners which are running Debian as the host. Version of docker.io in Debian is 20.10.5+dfsg1-1+b5.
I just tested this with the `registry.fedoraproject.org/fedora:36` image. The same issue occurs. You can see this Github Actions build log here[1]. [1]: https://github.com/gotmax23/Containerfiles/runs/3356064813
I'm not super familiar with Bugzilla. Is there a way to show that this bug effects multiple versions?
This affects all containers that use the new glibc that needs clone3(). May be Fedora 35 was the first to implement it.
Yes the version of glibc are different in F34 and F35 [cverna@localhost] $ podman run -it --rm fedora:35 rpm -qa | grep glibc glibc-minimal-langpack-2.34-1.fc35.x86_64 glibc-common-2.34-1.fc35.x86_64 glibc-2.34-1.fc35.x86_64 ~ [cverna@localhost] $ podman run -it --rm fedora:34 rpm -qa | grep glibc glibc-common-2.33-20.fc34.x86_64 glibc-minimal-langpack-2.33-20.fc34.x86_64 glibc-2.33-20.fc34.x86_64 I have tested docker that is provided by Fedora via the moby-engine package and it looks like it is working [core@fedoracoreos34~]$ sudo docker run -it --rm fedora:35 dnf makecache Fedora rawhide openh264 (From Cisco) - x86_64 3.7 kB/s | 2.5 kB 00:00 Fedora - Rawhide - Developmental packages for the next Fedora release 12 MB/s | 61 MB 00:05 Fedora - Modular Rawhide - Developmental packages for the next Fedora release 236 kB/s | 136 kB 00:00 Metadata cache created. [core@fedoracoreos34 ~]$ rpm -qa | grep moby moby-engine-20.10.7-1.fc34.x86_64 I found the related bz https://bugzilla.redhat.com/show_bug.cgi?id=1986092 which seems to indicate that the docker provided in the ce repo is built without the seccomp support. Once other distro update to a newest version of glibc you might experience the similar issues with docker.
Ha actually Razan had to patch the seccomp profile from upstream moby in https://bugzilla.redhat.com/show_bug.cgi?id=1988199#c3, not sure if this was already patched in Fedora tho (can't seems to find it)
As of writing, fedora:35, fedora:rawhide and centos:stream9-development are still unusable because of this issue for GitHub Actions.
Hi everyone, (In reply to Robert Scheck from comment #13) > As of writing, fedora:35, fedora:rawhide and centos:stream9-development are > still unusable because of this issue for GitHub Actions. This is still a problem for me, as well. I even applied the patch to the seccomp policy that Razvan recommended[1], but I still face the same issue. This[2] is the commit where I added a step to patch the seccomp policy to the workflow files in my Containerfiles Github repo. Following Razvan's advice on a regular Ubuntu 20.04 VM with Docker from the Docker CE repository did fix the problem, however. Github Actions actually uses a completely different version of Docker that is neither from the Docker CE repo nor the regular Ubuntu repo. Instead, Github Actions installs `moby-engine` from this apt repository [3] that Microsoft operates. Here [4,5] are the relevant scripts in the repository that holds the configuration for the Github Actions runner VM images. I have not done extensive debugging, yet, but I have some ideas. One thing that I noticed is that the Github Actions runners have version `20.10.7+azure` of Docker [6] instead of the latest version `20.10.8`. Could that be the problem? The new version is present in the Microsoft apt repository [7], though; Github just hasn't updated `moby-engine` in the runner images, yet. In the meantime, does it make sense to file a bug report upstream (with Docker)? Thanks, Maxwell [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1988199#c3 [2]: https://github.com/gotmax23/Containerfiles/commit/f77828d3c7255a555f360ce546a09dd9711550c3 [3]: https://packages.microsoft.com/ubuntu/20.04/prod [4]: https://github.com/actions/virtual-environments/blob/main/images/linux/scripts/base/repos.sh (Sets up repository) [5]: https://github.com/actions/virtual-environments/blob/main/images/linux/scripts/installers/docker-moby.sh (Installs Docker) [6]: https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-README.md#tools [7]: https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-engine/
Correction: Razvan's patch fixes the `dnf maekcache` issue on Docker (both the Github Actions and Docker CE versions). Docker Buildx still does not work.
I just tested Docker Buildx with the `moby-engine` package on Fedora and it worked! I had to install the Docker Buildx binary manually, because it isn't packaged in Fedora's repos. I also had to set SElinux to permissive mode, as there were a bunch of SElinux denials that I'm not knowledgeable enough to fix. It doesn't require making any changes to the seccomp policy, either. It seems there are two problems that we need to fix for this to work with other versions of Docker: 1. The default seccomp profile needs to be patched as Razvan explained. 2. Docker Buildx is not honoring the `seccomp-profile` directive in `/etc/docker/daemon.json. Adding `--security-opt seccomp=/etc/docker/seccomp.json` to the `docker buildx build` command does not work either. It seems that the only reason Docker Buildx works on Fedora to begin with is that we don't need to patch the seccomp profile at all. Fedora 34's `moby-engine` packages works because it is not at the latest version. I just tested the new update that the package maintainer submitted to Bodhi[1] and it now has that same issue. As I commented on Bodhi, it seems that this[2] is the problematic commit. Razvan's patch reverts part of that commit. [1]: https://bodhi.fedoraproject.org/updates/FEDORA-2021-1437ef97e1 [2]: https://github.com/moby/moby/commit/9f6b562dd12ef7b1f9e2f8e6f2ab6477790a6594#diff-ae1c2d16350cc6adbb1802390cbd195657d5562dd804e0dcc62607213a186c2b
I think it makes sense to open a ticket with upstream Docker, since I don't think there is anything we can do on Fedora's side.
Created attachment 1820738 [details] Razvan's seccomp patch but in a downloadable file
Created attachment 1820965 [details] Razvan's seccomp patch but in a downloadable file (2) There was an indentation error in the previous patch.
Well; I have the same issue. The issue is the upgrade in glibc, openssl... broke the internet access... because wget, curl doesn't work after the upgrade... Complete log here https://tinyurl.com/bu7kxrf9 2021-09-05T05:19:40.6729868Z Upgraded: 2021-09-05T05:19:40.6730779Z glibc-2.34-2.fc35.x86_64 glibc-common-2.34-2.fc35.x86_64 2021-09-05T05:19:40.6732172Z glibc-minimal-langpack-2.34-2.fc35.x86_64 openssl-libs-1:1.1.1l-1.fc35.x86_64 2021-09-05T05:19:40.6733064Z Installed: 2021-09-05T05:19:40.6734218Z glibc-gconv-extra-2.34-2.fc35.x86_64 glibc-langpack-en-2.34-2.fc35.x86_64 2021-09-05T05:19:40.6735556Z openssl-pkcs11-0.4.11-4.fc35.x86_64
Hi all, I recommend taking a look at a look at moby/moby#42681[1] and actions/virtual-environments#3812[2] on Github for more information. In short, Moby upstream is aware of this issue and has merged a patch (PR moby/moby#42681[1]) to fix it. At this point, we are just waiting for Docker to publish a release containing the fix. In the meantime, Github Actions users can look at my comment on actions/virtual-environments#3812[3] for a temporary solution. olem, can you please consider backporting the patch to Fedora's `moby-engine` package, as well? Thanks, Maxwell [1]: https://github.com/moby/moby/pull/42681 [2]: https://github.com/actions/virtual-environments/issues/3812 [3]: https://github.com/actions/virtual-environments/issues/3812#issuecomment-926236912
Hi Everyone, OpenSUSE Tumbleweed just recently switched to Glibc 3.34; here[1] is a link to their Bugzilla thread about this issue, and here[2] is an announcement about it on their mailing list, for your reference. OpenSUSE has already patched their Docker package, and I think Fedora should do the same. I previously linked a Github PR that backports the existing fix to 20.10. Thanks, Maxwell 😀 [1]: https://bugzilla.opensuse.org/show_bug.cgi?id=1190670 [2]: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/P5ATW7DDZ56ZEY6KNT6CLLQIWNNGZYTC/
Hi everyone, I created a PR in the `moby-engine` repository that updates the specfile to the the latest version of `moby/moby` and `docker/cli` and adds a patch from upstream to fix the issue. Here is a link: https://src.fedoraproject.org/rpms/moby-engine/pull-request/9 Thanks, Maxwell
Now that we patched Fedora's moby-engine package, there's not much else Fedora can do to fix this issue. We should probably close this issue. @clem
Will this fix be released for docker / podman on RHEL 7/8 as well? There's no way to run a Fedora 35 / Archlinux / Ubuntu >= 21.10 container in RHEL 7/8 at this time other than manually installing an updated upstream docker release.
The Red Hat Enterprise Linux 8 fix for this issue is tracked in bug 1985499. To my knowledge, there is no plan to correct this issue in Red Hat Enterprise Linux 7. I did not file a bug report because a fix was considered unlikely given the Red Hat Enterprise Linux 7 life-cycle phase.
FWIW this issue seems to be also affecting latest Docker Desktop in macOS, podman (with machine) is not affected