Fedora Account System
Red Hat Associate
Red Hat Customer
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0 Build Identifier: out of the bo dnf cannot function properly in toolbox 34 on silverblue 34 Reproducible: Always Steps to Reproduce: 1. Run `toolbox create && toolbox enter` 2. Attempt to `dnf upgrade` Actual Results: ```sh [andythurman@rockhopper ~]$ podman system reset WARNING! This will remove: - all containers - all pods - all images - all build cache Are you sure you want to continue? [y/N] y [andythurman@rockhopper ~]$ toolbox create Image required to create toolbox container. Download registry.fedoraproject.org/fedora-toolbox:34 (500MB)? [y/N]: y Created container: fedora-toolbox-34 Enter with: toolbox enter [andythurman@rockhopper ~]$ toolbox enter ⬢[andythurman@toolbox ~]$ sudo dnf upgrade We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. Fedora 34 openh264 (From Cisco) - x86_64 0.0 B/s | 0 B 00:01 Errors during downloading metadata for repository 'fedora-cisco-openh264': - Curl error (7): Couldn't connect to server for https://codecs.fedoraproject.org/openh264/34/x86_64/os/repodata/7f23e0ee7e5cc6ce4c032104628b22b061f29b3a9171a6b5b967b877dfc7a316-primary.xml.gz [] - Curl error (7): Couldn't connect to server for https://codecs.fedoraproject.org/openh264/34/x86_64/os/repodata/8978c2b860c29cecf93d4fbaec2f6c62bf9a0060fde751b58a1c7539af659620-filelists.xml.gz [] - Curl error (7): Couldn't connect to server for https://codecs.fedoraproject.org/openh264/34/x86_64/os/repodata/3d50825fb25440abc9579f078205d6ba3590b1d146187af8179eb43214c8b40e-comps-Temporary.x86_64.xml.gz [] Error: Failed to download metadata for repo 'fedora-cisco-openh264': Yum repo downloading error: Downloading error(s): repodata/7f23e0ee7e5cc6ce4c032104628b22b061f29b3a9171a6b5b967b877dfc7a316-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/8978c2b860c29cecf93d4fbaec2f6c62bf9a0060fde751b58a1c7539af659620-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/3d50825fb25440abc9579f078205d6ba3590b1d146187af8179eb43214c8b40e-comps-Temporary.x86_64.xml.gz - Cannot download, all mirrors were already tried without success Fedora Modular 34 - x86_64 0.0 B/s | 0 B 00:00 Errors during downloading metadata for repository 'fedora-modular': - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-34&arch=x86_64&countme=1 [] - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-34&arch=x86_64 [] Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-34&arch=x86_64 [] ⬢[andythurman@toolbox ~]$ ``` Expected Results: Toolbox works without a flaw like in 33
I wonder if this is addressed by bug 1932087 Is your Fedora 34 toolbox, really set up for Fedora 34 or is it still pointing at Rawhide? Maybe look at /etc/os-release? If it's actually a Rawhide container, then I'd suggest deleting the container and the fedora-toolbox:34 image that you have locally, and re-create the container. That will pull in a new and updated image.
I think it may have been related to that bug because with the updated toolbox version it now seems to work fine.
Hmm, I see this issue even with 0.0.99.1 with Fedora 34 container (on Silverblue 34), but with one small difference. This doesn't happen immediately after creating the container, but after the first `dnf udpate`. Any idea what is going on?
You are not able to upgrade or dnf breaks right after upgrading? I am able to upgrade with no issues: ``` [andythurman@rockhopper ~]$ podman system reset WARNING! This will remove: - all containers - all pods - all images - all build cache Are you sure you want to continue? [y/N] y [andythurman@rockhopper ~]$ toolbox create Image required to create toolbox container. Download registry.fedoraproject.org/fedora-toolbox:34 (500MB)? [y/N]: y Created container: fedora-toolbox-34 Enter with: toolbox enter [andythurman@rockhopper ~]$ toolbox enter ⬢[andythurman@toolbox ~]$ cat /etc/os-release NAME=Fedora VERSION="34 (Container Image Prerelease)" ID=fedora VERSION_ID=34 VERSION_CODENAME="" PLATFORM_ID="platform:f34" PRETTY_NAME="Fedora 34 (Container Image Prerelease)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:34" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/34/system-administrators-guide/" SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=34 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=34 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="Container Image" VARIANT_ID=container ⬢[andythurman@toolbox ~]$ sudo dnf upgrade We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. Fedora 34 openh264 (From Cisco) - x86_64 1.7 kB/s | 2.5 kB 00:01 Fedora Modular 34 - x86_64 4.6 MB/s | 4.8 MB 00:01 Fedora Modular 34 - x86_64 - Updates 697 B/s | 257 B 00:00 Fedora Modular 34 - x86_64 - Test Updates 1.4 MB/s | 1.5 MB 00:01 Fedora 34 - x86_64 - Test Updates 4.2 MB/s | 6.4 MB 00:01 Fedora 34 - x86_64 - Updates 337 B/s | 257 B 00:00 Fedora 34 - x86_64 22 MB/s | 73 MB 00:03 Dependencies resolved. =============================================================================================================================================================================================================================================== Package Architecture Version Repository Size =============================================================================================================================================================================================================================================== Upgrading: ``` And install packages afterwards: ``` ⬢[andythurman@toolbox ~]$ sudo dnf install sl Last metadata expiration check: 0:03:05 ago on Tue Mar 9 15:21:14 2021. Dependencies resolved. =============================================================================================================================================================================================================================================== Package Architecture Version Repository Size =============================================================================================================================================================================================================================================== Installing: sl x86_64 5.02-15.fc34 fedora 17 k Transaction Summary =============================================================================================================================================================================================================================================== Install 1 Package Total download size: 17 k Installed size: 22 k Is this ok [y/N]: y Downloading Packages: sl-5.02-15.fc34.x86_64.rpm 71 kB/s | 17 kB 00:00 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 33 kB/s | 17 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : sl-5.02-15.fc34.x86_64 1/1 Running scriptlet: sl-5.02-15.fc34.x86_64 1/1 Verifying : sl-5.02-15.fc34.x86_64 1/1 Installed: sl-5.02-15.fc34.x86_64 Complete! ⬢[andythurman@toolbox ~]$ ```
Oh, I have just realized what is wrong in my case, but perhaps related to your initial issue as well. Although the toolbox container is created for Fedora 34, the rawhide repos are still enabled and others are disabled. So the first "dnf update" upgrades the container to rawhide, which I suppose is not supported by toolbox and/or something is broken in rawhide. However, I suppose that rawhide repos should be disabled and vice versa for Fedora 34 image. $ # I have not run "podman system reset" this time as I don't want to remove one of my containers, but I ran it yesterday, so it should be fine. $ toolbox create test $ toolbox enter test ⬢ grep VERSION_ID /etc/os-release VERSION_ID=34 ⬢ sudo dnf update --nogpg -y ... # The transaction failed because of filesystem package failure, but it seems unrelated as everything else succeeds. ⬢ sudo dnf update --refresh Fedora rawhide openh264 (From Cisco) - x86_64 0.0 B/s | 0 B 00:00 Errors during downloading metadata for repository 'fedora-cisco-openh264': - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-rawhide&arch=x86_64 [] Error: Failed to download metadata for repo 'fedora-cisco-openh264': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-rawhide&arch=x86_64 [] Fedora - Modular Rawhide - Developmental packages for the next Fedora release 0.0 B/s | 0 B 00:00 Errors during downloading metadata for repository 'rawhide-modular': - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=rawhide-modular&arch=x86_64 [] Error: Failed to download metadata for repo 'rawhide-modular': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=rawhide-modular&arch=x86_64 [] ⬢ grep VERSION_ID /etc/os-release VERSION_ID=35 Rishi, can you please take a look at this, please?
Hmm but even if I disable the rawhide repos and enable the correct ones, the second dnf update fails... $ toolbox create test $ toolbox enter test ⬢ grep VERSION_ID /etc/os-release VERSION_ID=34 ⬢ sudo sed -i 's/enabled=0/enabled=1/g' /etc/yum.repos.d/*.repo ⬢ sudo sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/*rawhide*.repo ⬢ sudo dnf update # The transaction failed because of filesystem package failure, but it seems unrelated as everything else succeeds. ⬢ sudo dnf update --refresh Fedora 34 openh264 (From Cisco) - x86_64 0.0 B/s | 0 B 00:00 Errors during downloading metadata for repository 'fedora-cisco-openh264': - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-34&arch=x86_64 [] Error: Failed to download metadata for repo 'fedora-cisco-openh264': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-34&arch=x86_64 [] Fedora 34 openh264 (From Cisco) - x86_64 - Debug 0.0 B/s | 0 B 00:00 Errors during downloading metadata for repository 'fedora-cisco-openh264-debuginfo': - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-debug-34&arch=x86_64 [] Error: Failed to download metadata for repo 'fedora-cisco-openh264-debuginfo': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-debug-34&arch=x86_64 [] Fedora Modular 34 - x86_64 0.0 B/s | 0 B 00:00 Errors during downloading metadata for repository 'fedora-modular': - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-34&arch=x86_64 [] Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-34&arch=x86_64 []
Hmm, I have just tried "curl --verbose https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-34&arch=x86_64" to see why it failed and it works correctly, so I tried again `sudo dnf update --refresh` and it works correctly now as well. So I have totally no idea what is going on here...
It has stopped working again: ⬢ curl --verbose https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-34\&arch=x86_64 * Trying 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6:443... * Immediate connect fail for 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6: Network is unreachable * Trying 2604:1580:fe00:0:dead:beef:cafe:fed1:443... * Immediate connect fail for 2604:1580:fe00:0:dead:beef:cafe:fed1: Network is unreachable * Trying 2610:28:3090:3001:dead:beef:cafe:fed3:443... * Immediate connect fail for 2610:28:3090:3001:dead:beef:cafe:fed3: Network is unreachable * Trying 2605:bc80:3010:600:dead:beef:cafe:fed9:443... * Immediate connect fail for 2605:bc80:3010:600:dead:beef:cafe:fed9: Network is unreachable * Trying 2a05:d014:10:7803:f774:4d7c:e277:a457:443... * Immediate connect fail for 2a05:d014:10:7803:f774:4d7c:e277:a457: Network is unreachable * Trying 2620:52:3:1:dead:beef:cafe:fed6:443... * Immediate connect fail for 2620:52:3:1:dead:beef:cafe:fed6: Network is unreachable * Trying 2620:52:3:1:dead:beef:cafe:fed7:443... * Immediate connect fail for 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable * Trying 2001:4178:2:1269::fed2:443... * Immediate connect fail for 2001:4178:2:1269::fed2: Network is unreachable * Closing connection 0 curl: (7) Couldn't connect to server But it works correctly from the host at the same time (resp. not sure if it is correct to see all those IPv6 failures, but it fallbacks to IPv4 and prints the data as expected): $ curl --verbose https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-34\&arch=x86_64 * Trying 67.219.144.68:443... * Trying 2605:bc80:3010:600:dead:beef:cafe:fed9:443... * Immediate connect fail for 2605:bc80:3010:600:dead:beef:cafe:fed9: Network is unreachable * Trying 2610:28:3090:3001:dead:beef:cafe:fed3:443... * Immediate connect fail for 2610:28:3090:3001:dead:beef:cafe:fed3: Network is unreachable * Trying 2620:52:3:1:dead:beef:cafe:fed7:443... * Immediate connect fail for 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable * Trying 2a05:d014:10:7803:f774:4d7c:e277:a457:443... * Immediate connect fail for 2a05:d014:10:7803:f774:4d7c:e277:a457: Network is unreachable * Trying 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6:443... * Immediate connect fail for 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6: Network is unreachable * Trying 2604:1580:fe00:0:dead:beef:cafe:fed1:443... * Immediate connect fail for 2604:1580:fe00:0:dead:beef:cafe:fed1: Network is unreachable * Trying 2620:52:3:1:dead:beef:cafe:fed6:443... * Immediate connect fail for 2620:52:3:1:dead:beef:cafe:fed6: Network is unreachable * Trying 2001:4178:2:1269::fed2:443... * Immediate connect fail for 2001:4178:2:1269::fed2: Network is unreachable * Connected to mirrors.fedoraproject.org (67.219.144.68) port 443 (#0) ...
Just note that I don't see these issues with Fedora 33 toolbox on the same Silverblue 34 host.
Interesting. Maybe try deleting the Fedora 34 image you already have, and try re-creating the container? It could be that you had an old F34 image from the Rawhide times. One problem with the OCI images is that their local copies don't get updated in-place even if we have a new image on the registry.
Good to know, removal of the F34 image solved the issue with rawhide repos, however, there are still the random DNS issues: $ toolbox rmi --force registry.fedoraproject.org/f34/fedora-toolbox:34 $ toolbox create Image required to create toolbox container. Download registry.fedoraproject.org/fedora-toolbox:34 (500MB)? [y/N]: y Created container: fedora-toolbox-34 Enter with: toolbox enter $ toolbox enter ⬢ sudo dnf update -y ... Fedora 34 openh264 (From Cisco) - x86_64 0.0 B/s | 0 B 00:00 Errors during downloading metadata for repository 'fedora-cisco-openh264': - Curl error (7): Couldn't connect to server for https://codecs.fedoraproject.org/openh264/34/x86_64/os/repodata/7f23e0ee7e5cc6ce4c032104628b22b061f29b3a9171a6b5b967b877dfc7a316-primary.xml.gz [] - Curl error (7): Couldn't connect to server for https://codecs.fedoraproject.org/openh264/34/x86_64/os/repodata/8978c2b860c29cecf93d4fbaec2f6c62bf9a0060fde751b58a1c7539af659620-filelists.xml.gz [] - Curl error (7): Couldn't connect to server for https://codecs.fedoraproject.org/openh264/34/x86_64/os/repodata/3d50825fb25440abc9579f078205d6ba3590b1d146187af8179eb43214c8b40e-comps-Temporary.x86_64.xml.gz [] Error: Failed to download metadata for repo 'fedora-cisco-openh264': Yum repo downloading error: Downloading error(s): repodata/7f23e0ee7e5cc6ce4c032104628b22b061f29b3a9171a6b5b967b877dfc7a316-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/8978c2b860c29cecf93d4fbaec2f6c62bf9a0060fde751b58a1c7539af659620-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/3d50825fb25440abc9579f078205d6ba3590b1d146187af8179eb43214c8b40e-comps-Temporary.x86_64.xml.gz - Cannot download, all mirrors were already tried without success ...
Are you on a F34 host? I heard that on F34 systemd-resolved now uses a Varlink interface, and versions of /usr/bin/toolbox will set up the containers to use that too. I hope it's not about that.
> and versions of /usr/bin/toolbox I meant "new versions".
Yes, I am on up-to-date F34 (Silverblue) host.
I would say that this is probably related to https://bugzilla.redhat.com/show_bug.cgi?id=1933433.
Just in case this is related to something I've seen, could you try "readlink -f /etc/resolv.conf" inside the sandbox?
Debarshi Ray, afaik the varlink bind mount change is not yet built through Koji at all on F34 so should not be related.
(In reply to Seppo Yli-Olli from comment #16) > Just in case this is related to something I've seen, could you try "readlink > -f /etc/resolv.conf" inside the sandbox? ⬢ readlink -f /etc/resolv.conf /run/host/run/systemd/resolve/stub-resolv.conf
So let me get this straight: if you resolve that hostname, do you get any IPv4 addresses in the guest? Do you have any IPv6 capability on your machine at all? Which systemd-resolved version do you have in your host?
Erratum to above: I meant, do you get IPv4 addresses for mirrors.fedoraproject.org when you try to resolve it?
The /etc/resolv.conf linkage looks just fine. I would suggest trying sudo ln -sf /run/host/run/systemd/resolve/resolv.conf /etc/resolv.conf inside the container to see if your issues are related to stub resolver or not. (that should make it use real DNS instead)
I've just upgraded my host and rebooted and it seems to work fine now, so closing for now. If this starts happening again, I will send immediately answers to the questions above.