Created attachment 1948365 [details] virt-builder log Description of problem: On CentOS stream 9, virt-builder fails to install packages. Version-Release number of selected component (if applicable): # virt-builder --version virt-builder 1.48.2 How reproducible: 100% Steps to Reproduce: $ podman container run -it quay.io/centos/centos:stream9 bash Run the following commands inside the container: 1. dnf install -y guestfs-tools 2. export LIBGUESTFS_BACKEND=direct 3. virt-builder centosstream-9 --install cloud-init Actual results: [root@b7c3d354fa75 /]# virt-builder centosstream-9 --install cloud-init [ 1.7] Downloading: http://builder.libguestfs.org/centosstream-9.xz [ 3.3] Planning how to build this image [ 3.3] Uncompressing [ 6.7] Opening the new disk [ 31.1] Setting a random seed [ 31.2] Installing packages: cloud-init CentOS Stream 9 - BaseOS 0.0 B/s | 0 B 00:12 Errors during downloading metadata for repository 'baseos': - Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [error:0200008A:rsa routines::invalid padding] Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [error:0200008A:rsa routines::invalid padding] virt-builder: error: dnf -y install 'cloud-init': command exited with an error Expected results: virt-builder should install the package and create the image. Additional info:
That's weird. Is the host RHEL 9?
It was tested in a CentOS stream 9 container. If it helps, I can test it on a CentOS stream 9 VM.
Or RHEL 9.
If it's reproducible on RHEL 9 then we can move this bug to RHEL 9, which makes it far more likely to be fixed. The bug itself is a bit of a mystery to me. Could it be something to do with clocks being out of synch? It might also be related to the fallout of removing SHA-1 support, but I'd expect a different error in that case.
The bug does not reproduce on RHEL 9.1 or CentOS stream 9 VMs (virt-builder 1.48.2).
You might want to ask the openssl maintainers or reassign the bug to openssl because AFAIK it's nothing to do with virt-builder.
Thank you for your help Richard.
I think this must have been a transitory server-side issue (problem with the certificate or other TLS setting). The "invalid padding" error may be reported by one of the following (very old) openssl commits, on the client side: 429168e7eecd ("Add pss/x931 files.", 2005-05-28) a78c0632edd4 ("Fix for padding X9.31 padding check and zero padding bytes.", 2005-06-06) ba2de73b1850 ("RT4148", 2016-02-03) In the same environment where virt-builder encounters the problem, a command line "curl -I" with the same URL should reproduce the problem. I can't reproduce the issue now (on RHEL-9.1) with command line "curl", so I suspect problem must have been fixed or otherwise eliminated on the server side. The interesting question is when virt-builder fails, and at the same time, in the same environment, "curl -I" (or just plain "curl") succeeds, on the same URL.
The problem reproduces only in the CentOS Stream 9 container. There was no problem when creating a CentOS Stream 8 image on a CentOS Stream 8. This is the output from the CentOS Stream 9 container: [root@d4be491be3fc /]# virt-builder centosstream-9 --install cloud-init [ 2.8] Downloading: http://builder.libguestfs.org/centosstream-9.xz ######################################################################### 100.0%######################################################################### 100.0% [ 29.4] Planning how to build this image [ 29.4] Uncompressing [ 32.5] Opening the new disk [ 55.5] Setting a random seed [ 55.6] Installing packages: cloud-init CentOS Stream 9 - BaseOS 0.0 B/s | 0 B 00:09 Errors during downloading metadata for repository 'baseos': - Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http&countme=1 [error:0200008A:rsa routines::invalid padding] - Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [error:0200008A:rsa routines::invalid padding] Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [error:0200008A:rsa routines::invalid padding] virt-builder: error: dnf -y install 'cloud-init': command exited with an error If reporting bugs, run virt-builder with debugging enabled and include the complete output: virt-builder -v -x [...] [root@d4be491be3fc /]# curl "https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http" <?xml version="1.0" encoding="utf-8"?> <metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Tue, 07 Mar 2023 10:41:16 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager"> <files> <file name="repomd.xml"> <mm0:timestamp>1677497642</mm0:timestamp> <size>3997</size> <verification> <hash type="md5">8f980a93890198986b7fa0e89e196cef</hash> <hash type="sha1">ae76551ed8707bece6388cd4fd2112e8bbbeb532</hash> <hash type="sha256">a5b3bc7764b09cd764a16e30659a1440205353f7d84510feae66641cf6808a5a</hash> <hash type="sha512">1d608c0d2a4e45bd41296a5f1a182c0e81fa1e95ef3e272bc88cccc5b583960a2f3399ac5d8246615260577012c1a99377a89c087dca93591b46f1605d5a4282</hash> </verification> <resources maxconnections="1"> <url protocol="http" type="http" location="DE" preference="100">http://mirrors.xtom.de/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="DE" preference="100">https://mirrors.xtom.de/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="CH" preference="99">http://linuxsoft.cern.ch/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="CH" preference="99">https://linuxsoft.cern.ch/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="RU" preference="98">http://ftp.nsc.ru/pub/centos-9/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="RU" preference="98">https://ftp.nsc.ru/pub/centos-9/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="AT" preference="97">http://centos.mirror.alwyzon.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="AT" preference="97">https://centos.mirror.alwyzon.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="CZ" preference="96">https://mirror.karneval.cz/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="FR" preference="95">http://mirror.in2p3.fr/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="FR" preference="95">https://mirror.in2p3.fr/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="DK" preference="94">http://mirror.netsite.dk/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="DK" preference="94">https://mirror.netsite.dk/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="CZ" preference="93">http://ftp.fi.muni.cz/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="CZ" preference="93">https://ftp.fi.muni.cz/pub/linux/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="DE" preference="92">https://mirror.scaleuptech.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="DE" preference="92">http://mirror.scaleuptech.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="DE" preference="91">http://mirror.dogado.de/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="DE" preference="91">https://mirror.dogado.de/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="CZ" preference="90">http://ftp.sh.cvut.cz/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="CZ" preference="90">https://ftp.sh.cvut.cz/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="AT" preference="89">http://centos.anexia.at/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="AT" preference="89">https://centos.anexia.at/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="BY" preference="88">https://ftp.byfly.by/pub/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="BY" preference="88">http://ftp.byfly.by/pub/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="UA" preference="87">http://mirror.lanet.network/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="UA" preference="87">https://mirror.lanet.network/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="DE" preference="86">https://mirror1.hs-esslingen.de/pub/Mirrors/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="DE" preference="86">http://mirror1.hs-esslingen.de/pub/Mirrors/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="LV" preference="85">http://centos-stream.koyanet.lv/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="LV" preference="85">https://centos-stream.koyanet.lv/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="PT" preference="84">http://mirrors.ptisp.pt/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="PT" preference="84">https://mirrors.ptisp.pt/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="DE" preference="83">http://ftp.plusline.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="DE" preference="83">https://ftp.plusline.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="SK" preference="82">http://ftp.upjs.sk/pub/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="SK" preference="82">https://ftp.upjs.sk/pub/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="GB" preference="81">http://lon.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="GB" preference="81">https://lon.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="RO" preference="80">http://mirrors.hostico.ro/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="RO" preference="80">https://mirrors.hostico.ro/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="UA" preference="79">http://fastmirror.pp.ua/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="UA" preference="79">https://fastmirror.pp.ua/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="NL" preference="78">https://mirrors.evoluso.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="NL" preference="78">http://mirrors.evoluso.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="FR" preference="77">http://fr2.rpmfind.net/linux/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="FR" preference="77">https://fr2.rpmfind.net/linux/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="IS" preference="76">http://centos.is/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="IS" preference="76">https://centos.is/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="DE" preference="75">https://mirror.netzwerge.de/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="DE" preference="75">http://mirror.netzwerge.de/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="GB" preference="74">http://creeperhost.mm.fcix.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="GB" preference="74">https://creeperhost.mm.fcix.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml</url> </resources> </file> </files> </metalink> [root@d4be491be3fc /]# curl -I "https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http" HTTP/2 200 strict-transport-security: max-age=31536000; includeSubDomains; preload x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block x-content-type-options: nosniff referrer-policy: same-origin content-type: application/metalink+xml content-length: 9404 date: Tue, 07 Mar 2023 10:42:25 GMT apptime: D=6270 x-fedora-proxyserver: proxy35.fedoraproject.org x-fedora-requestid: ZAcVErFz6bET16mYDmWkWQACCBA server: Apache
Unfathomable. We have three scenarios AFACIT: (a) L1: RHEL-9 VM, L2 (TCG): "centosstream-9" guest --> no error (b) L1: CentOS stream 9 VM, L2 (TCG): "centosstream-9" guest --> no error (c) L1: CentOS Stream 9 container, L2 (TCG): "centosstream-9" guest --> error L1 should not play any role in this. The network connection is made by the "centosstream-9" guest's package manager. It's as if L1 being a container corrupted or otherwise altered the TCP stream, and SSL (in L2) caught that. Can you do this: - just run "virt-builder centosstream-9" (no --install option) in the CentOS Stream 9 container - boot the just-built image with QEMU TCG, and attempt the "curl" and "curl -I" commands from inside the "centosstream-9" (L2) guest? Basically your test in comment 9 compared "curl" in *L1* with "dnf" in L2. But we should compare "curl" in *L2* with "dnf" in L2.
BTW I can reproduce the problem: - L0: RHEL-9.1 - L1: created with "podman container run -it quay.io/centos/centos:stream9 bash" (comment#0) - L2: see the rest of the repro instructions in comment#0
I can reproduce the issue like this, too: - L0: RHEL-9.1 - L1: same container, created with podman - L2: created with "virt-builder centosstream-9" in L1, then entered with: virt-rescue --ro -a ./centosstream-9.img -i --network Then, at the virt-rescue prompt in L2: ><rescue> curl -I "https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http" curl: (35) error:0200008A:rsa routines::invalid padding
The problem reproduces *without* the "-i" option; that is: with /sysroot unmounted. In other words, it is the libguestfs appliance's own curl (in L2) that fails!
L1: [root@dfd4c06fcd52 /]# rpm -qf /lib64/libssl.so.3 openssl-libs-3.0.7-5.el9.x86_64 [root@dfd4c06fcd52 /]# rpm -qf /lib64/libcrypto.so.3 openssl-libs-3.0.7-5.el9.x86_64 L2: ><rescue> rpm -qf /lib64/libssl.so.3 file /lib64/libssl.so.3 is not owned by any package ><rescue> rpm -qf /lib64/libcrypto.so.3 file /lib64/libcrypto.so.3 is not owned by any package There's something wrong with the appliance, but I don't know where the files in the appliance come from. Normally they come from "the host", which in this case is supposed to be L1. However, I remember that, for containers, the appliance is prebuilt somewhere else. Let me check... L2: ><rescue> md5sum /lib64/libssl.so.3 /lib64/libcrypto.so.3 0fe7576dae4357d9354debf10a55dc11 /lib64/libssl.so.3 daf7231c4f1f227a52b775b98dd9dc9b /lib64/libcrypto.so.3 L1: [root@dfd4c06fcd52 /]# md5sum /lib64/libssl.so.3 /lib64/libcrypto.so.3 0fe7576dae4357d9354debf10a55dc11 /lib64/libssl.so.3 daf7231c4f1f227a52b775b98dd9dc9b /lib64/libcrypto.so.3 Yeah, same OpenSSL library files between L1 and L2. OK, let's pass "-v" too, to curl: L1: [root@dfd4c06fcd52 /]# curl -v -I "https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http" * Trying 209.132.190.2:443... * Connected to mirrors.centos.org (209.132.190.2) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/pki/tls/certs/ca-bundle.crt * TLSv1.0 (OUT), TLS header, Certificate Status (22): * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS header, Finished (20): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.2 (OUT), TLS header, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS header, Unknown (23): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=mirrors.centos.org * start date: Jan 23 22:36:36 2023 GMT * expire date: Apr 23 22:36:35 2023 GMT * subjectAltName: host "mirrors.centos.org" matched cert's "mirrors.centos.org" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 L2: * Trying 38.145.60.20:443... * Connected to mirrors.centos.org (38.145.60.20) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/pki/tls/certs/ca-bundle.crt * TLSv1.0 (OUT), TLS header, Certificate Status (22): * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS header, Finished (20): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS header, Unknown (21): * TLSv1.3 (OUT), TLS alert, decrypt error (563): * error:0200008A:rsa routines::invalid padding * Closing connection 0 Hmmm. This again suggests that the problem may be on the server side -- but alas that's not the case: I've tried several IP addresses in L2, and they all fail. (And they all work in L1.) More interestingly: L2: ><rescue> curl -I https://38.145.60.20:443 curl: (35) error:02000086:rsa routines::last octet invalid Note that this is a *different* OpenSSL error! Something definitely corrupts the network traffic for the L2 guest. It's some weird interaction between QEMU's "user" netdev backend, and the networking offered by the container. OK, let me boot L2 *without* any trace of libguestfs: L1: /usr/libexec/qemu-kvm \ -nodefaults \ -no-user-config \ -nographic \ -m 2048 \ -machine accel=tcg \ -cpu max \ \ -chardev stdio,signal=off,mux=on,id=char0 \ -mon chardev=char0,mode=readline \ -serial chardev:char0 \ \ -drive if=none,format=raw,id=drive0,file=centosstream-9.img \ -device virtio-blk-pci,drive=drive0 \ \ -netdev user,id=usernet \ -device virtio-net-pci,netdev=usernet L2: CentOS Stream 9 Kernel 5.14.0-71.el9.x86_64 on an x86_64 localhost login: root Password: Last login: Tue Mar 7 11:25:04 on ttyS0 [root@ibm-p8-kvm-03-guest-02 ~]# curl -I "https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http" curl: (35) error:0200008A:rsa routines::invalid padding So, yes. The bug is somewhere in the *combination* of QEMU user networking and container networking. It has nothing to do with virt-builder; the last L1 command shown above does not involve any libguestfs element, yet it reproduces the symptom in L2. I suggest that we move this BZ over to "container networking" -- but I don't know what Product and Component to pick for that. If we can't figure out those fields, we'll just have to close this as NOTABUG (as there is no bug in virt-builder).
Thanks for sharing a clear standalone reproducer https://bugzilla.redhat.com/show_bug.cgi?id=2175778#c14! I took the liberty of moving this under the Podman component. It may not be the final destination of this BZ, but I believe that people there have a good idea about both the container and VM networking.
I don't see this as an image issue. This appears to be related to nested slirp processes. QEMU is using user networking (I expect this is effectively required, given the limitations of the rootless user namespace the VM is running in - but I'm not an expert on VMs, so I could well be wrong). User networking uses the Slirp networking stack. QEMU is running inside rootless Podman, which uses slirp4netns (based on QEMU's slirp stack) to tunnel container traffic to the outside world. Effectively, this configuration is layering slirp on top of slirp, which I do not believe is a sane configuration - I'm not 100% sure of this, but I strongly suspect there's a kernel limitation or two that will make this impossible. For container-in-container scenarios, we generally recommend using `--network=host` for the outer container, to ensure there are no networking conflicts with the inner container. I would recommend this as a workaround for the current issue. If this is not acceptable, we'll want to get this reassigned to whoever owns the slirp stack in QEMU, to see if this sort of slirp-on-slirp is possible.
Thanks for analyzing this. I have tried running the original reproducer with --network=host and it hit the same "invalid padding" issue. According to Orel, the same problem happens while using Docker, so that should could for non-rootless, right? For what it's worth, this also reproduces when Fedora rawhide container is used at L1. Anything else we could check to see whether it containter or VM issue?
`--net=host` should take container networking completely out of the picture, so I think we're close to eliminating Podman as a culprit. Could you try a combination of `--net=host` and `--privileged` (mostly to remove any Seccomp filters in place and get things closer to the host system)? If that fails, I think we should reassign to QEMU for investigation of the slirp stack itself.
Aha! With `--privileged` it works fine, both with and without `--net=host`. Thanks Matthew. Is running VMs in an unprivileged container a use-case that should we are interested in supporting? Orel, is this solution good enough for you?
We can probably drill deeper if desired and figure out exactly what parts of privileged are necessary. I suspect that a fully-privileged container could be avoided in favor of adding a device node or two and some capabilities (/dev/net/tun and CAP_NET_ADMIN would be my first guess?) and possibly disabling Seccomp.
The "net.ipv4.ping_group_range" sysctl could be missing too, perhaps.
(In reply to Petr Horáček from comment #23) > Aha! With `--privileged` it works fine, both with and without `--net=host`. > Thanks Matthew. > > Is running VMs in an unprivileged container a use-case that should we are > interested in supporting? > > Orel, is this solution good enough for you? With the `--privileged` flag it works for me. Thank you for the workaround @mheon
We don't know what the bug is. I would try to see if this fails with SELinux disabled instead of --privileged --security-opt label=disable If that still fails then try --cap-add all And see if it is a missing capability.
It fails with the same error. I tried: podman container run -it quay.io/centos/centos:stream9 bash podman container run --security-opt label=disable -it quay.io/centos/centos:stream9 bash podman container run --cap-add all -it quay.io/centos/centos:stream9 bash podman container run --security-opt label=disable --cap-add all -it quay.io/centos/centos:stream9 bash setenforce 0 && podman container run --security-opt label=disable --cap-add all -it quay.io/centos/centos:stream9 bash
Stefano (CC'd) recommends also trying "--net=pasta" with podman (cf. comment#22); see if "libslirp-on-pasta" changes the symptom.