Bug 2063853

Simon, CCing you FYI.

Thanks for the report, but that virt-install command works for me on a fully updated rawhide VM. I thought maybe it had something to do with curl vs curl-minimal, but `sudo dnf swap curl curl-minimal` didn't change the behavior.

Would be interesting to see if there's more info in /var/log/libvirt/qemu/test1.log

Unlikely this is virt-install specific though so moving to qemu

Check 'qemu-img info https://archive.fedoraproject.org/pub/archive/fedora/linux/releases/28/Server/x86_64/os/images/boot.iso' works to validate if the QEMU curl block driver is functional

This is more subtle indeed. It clearly still happens on the Testing Farm (https://artifacts.dev.testing-farm.io/57feadaa-cbd6-476b-a7bb-a55226f2ba2c/) and I can reproduce it on an upgraded Fedora 36 VM.

However, I tried to run a pristine rawhide VM (which should be very similar to a Testing Farm machine):

IMG=$(curl -s -L https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Cloud/x86_64/images/ | grep -o 'Fedora-Cloud-Base-Rawhide[^"]*qcow2' | head -n1)
curl -L -o rawhide.qcow2 https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Cloud/x86_64/images/$IMG

# nothing fancy, just admin:foobar and root:foobar users
curl -L -O https://github.com/cockpit-project/bots/raw/main/machine/cloud-init.iso
qemu-system-x86_64 -cpu host -enable-kvm -nographic -m 2048 -drive file=rawhide.qcow2,if=virtio -snapshot -cdrom cloud-init.iso -net nic,model=virtio -net user,hostfwd=tcp::2201-:22

# comfortable login (password "foobar")
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -p 2201 root@localhost


Inside the VM:

sudo dnf install -y virt-install libvirt-daemon-config-network libvirt-daemon-kvm libvirt-client
sudo systemctl start virtqemud virtstoraged virtnetworkd

qemu-img info https://archive.fedoraproject.org/pub/archive/fedora/linux/releases/28/Server/x86_64/os/images/boot.iso
# image: https://archive.fedoraproject.org/pub/archive/fedora/linux/releases/28/Server/x86_64/os/images/boot.iso
# file format: raw 
# virtual size: 583 MiB (611319808 bytes)
# disk size: unavailable

and the virt-install command also works.

So I'll investigate this some more what is different in these environments.

I'm still puzzled by this -- running the virt-install command right after the VM boot works (see [1], in particular this run [2]), but it does not work later on. There is *something* that breaks iso files. I'll keep digging.

However, I want to note that this is *not* a duplicate of bug 2014229 -- qemu-block-curl is installed (the error message is different as well).

[1] https://github.com/cockpit-project/cockpit-machines/pull/624
[2] https://artifacts.dev.testing-farm.io/43e20dc6-a896-427d-b3af-ca2ade3adf17/

(In reply to Martin Pitt from comment #5)
> I'm still puzzled by this -- running the virt-install command right after
> the VM boot works (see [1], in particular this run [2]), but it does not
> work later on. There is *something* that breaks iso files. I'll keep digging.

It sounds like rather than being a problem with QEMU curl code, it might be an environment issue. Is something in your test environment blocking or breaking network access / routing in some manner  ?  Or is it something silly like CA certificates not being available to validate the TLS connection ?

Our own CI runs VMs completely offline, without network connection. This works "well enough" in the sense that it at least starts the virt-install domain (we cancel it then, as it can't possibly succeed). However, my interactive reproducer and also the Testing Farm machines *do* have internet connection. This is not a race condition, it fails 100% reliably on rawhide for us. I just didn't find the right "trigger" yet which flips it from "working" to "breaking".

> However, my interactive reproducer and also the Testing Farm machines *do* have internet connection.

Ah! But the tests redirect "archive.fedoraproject.org" to localhost and run against a mock server (to make sure it all happens offline).

So apparently the virt-install behaviour changed slightly to fail on curl errors more immediatey, instead of only while the domain is already running. That seems fine, but we need to adjust our mock http server .

Sorry for the noise! (I'm still quite new to the cockpit-machines tests)

This actually seems to be a regression in curl, when validating certificates.

Create mock http(s) server with python:

cat <<EOF > /tmp/https.py 
from http.server import HTTPServer, SimpleHTTPRequestHandler
import sys
import ssl

httpd = HTTPServer(('localhost', 4443), SimpleHTTPRequestHandler)
httpd.socket = ssl.wrap_socket(httpd.socket, certfile=sys.argv[1], server_side=True)
httpd.serve_forever()
EOF

Create mock certificate+key for "archive.fedoraproject.org":

cat << EOF > /tmp/server.pem

-----BEGIN CERTIFICATE-----
MIIEzzCCArcCFHl80XzVPLuWK3eByyJ8scDFhNtNMA0GCSqGSIb3DQEBCwUAMCQx
IjAgBgNVBAMMGWFyY2hpdmUuZmVkb3JhcHJvamVjdC5vcmcwHhcNMjIwMzE2MTA0
NDExWhcNMjMwMzE2MTA0NDExWjAkMSIwIAYDVQQDDBlhcmNoaXZlLmZlZG9yYXBy
b2plY3Qub3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA6bRriK77
UCdy6aJiZ7yT4NZ6J1XLranOsuFBHzVneMmxxsdvItiDwA3kz1iyTBhZHZIJab42
Y+orB8AdHgnfSXlV3li61KBZFj6HJe5NRM5It4ZhvrKHSpqv0unDg+keXsrpXrRs
Am2+P4niGDVXaho1LGU7CF4IKLdk7TP7YYIRC5n+XoF9XPVyvf54PV2FEnjPscb0
HD8lGbOiKyYv+cgjjdC4J+sI8QaTep4xV2KJALCDsp+BaJcZzjZ9/+GwMEEQJeL4
B5lpIRABidTZHZFdqQ5tchO2r/aSTES1ToYCNtl/whCPIiQVzfK1AjSmh+Ub0yps
s1nIQVRfb6cFPrtReFDyPMG/RQRTqRkDhW5mYTbEnm1wfUGxtpapZGVM4HRNP33I
M5X8D3iQXx8cxx5jcHU9h9HfPkClR4GHVDTfD09CWunrNxu5VXKkerPGfOZocwe/
P105Z78MnbtZnG7aBO6xU56hsDPO8BCYjMfrgPZih0QAqMvbRBSpCH10D9UM9507
Y1pF/EH8a30j2kjiNmuG6CYdR0JJpj/8cPPWLQTKHG7vFNdP8G++WThBK6dljc+7
arrJ5mPJbbhio+pcOHuOQIbIoTRsaRLQF/rrubCdvoqC3AfwWsBhyTNX72sy4Xkv
G85g+ZIzBhxRpzVazTkcRxSLUk+ltfxlLVsCAwEAATANBgkqhkiG9w0BAQsFAAOC
AgEATKZQDocdp/2clxcSWDbMlUxGGpZWPD4EVErnd7qt/fz0KqFIg/yG1mTa/0hE
2jf9VnrhUJmiEhLuWquNxtvPIpHAe9r/Lr1NcCLcKJ8KG85x9HijPcCyd1hsgj22
GPSzCjudx4fSw5UW19DyuAB3MtYClIOZg/a8Oz0Jc+QzMY9dMZC5YoVZ86qMaxv/
+z7yXg+h8HYSaceXPMlb0z1XURx6qGlLJWtD7imJs7TVj2K2R08bIv3Z78wPtOS7
SD0LtJocvSyarw/s5LhntFphkx1syWllkyAJF0SJ9LafHXD1R1KM4x/Pgvjf3Dr8
ucDX42udqWv0KLbgAe84X0p0WRpWxjrtQ1cqaokcQTq+pjBl0ewQZnGUharcwloX
OGnOI9Bo/paAGJIt3HaKmi3TC5nLbZMJmtWzJwtGh6vOGEX8og3YsAq3PTnwSg4R
gIZmyIZ/K4XSxn2pfB4HUpVB5as6I01m165aHVDIXWRUHECIIDPyIVFJVVl43Gkk
Xy9KLVmIm/n8nJJREAZRfydd3sUmBIGGt5vz3p9BuQaE9tv37c+DUDWlOqHgj4Pw
Jiv5Zc6FBMfhAwrT7ftZU6w2gOgE2EDjFazQELtwglrD9xSMFynmKj0MBz0GMLE0
2XS/211cA4nrLAlKeIbpgrXo6WcDSNUuyIPk596W4WSEPQE=
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQDptGuIrvtQJ3Lp
omJnvJPg1nonVcutqc6y4UEfNWd4ybHGx28i2IPADeTPWLJMGFkdkglpvjZj6isH
wB0eCd9JeVXeWLrUoFkWPocl7k1Ezki3hmG+sodKmq/S6cOD6R5eyuletGwCbb4/
ieIYNVdqGjUsZTsIXggot2TtM/thghELmf5egX1c9XK9/ng9XYUSeM+xxvQcPyUZ
s6IrJi/5yCON0Lgn6wjxBpN6njFXYokAsIOyn4FolxnONn3/4bAwQRAl4vgHmWkh
EAGJ1NkdkV2pDm1yE7av9pJMRLVOhgI22X/CEI8iJBXN8rUCNKaH5RvTKmyzWchB
VF9vpwU+u1F4UPI8wb9FBFOpGQOFbmZhNsSebXB9QbG2lqlkZUzgdE0/fcgzlfwP
eJBfHxzHHmNwdT2H0d8+QKVHgYdUNN8PT0Ja6es3G7lVcqR6s8Z85mhzB78/XTln
vwydu1mcbtoE7rFTnqGwM87wEJiMx+uA9mKHRACoy9tEFKkIfXQP1Qz3nTtjWkX8
QfxrfSPaSOI2a4boJh1HQkmmP/xw89YtBMocbu8U10/wb75ZOEErp2WNz7tqusnm
Y8ltuGKj6lw4e45AhsihNGxpEtAX+uu5sJ2+ioLcB/BawGHJM1fvazLheS8bzmD5
kjMGHFGnNVrNORxHFItST6W1/GUtWwIDAQABAoICACgzhnn1K7ruhkhlTAsvN7rp
+gpRKl+9wZYoxGSDccCn5WJdxSLX3tYHLdBzH/ySBaZeDCt58m4+hldndzrHFPdV
HW3Jkp3n1VtLt0DRnM4UtBKJ1b+Rg09Ui6E64AyNadGBkyJc4+WJDjxs0g07A8X3
KyVvZne6gSnL3lslYX00/AykWUrVeISonivLndBkiXMTgBrDMBO50v54VZGZmJAd
muwWts77EccwS+dAVhmDlDCfrI/p+z9MGr7LxeHh50ZtssLpp6PbAmgTEkis6go5
isR9fFsuw/LB9gmTmiKfT7syr9Tot052qZ+LPBLcjrb6tM4e0qMJcza1tlMFBpQf
OAoFmrGpnLYq72B+aYDsNrEdZKO5kdPiZZqfEvdYvbMMuUzMLsu7UggvwowDVPtm
01ZNtnXYLQKb9wtZlR1/B031NynDpPEVBbBzMJL/Y4VJ7pKtcM23HdkYM9avotB7
ZKJH5YOhnwYZaGJpqBZeaoJCnkxtvCgmoAvd3pI6TRHFkqn5E+DYOSwXOH9ZjZkd
T2vWUv22gaamz7zNX28aQdoYohtwL+pap6jmZwwEsIjOo7zdl2Ljy52bjIDsaVFF
apbgny8x1b0OGKYsRDDcp4tmt75dNyYxYq2wyE0Y8NuKOKbDnuo1yorNQ6NckTIb
y879ZeDBWTfnoTEu757hAoIBAQDqi9bI5apxYJgbHDZ2Ek2LGE6bRTNNZBnq4Dua
KgXN5G2rXjzkszHI4hi06HqiyF7IybWrEAFE+zvHp0uqYzgi+iIX1BlXzTKj/FGG
tDcinfFHKjzEAHbmoE/Jv7Zepkj7wgpCqk2jKnCeSqw/d5815Ey5co6P1b+icPJT
Udm2Bw63xvyGTI367f5LFqsa7LI4wEgInLcCfDPnwxpSCqvujJJ7wDgkvNl08Mww
XtGYFqzAqcg9mIJo1dvu3GhoL1Xx3F+UM9T51LFXKeT/yFxjUK2F4RzqugkMv4E+
odcKTAL4CUUYe8ZorRnZNKDeNFhc4Rqjs7RP+5GhHXq4LSohAoIBAQD/FOB5n+Jo
7nffc3pz7778JSJDQnHJB5kBMpR19hbf3lqLtGCWV/bAQ+2pvZV8oupjXhheaROK
v3/4OOSPyv1mObJECtQ/NInslpIzvbYedZE7Zgc2jWDEuU/GC9vfVDXYR0a4hYSw
xnjVMMs/SSS9Xo7lttNQaK3xpqqXYJq5lv1xmfc8LXnBFD8guUx68h64gEe0BVDi
m4rmwDBG6NK7ZXVfsaKQ2xmXzbTIiAr5s/KyibBanuIGnnTFvdV7Bb940Jc3N1zq
Lr39WmYULhhhj/zGtzPGSSlCiutvj47C6vaWVUR9upHo5KBKuXZNmaGGq9eMOd7e
cws5gO9iZv/7AoIBAQC3zjTZJVqoFTm+bSKk7v0D112qwtVMffjtnyI2U15PFwd2
95gi5edK9RzFkTaiIQJeuex7jURhab1UXl+mvLC+gGV9ZQMjUrSv3EmLlelrYHk3
+vq5419KpzE/+tSRYiDp4wulaM8E4np1IjHQXw9O7YXPAt6EC89tHrRY5VkHFIrN
LMyy9oWBs0bIU5BtvjaA0PbigOXOtcT7iGM1DhXlERauklmtOK31vctW0MfSN1rU
dXluA/mW7Jm5m9LaatYgf0IL436w9qSvmS5T4RG95vjFZUADFx2GdwFzMuNRt3XQ
KXuTj1RSre0G0f6LhFwHFYWJ6oeQ8qYSrAaDFSnhAoIBAQDM02XBJCq5Gav2KRNO
H1ibvfG6UpNt2oT5GjVuV3Jcx36rqebZMr83+hPQdiSgVW2i5v7oaLXBdy56Eq0t
7dD2kiPcUVnHexe5e9rgY1jnMn6xVKc8HeBzd15gaEthFIy8I9fc+ZHoJScIActj
AQfYbCKQwPMEIVpzreJT9cWoe17+JqUJlHatHOXoEAMiwmL6wj9RhedCX6X0zqS1
e+Ujhz/t3ITujpEJfavKTruJcP7UcI3E5soWiIy58EqJI79xcwXJ19HrVidHVpyb
XbxEM0vGAXw4n6za+xQ8m+kGCYiChOufjf32Jge0mSIvA8tZiYlm111oALclsKwI
ty6bAoIBADCuz+5YqNYdaeuyGnze8g92VpWzaS+nzSaoo5pjp/thfbjTxahwmuGA
pLjscruV0HCNsM26/Z8gBFgiE/lf1Yfy+AEUHQXzRcz2AIJJUCwLGc3Nz0uk8f7j
k3F2MT1A30SM9oOrXLHxkxv9tMXXqCyMiEHMm7YvEhKl0+vArk9G51pSp7ZLuqxI
iVWut7sO1gflMOq1JCKp+96rGj+7XyMhnVC7lW1sfEEGq5GKnvjRUhGE1bmw6v3H
DfxdIobxWHx2lyvDuC9+wqyx7kcsrw1d1qFYoWSUd7uT4GaVOMXXC3Lb0SQvYWEz
+IZhNo8/BtfRTYG+3A6fcPpF8chpJrg=
-----END PRIVATE KEY-----
EOF

Use the mock server:

echo "127.0.0.1 archive.fedoraproject.org" | sudo tee -a /etc/hosts

(This is needed so that cert validation can actually work)


Run the mock server:
python3 /tmp/https.py /tmp/server.pem


Try to curl it:

# curl  https://archive.fedoraproject.org:4443/ 
curl: (27) Out of memory

Disabling certificate validation with "curl -k" works (but the point is to provide a valid cert).

OpenSSL itself is happy enough with the fake cert/key and the mocked DNS:

openssl s_client -connect archive.fedoraproject.org:4443 -verify 5 < /dev/null


Moving to Fedora 36 as with our latest image refresh we now started to see this bug there as well.

Increasing debug level isn't too useful, it just shows it dies during handshake:

 curl -vvvv https://archive.fedoraproject.org:4443/ 
*   Trying 127.0.0.1:4443...
* Connected to archive.fedoraproject.org (127.0.0.1) port 4443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=archive.fedoraproject.org
*  start date: Mar 16 10:44:11 2022 GMT
*  expire date: Mar 16 10:44:11 2023 GMT
* Closing connection 0
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (27) Out of memory

curl-7.82.0-2.fc36 (https://bodhi.fedoraproject.org/updates/FEDORA-2022-5037545546), built yesterday, is likely to fix this.

Exactly.  Thanks for the quick and accurate answer, Paul!

FEDORA-2022-5037545546 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-5037545546

Cheers! Indeed that fixes it.

Does this fix the original virt-install problem too?

Richard: Yes, it does. That and our automated tests are happy again.

Perfect.  Thanks for confirmation!

FEDORA-2022-5037545546 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-5037545546`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-5037545546

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-5037545546 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.