Description of problem: A few days ago, cockpit-machine's tests started to regress on Rawhide (example: [1]). Installing with an ISO download now fails. Version-Release number of selected component (if applicable): virt-install-4.0.0-1.fc37.noarch libvirt-daemon-8.1.0-2.fc37.x86_64 How reproducible: Always Steps to Reproduce: virt-install --name test1 --os-variant fedora28 --memory 128 --check path_in_use=off --wait -1 --noautoconsole --disk none --cdrom https://archive.fedoraproject.org/pub/archive/fedora/linux/releases/28/Server/x86_64/os/images/boot.iso Actual results: This happens on Rawhide: Starting install... ERROR internal error: process exited while connecting to monitor: 2022-03-14T13:52:28.403193Z qemu-system-x86_64: -blockdev {"driver":"https","url":"https://archive.fedoraproject.org:443/pub/archive/fedora/linux/releases/28/Server/x86_64/os/images/boot.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: CURL: Error opening file: Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system start test1 otherwise, please restart your installation. Expected results: This succeeds on Fedora 35, 36, RHEL 8/9, Debian/Ubuntu: Starting install... Creating domain... | 0 B 00:00:00 Domain is still running. Installation may be in progress. Waiting for the installation to complete. Additional info: [1] https://artifacts.dev.testing-farm.io/57feadaa-cbd6-476b-a7bb-a55226f2ba2c/
Simon, CCing you FYI.
Thanks for the report, but that virt-install command works for me on a fully updated rawhide VM. I thought maybe it had something to do with curl vs curl-minimal, but `sudo dnf swap curl curl-minimal` didn't change the behavior. Would be interesting to see if there's more info in /var/log/libvirt/qemu/test1.log Unlikely this is virt-install specific though so moving to qemu
Check 'qemu-img info https://archive.fedoraproject.org/pub/archive/fedora/linux/releases/28/Server/x86_64/os/images/boot.iso' works to validate if the QEMU curl block driver is functional
This is more subtle indeed. It clearly still happens on the Testing Farm (https://artifacts.dev.testing-farm.io/57feadaa-cbd6-476b-a7bb-a55226f2ba2c/) and I can reproduce it on an upgraded Fedora 36 VM. However, I tried to run a pristine rawhide VM (which should be very similar to a Testing Farm machine): IMG=$(curl -s -L https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Cloud/x86_64/images/ | grep -o 'Fedora-Cloud-Base-Rawhide[^"]*qcow2' | head -n1) curl -L -o rawhide.qcow2 https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Cloud/x86_64/images/$IMG # nothing fancy, just admin:foobar and root:foobar users curl -L -O https://github.com/cockpit-project/bots/raw/main/machine/cloud-init.iso qemu-system-x86_64 -cpu host -enable-kvm -nographic -m 2048 -drive file=rawhide.qcow2,if=virtio -snapshot -cdrom cloud-init.iso -net nic,model=virtio -net user,hostfwd=tcp::2201-:22 # comfortable login (password "foobar") ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -p 2201 root@localhost Inside the VM: sudo dnf install -y virt-install libvirt-daemon-config-network libvirt-daemon-kvm libvirt-client sudo systemctl start virtqemud virtstoraged virtnetworkd qemu-img info https://archive.fedoraproject.org/pub/archive/fedora/linux/releases/28/Server/x86_64/os/images/boot.iso # image: https://archive.fedoraproject.org/pub/archive/fedora/linux/releases/28/Server/x86_64/os/images/boot.iso # file format: raw # virtual size: 583 MiB (611319808 bytes) # disk size: unavailable and the virt-install command also works. So I'll investigate this some more what is different in these environments.
I'm still puzzled by this -- running the virt-install command right after the VM boot works (see [1], in particular this run [2]), but it does not work later on. There is *something* that breaks iso files. I'll keep digging. However, I want to note that this is *not* a duplicate of bug 2014229 -- qemu-block-curl is installed (the error message is different as well). [1] https://github.com/cockpit-project/cockpit-machines/pull/624 [2] https://artifacts.dev.testing-farm.io/43e20dc6-a896-427d-b3af-ca2ade3adf17/
(In reply to Martin Pitt from comment #5) > I'm still puzzled by this -- running the virt-install command right after > the VM boot works (see [1], in particular this run [2]), but it does not > work later on. There is *something* that breaks iso files. I'll keep digging. It sounds like rather than being a problem with QEMU curl code, it might be an environment issue. Is something in your test environment blocking or breaking network access / routing in some manner ? Or is it something silly like CA certificates not being available to validate the TLS connection ?
Our own CI runs VMs completely offline, without network connection. This works "well enough" in the sense that it at least starts the virt-install domain (we cancel it then, as it can't possibly succeed). However, my interactive reproducer and also the Testing Farm machines *do* have internet connection. This is not a race condition, it fails 100% reliably on rawhide for us. I just didn't find the right "trigger" yet which flips it from "working" to "breaking".
> However, my interactive reproducer and also the Testing Farm machines *do* have internet connection. Ah! But the tests redirect "archive.fedoraproject.org" to localhost and run against a mock server (to make sure it all happens offline). So apparently the virt-install behaviour changed slightly to fail on curl errors more immediatey, instead of only while the domain is already running. That seems fine, but we need to adjust our mock http server . Sorry for the noise! (I'm still quite new to the cockpit-machines tests)
This actually seems to be a regression in curl, when validating certificates. Create mock http(s) server with python: cat <<EOF > /tmp/https.py from http.server import HTTPServer, SimpleHTTPRequestHandler import sys import ssl httpd = HTTPServer(('localhost', 4443), SimpleHTTPRequestHandler) httpd.socket = ssl.wrap_socket(httpd.socket, certfile=sys.argv[1], server_side=True) httpd.serve_forever() EOF Create mock certificate+key for "archive.fedoraproject.org": cat << EOF > /tmp/server.pem -----BEGIN CERTIFICATE----- MIIEzzCCArcCFHl80XzVPLuWK3eByyJ8scDFhNtNMA0GCSqGSIb3DQEBCwUAMCQx IjAgBgNVBAMMGWFyY2hpdmUuZmVkb3JhcHJvamVjdC5vcmcwHhcNMjIwMzE2MTA0 NDExWhcNMjMwMzE2MTA0NDExWjAkMSIwIAYDVQQDDBlhcmNoaXZlLmZlZG9yYXBy b2plY3Qub3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA6bRriK77 UCdy6aJiZ7yT4NZ6J1XLranOsuFBHzVneMmxxsdvItiDwA3kz1iyTBhZHZIJab42 Y+orB8AdHgnfSXlV3li61KBZFj6HJe5NRM5It4ZhvrKHSpqv0unDg+keXsrpXrRs Am2+P4niGDVXaho1LGU7CF4IKLdk7TP7YYIRC5n+XoF9XPVyvf54PV2FEnjPscb0 HD8lGbOiKyYv+cgjjdC4J+sI8QaTep4xV2KJALCDsp+BaJcZzjZ9/+GwMEEQJeL4 B5lpIRABidTZHZFdqQ5tchO2r/aSTES1ToYCNtl/whCPIiQVzfK1AjSmh+Ub0yps s1nIQVRfb6cFPrtReFDyPMG/RQRTqRkDhW5mYTbEnm1wfUGxtpapZGVM4HRNP33I M5X8D3iQXx8cxx5jcHU9h9HfPkClR4GHVDTfD09CWunrNxu5VXKkerPGfOZocwe/ P105Z78MnbtZnG7aBO6xU56hsDPO8BCYjMfrgPZih0QAqMvbRBSpCH10D9UM9507 Y1pF/EH8a30j2kjiNmuG6CYdR0JJpj/8cPPWLQTKHG7vFNdP8G++WThBK6dljc+7 arrJ5mPJbbhio+pcOHuOQIbIoTRsaRLQF/rrubCdvoqC3AfwWsBhyTNX72sy4Xkv G85g+ZIzBhxRpzVazTkcRxSLUk+ltfxlLVsCAwEAATANBgkqhkiG9w0BAQsFAAOC AgEATKZQDocdp/2clxcSWDbMlUxGGpZWPD4EVErnd7qt/fz0KqFIg/yG1mTa/0hE 2jf9VnrhUJmiEhLuWquNxtvPIpHAe9r/Lr1NcCLcKJ8KG85x9HijPcCyd1hsgj22 GPSzCjudx4fSw5UW19DyuAB3MtYClIOZg/a8Oz0Jc+QzMY9dMZC5YoVZ86qMaxv/ +z7yXg+h8HYSaceXPMlb0z1XURx6qGlLJWtD7imJs7TVj2K2R08bIv3Z78wPtOS7 SD0LtJocvSyarw/s5LhntFphkx1syWllkyAJF0SJ9LafHXD1R1KM4x/Pgvjf3Dr8 ucDX42udqWv0KLbgAe84X0p0WRpWxjrtQ1cqaokcQTq+pjBl0ewQZnGUharcwloX OGnOI9Bo/paAGJIt3HaKmi3TC5nLbZMJmtWzJwtGh6vOGEX8og3YsAq3PTnwSg4R gIZmyIZ/K4XSxn2pfB4HUpVB5as6I01m165aHVDIXWRUHECIIDPyIVFJVVl43Gkk Xy9KLVmIm/n8nJJREAZRfydd3sUmBIGGt5vz3p9BuQaE9tv37c+DUDWlOqHgj4Pw Jiv5Zc6FBMfhAwrT7ftZU6w2gOgE2EDjFazQELtwglrD9xSMFynmKj0MBz0GMLE0 2XS/211cA4nrLAlKeIbpgrXo6WcDSNUuyIPk596W4WSEPQE= -----END CERTIFICATE----- -----BEGIN PRIVATE KEY----- MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQDptGuIrvtQJ3Lp omJnvJPg1nonVcutqc6y4UEfNWd4ybHGx28i2IPADeTPWLJMGFkdkglpvjZj6isH wB0eCd9JeVXeWLrUoFkWPocl7k1Ezki3hmG+sodKmq/S6cOD6R5eyuletGwCbb4/ ieIYNVdqGjUsZTsIXggot2TtM/thghELmf5egX1c9XK9/ng9XYUSeM+xxvQcPyUZ s6IrJi/5yCON0Lgn6wjxBpN6njFXYokAsIOyn4FolxnONn3/4bAwQRAl4vgHmWkh EAGJ1NkdkV2pDm1yE7av9pJMRLVOhgI22X/CEI8iJBXN8rUCNKaH5RvTKmyzWchB VF9vpwU+u1F4UPI8wb9FBFOpGQOFbmZhNsSebXB9QbG2lqlkZUzgdE0/fcgzlfwP eJBfHxzHHmNwdT2H0d8+QKVHgYdUNN8PT0Ja6es3G7lVcqR6s8Z85mhzB78/XTln vwydu1mcbtoE7rFTnqGwM87wEJiMx+uA9mKHRACoy9tEFKkIfXQP1Qz3nTtjWkX8 QfxrfSPaSOI2a4boJh1HQkmmP/xw89YtBMocbu8U10/wb75ZOEErp2WNz7tqusnm Y8ltuGKj6lw4e45AhsihNGxpEtAX+uu5sJ2+ioLcB/BawGHJM1fvazLheS8bzmD5 kjMGHFGnNVrNORxHFItST6W1/GUtWwIDAQABAoICACgzhnn1K7ruhkhlTAsvN7rp +gpRKl+9wZYoxGSDccCn5WJdxSLX3tYHLdBzH/ySBaZeDCt58m4+hldndzrHFPdV HW3Jkp3n1VtLt0DRnM4UtBKJ1b+Rg09Ui6E64AyNadGBkyJc4+WJDjxs0g07A8X3 KyVvZne6gSnL3lslYX00/AykWUrVeISonivLndBkiXMTgBrDMBO50v54VZGZmJAd muwWts77EccwS+dAVhmDlDCfrI/p+z9MGr7LxeHh50ZtssLpp6PbAmgTEkis6go5 isR9fFsuw/LB9gmTmiKfT7syr9Tot052qZ+LPBLcjrb6tM4e0qMJcza1tlMFBpQf OAoFmrGpnLYq72B+aYDsNrEdZKO5kdPiZZqfEvdYvbMMuUzMLsu7UggvwowDVPtm 01ZNtnXYLQKb9wtZlR1/B031NynDpPEVBbBzMJL/Y4VJ7pKtcM23HdkYM9avotB7 ZKJH5YOhnwYZaGJpqBZeaoJCnkxtvCgmoAvd3pI6TRHFkqn5E+DYOSwXOH9ZjZkd T2vWUv22gaamz7zNX28aQdoYohtwL+pap6jmZwwEsIjOo7zdl2Ljy52bjIDsaVFF apbgny8x1b0OGKYsRDDcp4tmt75dNyYxYq2wyE0Y8NuKOKbDnuo1yorNQ6NckTIb y879ZeDBWTfnoTEu757hAoIBAQDqi9bI5apxYJgbHDZ2Ek2LGE6bRTNNZBnq4Dua KgXN5G2rXjzkszHI4hi06HqiyF7IybWrEAFE+zvHp0uqYzgi+iIX1BlXzTKj/FGG tDcinfFHKjzEAHbmoE/Jv7Zepkj7wgpCqk2jKnCeSqw/d5815Ey5co6P1b+icPJT Udm2Bw63xvyGTI367f5LFqsa7LI4wEgInLcCfDPnwxpSCqvujJJ7wDgkvNl08Mww XtGYFqzAqcg9mIJo1dvu3GhoL1Xx3F+UM9T51LFXKeT/yFxjUK2F4RzqugkMv4E+ odcKTAL4CUUYe8ZorRnZNKDeNFhc4Rqjs7RP+5GhHXq4LSohAoIBAQD/FOB5n+Jo 7nffc3pz7778JSJDQnHJB5kBMpR19hbf3lqLtGCWV/bAQ+2pvZV8oupjXhheaROK v3/4OOSPyv1mObJECtQ/NInslpIzvbYedZE7Zgc2jWDEuU/GC9vfVDXYR0a4hYSw xnjVMMs/SSS9Xo7lttNQaK3xpqqXYJq5lv1xmfc8LXnBFD8guUx68h64gEe0BVDi m4rmwDBG6NK7ZXVfsaKQ2xmXzbTIiAr5s/KyibBanuIGnnTFvdV7Bb940Jc3N1zq Lr39WmYULhhhj/zGtzPGSSlCiutvj47C6vaWVUR9upHo5KBKuXZNmaGGq9eMOd7e cws5gO9iZv/7AoIBAQC3zjTZJVqoFTm+bSKk7v0D112qwtVMffjtnyI2U15PFwd2 95gi5edK9RzFkTaiIQJeuex7jURhab1UXl+mvLC+gGV9ZQMjUrSv3EmLlelrYHk3 +vq5419KpzE/+tSRYiDp4wulaM8E4np1IjHQXw9O7YXPAt6EC89tHrRY5VkHFIrN LMyy9oWBs0bIU5BtvjaA0PbigOXOtcT7iGM1DhXlERauklmtOK31vctW0MfSN1rU dXluA/mW7Jm5m9LaatYgf0IL436w9qSvmS5T4RG95vjFZUADFx2GdwFzMuNRt3XQ KXuTj1RSre0G0f6LhFwHFYWJ6oeQ8qYSrAaDFSnhAoIBAQDM02XBJCq5Gav2KRNO H1ibvfG6UpNt2oT5GjVuV3Jcx36rqebZMr83+hPQdiSgVW2i5v7oaLXBdy56Eq0t 7dD2kiPcUVnHexe5e9rgY1jnMn6xVKc8HeBzd15gaEthFIy8I9fc+ZHoJScIActj AQfYbCKQwPMEIVpzreJT9cWoe17+JqUJlHatHOXoEAMiwmL6wj9RhedCX6X0zqS1 e+Ujhz/t3ITujpEJfavKTruJcP7UcI3E5soWiIy58EqJI79xcwXJ19HrVidHVpyb XbxEM0vGAXw4n6za+xQ8m+kGCYiChOufjf32Jge0mSIvA8tZiYlm111oALclsKwI ty6bAoIBADCuz+5YqNYdaeuyGnze8g92VpWzaS+nzSaoo5pjp/thfbjTxahwmuGA pLjscruV0HCNsM26/Z8gBFgiE/lf1Yfy+AEUHQXzRcz2AIJJUCwLGc3Nz0uk8f7j k3F2MT1A30SM9oOrXLHxkxv9tMXXqCyMiEHMm7YvEhKl0+vArk9G51pSp7ZLuqxI iVWut7sO1gflMOq1JCKp+96rGj+7XyMhnVC7lW1sfEEGq5GKnvjRUhGE1bmw6v3H DfxdIobxWHx2lyvDuC9+wqyx7kcsrw1d1qFYoWSUd7uT4GaVOMXXC3Lb0SQvYWEz +IZhNo8/BtfRTYG+3A6fcPpF8chpJrg= -----END PRIVATE KEY----- EOF Use the mock server: echo "127.0.0.1 archive.fedoraproject.org" | sudo tee -a /etc/hosts (This is needed so that cert validation can actually work) Run the mock server: python3 /tmp/https.py /tmp/server.pem Try to curl it: # curl https://archive.fedoraproject.org:4443/ curl: (27) Out of memory Disabling certificate validation with "curl -k" works (but the point is to provide a valid cert). OpenSSL itself is happy enough with the fake cert/key and the mocked DNS: openssl s_client -connect archive.fedoraproject.org:4443 -verify 5 < /dev/null Moving to Fedora 36 as with our latest image refresh we now started to see this bug there as well.
Increasing debug level isn't too useful, it just shows it dies during handshake: curl -vvvv https://archive.fedoraproject.org:4443/ * Trying 127.0.0.1:4443... * Connected to archive.fedoraproject.org (127.0.0.1) port 4443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/pki/tls/certs/ca-bundle.crt * CApath: none * TLSv1.0 (OUT), TLS header, Certificate Status (22): * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS header, Finished (20): * TLSv1.2 (IN), TLS header, Supplemental data (23): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.2 (IN), TLS header, Supplemental data (23): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS header, Supplemental data (23): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.2 (IN), TLS header, Supplemental data (23): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.2 (OUT), TLS header, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS header, Supplemental data (23): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=archive.fedoraproject.org * start date: Mar 16 10:44:11 2022 GMT * expire date: Mar 16 10:44:11 2023 GMT * Closing connection 0 * TLSv1.2 (IN), TLS header, Supplemental data (23): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.2 (IN), TLS header, Supplemental data (23): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * old SSL session ID is stale, removing * TLSv1.2 (OUT), TLS header, Supplemental data (23): * TLSv1.3 (OUT), TLS alert, close notify (256): curl: (27) Out of memory
curl-7.82.0-2.fc36 (https://bodhi.fedoraproject.org/updates/FEDORA-2022-5037545546), built yesterday, is likely to fix this.
Exactly. Thanks for the quick and accurate answer, Paul!
FEDORA-2022-5037545546 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-5037545546
Cheers! Indeed that fixes it.
Does this fix the original virt-install problem too?
Richard: Yes, it does. That and our automated tests are happy again.
Perfect. Thanks for confirmation!
FEDORA-2022-5037545546 has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-5037545546` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-5037545546 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-5037545546 has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.