The Fedora IoT installer currently fails to boot to anaconda: anaconda 41.24-1.fc41 for Fedora 41 (pre-release) started. * installation log files are stored in /tmp during the installation * shell is available on TTY2 and in second TMUX pane (ctrl+b, then press 2) * when reporting a bug add logs from /tmp as separate text/plain attachments 16:23:20 Kickstart file /run/install/ks.cfg is missing. While booting the iso: [ OK ] Finished systemd-vconsole-setup.service - Virtual Console Setup. [ 11.697574] dracut-initqueue[1391]: Traceback (most recent call last): [ 11.752351] dracut-initqueue[1391]: File "/sbin/parse-kickstart", line 35, in <module> [ 11.755574] dracut-initqueue[1391]: from pykickstart.parser import KickstartParser, preprocessKickstart [ 11.755693] dracut-initqueue[1391]: File "/usr/lib/python3.13/site-packages/pykickstart/parser.py", line 40, in <module> [ 11.755797] dracut-initqueue[1391]: from pykickstart import constants, version [ 11.755865] dracut-initqueue[1391]: File "/usr/lib/python3.13/site-packages/pykickstart/version.py", line 55, in <module> [ 11.755931] dracut-initqueue[1391]: from pykickstart.load import load_to_str [ 11.756046] dracut-initqueue[1391]: File "/usr/lib/python3.13/site-packages/pykickstart/load.py", line 20, in <module> [ 11.756152] dracut-initqueue[1391]: import requests [ 11.756217] dracut-initqueue[1391]: File "/usr/lib/python3.13/site-packages/requests/__init__.py", line 164, in <module> [ 11.756304] dracut-initqueue[1391]: from .api import delete, get, head, options, patch, post, put, request [ 11.756371] dracut-initqueue[1391]: File "/usr/lib/python3.13/site-packages/requests/api.py", line 11, in <module> [ 11.756430] dracut-initqueue[1391]: from . import sessions [ 11.756490] dracut-initqueue[1391]: File "/usr/lib/python3.13/site-packages/requests/sessions.py", line 15, in <module> [ 11.756550] dracut-initqueue[1391]: from .adapters import HTTPAdapter [ 11.756610] dracut-initqueue[1391]: File "/usr/lib/python3.13/site-packages/requests/adapters.py", line 81, in <module> [ 11.756666] dracut-initqueue[1391]: _preloaded_ssl_context.load_verify_locations( [ 11.756767] dracut-initqueue[1391]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ [ 11.756855] dracut-initqueue[1391]: extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH) [ 11.756925] dracut-initqueue[1391]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [ 11.756987] dracut-initqueue[1391]: ) [ 11.757140] dracut-initqueue[1391]: ^ [ 11.757221] dracut-initqueue[1391]: FileNotFoundError: [Errno 2] No such file or directory [ 11.843396] dracut-initqueue[1371]: /lib/anaconda-lib.sh: line 306: /tmp/ks.info: No such file or directory In OpenQA: https://openqa.fedoraproject.org/tests/2727458#step/_boot_to_anaconda/9 Reproducible: Always Steps to Reproduce: 1. Boot the latest iot-installer iso - https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-41-20240712.0/compose/IoT/ Actual Results: anaconda 41.24-1.fc41 for Fedora 41 (pre-release) started. * installation log files are stored in /tmp during the installation * shell is available on TTY2 and in second TMUX pane (ctrl+b, then press 2) * when reporting a bug add logs from /tmp as separate text/plain attachments 16:23:20 Kickstart file /run/install/ks.cfg is missing The iot-installer uses an embedded kickstart with a single line: cat osbuild.ks ostreesetup --osname=fedora-iot --url=file:///run/install/repo/ostree/repo --ref=fedora/rawhide/x86_64/iot --remote=fedora-iot --nogpg
Created attachment 2039498 [details] anaconda.log
Created attachment 2039499 [details] syslog
Created attachment 2039500 [details] program.log
Created attachment 2039501 [details] dbus.log
I'm gonna say this is an automatic blocker per "Complete failure of any release-blocking image to boot at all under any circumstance - "DOA" image (conditional failure is not an automatic blocker)" - https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Automatic_blockers . The IoT installer is a release-blocking image per https://docs.fedoraproject.org/en-US/releases/f40/blocking/ (the F41 blocking page doesn't exist yet).
so it took me a minute, but the problem here is that the '/tmp/ks.info' file that line 306 of /lib/anaconda-lib.sh is looking for is the *output* of the /sbin/parse-kickstart command that's crashing. line 304 of anaconda-lib.sh calls /sbin/parse-kickstart to produce /tmp/ks.info ; that's crashing , so we don't have /tmp/ks.info when we get to line 306, so we get the `/tmp/ks.info: No such file or directory" error. parse-kickstart is failing because it imports requests, and requests can't find its default CA bundle, apparently. This is still a bit odd...the `import requests` in parse-kickstarts is not new. So I'm not sure what's changed here, exactly. We used to have the CA bundle it's looking for in the initramfs but somehow it's dropped out? Not sure this is an anaconda bug, anyhow.
okay, so it's looking for /etc/pki/tls/certs/ca-bundle.crt , and indeed in the initramfs environment on the IoT installer image, that file is not there. But...it's also not there on the F40 IoT installer image, and `import requests` works OK there. So this is a change in requests, it seems: on F40, you can `import requests` without the CA cert bundle being present, but on F41 you cannot. Will look into that further (but will also look into why the CA cert bundle isn't there, as it seems like it probably should be, we will probably have problems with e.g. doing `inst.ks=https://www.somewhere.com/some.ks` without it).
Ah, I see what's going on here. https://github.com/psf/requests/pull/6724 almost fixed it, but not quite - that fixes it for ImportError, which you get if the ssl module isn't present, but not for FileNotFoundError, which you get if the bundle isn't present. I'll send a follow-up PR and backport it to Rawhide.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-52f1cfff36 should fix this when it goes stable, using https://github.com/psf/requests/pull/6781 . When https://github.com/psf/requests/pull/6767 is ready we should switch to that, seems like upstream will be going with that.
Fix confirmed in openQA - https://openqa.fedoraproject.org/tests/2763814