Description of problem: I'd like to be able to use libvirt's built in ipxe support for downloading/running kernel/initrd files from a HTTPS server. Looks like this should be able to be done by setting the DOWNLOAD_PROTO_HTTPS #define. See https://ipxe.org/buildcfg/download_proto_https. Maybe there is a reason this isn't done already. If not it seems like it would be useful to have. Version-Release number of selected component (if applicable): Fedora 31. ipxe-roms-qemu-20190125-3.git36a4c85f.fc31.noarch
Gerd, any reason not to enable DOWNLOAD_PROTO_HTTPS for ipxe?
(In reply to Cole Robinson from comment #1) > Gerd, any reason not to enable DOWNLOAD_PROTO_HTTPS for ipxe? Doesn't hurt to have it, even though I fail to see the point to use https on a boot server. I don't think ipxe does full certificate verification, so using https instead of http doesn't change much security wise ... We need to keep an eye on the total rom size though, not sure how much https would add there, size should stay below 256k for live migration compatibility.
(In reply to Gerd Hoffmann from comment #2) > (In reply to Cole Robinson from comment #1) > > Gerd, any reason not to enable DOWNLOAD_PROTO_HTTPS for ipxe? > > Doesn't hurt to have it, even though I fail to see the point to use https on > a boot server. > I don't think ipxe does full certificate verification, so using https > instead of http doesn't > change much security wise ... That is interesting. Even without the security element it's possible kernel/initrd images are hosted on a server/service that *only* supports https. In this case if you wanted to supply an iPXE script that used a remote kernel/initrd you'd have to download and rehost them somewhere else that supports http in order to use them. Download/re-host is really easy for some people to do and not so easy for others. Either way it's a barrier that we can hopefully overcome.
(In reply to Gerd Hoffmann from comment #2) > Doesn't hurt to have it, even though I fail to see the point to use https on > a boot server. > I don't think ipxe does full certificate verification, so using https > instead of http doesn't change much security wise ... The reality of modern infra is that HTTPS fast becoming non-optional. The web server hosting the boot images may well host other files for which https is requisite, so we can't just tell people to turn off https on servers as an answer. Regardless of whether ipxe currently does useful cert validation or not, we really do need to support https, otherwise it will be increasingly useless.
> The reality of modern infra is that HTTPS fast becoming non-optional. The > web server hosting the boot images may well host other files for which https > is requisite, so we can't just tell people to turn off https on servers as > an answer. Regardless of whether ipxe currently does useful cert validation > or not, we really do need to support https, otherwise it will be > increasingly useless. Agree. That is the --verbose version of my "doesn't hurt to have it". Even if https support in ipxe doesn't actually help improving security for boot it can make your life easier in a world where https is becoming the new default ...
+1 Does anyone know of a link or issue somewhere where the HTTPS in iPXE doesn't really do cert validation? This page says: "In the default configuration, iPXE will therefore automatically trust the same set of certificates as the Firefox web browser.": https://ipxe.org/crypto
(In reply to Dusty Mabe from comment #6) > +1 > > Does anyone know of a link or issue somewhere where the HTTPS in iPXE > doesn't really do cert validation? > > This page says: "In the default configuration, iPXE will therefore > automatically trust the same set of certificates as the Firefox web > browser.": https://ipxe.org/crypto Oh. Ok, seems my info is *very* outdated. The claim was true for the initial tls/https implementation. Server certificate checks have been added in 2012 according to the git log.
bump. any chance this change is being considered?
(In reply to Dusty Mabe from comment #8) > bump. any chance this change is being considered? Yeah I'll give it try tomorrow
Gerd I just tried building with DOWNLOAD_PROTO_HTTPS and it spit out this error, looks like romfile exceeding 256k but my recollection is fuzzy about that stuff. Any thoughts? + truncate -s '>256K' bin-combined/10222000.rom ++ stat -c %s bin-combined/10222000.rom + test 308224 -le 262144 error: Bad exit status from /var/tmp/rpm-tmp.dUdGMK (%build) Comes from here in the spec: EfiRom -d bin-combined/${rom}.rom # truncate to at least 256KiB truncate -s \>256K bin-combined/${rom}.rom # verify rom fits in 256KiB test $(stat -c '%s' bin-combined/${rom}.rom) -le $((256 * 1024))
Yep, seems https makes things alot larger. Which roms are in combined? efi ia32 included still? dropping that would be easiest probably to keep the total size below 256k.
(In reply to Gerd Hoffmann from comment #11) > Yep, seems https makes things alot larger. > Which roms are in combined? efi ia32 included still? dropping that would > be easiest probably to keep the total size below 256k. Yeah setting '%global edk2_ia32 0' lets the build complete. Is there anything to consider with disabling that or is it safe for Fedora? Other bit to consider is that due to RHEL crypto policies generally only blessing a select few crypto libraries, this probably won't ever be suitable to enable in RHEL (though if using pxe+UEFI HTTPS boot works, via openssl). Just figured I'd mention it, Dusty I'm not sure if you had a specific usecase in mind that could be affected. But if no one objects I'll push a build next week
> Yeah setting '%global edk2_ia32 0' lets the build complete. Is there > anything to consider with disabling that or is it safe for Fedora? IMHO very unlikely to cause any problems. You'll have a hard time finding hardware and distros which actually support ia32 uefi because that never took off. Broad uefi support basically came to late for ia32, it happened at a time where most systems where x64-capable anyway so vendors switched from legacy bios straight to x64 uefi, skipping ia32 uefi. Exceptions do exist though.
(In reply to Cole Robinson from comment #12) > (In reply to Gerd Hoffmann from comment #11) > > Yep, seems https makes things alot larger. > > Which roms are in combined? efi ia32 included still? dropping that would > > be easiest probably to keep the total size below 256k. > > Yeah setting '%global edk2_ia32 0' lets the build complete. Is there > anything to consider with disabling that or is it safe for Fedora? > > Other bit to consider is that due to RHEL crypto policies generally only > blessing a select few crypto libraries, > this probably won't ever be suitable to enable in RHEL (though if using > pxe+UEFI HTTPS boot works, via openssl). > Just figured I'd mention it, Dusty I'm not sure if you had a specific > usecase in mind that could be affected. My specific use case here was that I wanted to test iPXE's support of HTTPS to see if was causing performance issues users were seeing [1]. The only option I had was to use http because ipxe wasn't built with HTTPS support. The HTTP performance was fine and I couldn't test HTTPS because ipxe wasn't compiled with support for it, so I opened this request because I could see that upstream did support it. I think if it doesn't make it into RHEL it's not the end of the world, but I could see people wanting to be able to start their VMs using an HTTPS remote too. [1] https://github.com/coreos/fedora-coreos-tracker/issues/390 > > But if no one objects I'll push a build next week +1
(In reply to Gerd Hoffmann from comment #5) > Even if https support in ipxe doesn't actually help improving security > for boot it can make your life easier in a world where https is > becoming the new default ... Apologies for getting rather philosophical here, but do we realize how *wrong* this is? "HTTPS Everywhere" is touted as a big security improvement for everyone. So we're disabling plain HTTP. And when someone *does* need plain HTTP, we still give them HTTPS, except we tell them -- out-of-band --, "don't really trust that letter "S" in HTTPS, just this once, because there's no cert validation". So it's worse than before. Before, you'd just look at the scheme, and (upon seeing HTTP) you'd *know* you couldn't expect confidentiality / integrity. Now, with HTTPS, you may or may not have a MITM. (Of course, even before, HTTPS with no or weak cert validation existed, but noone enabled or supported that on purpose; they'd just pick HTTP then.) On the flip side, "HTTPS Everywhere" *does* break white-hat activities such as easy tunneling / inspecting of web service traffic. Not to mention the wattage wasted on (ultimately untrustworthy) encryption and decryption. Anyway this is not an argument against changing iPXE build flags; I understand we're not dictating the above status quo, just accommodating it with iPXE build flags. I'll get off my soap box now.
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle. Changing version to 33.
This is in ipxe-20190125-7.git36a4c85f.fc33 now
This caused a regression booting UEFI VMs, they just hang. Has nothing to do with PXE booting. Reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1869102 Affects existing UEFI VMs and newly created ones. I confirmed it on f32 with the busted ipxe It doesn't seem to be related to edk2_ia32 which was disabled to make space for HTTPS support. If I do a build with both bits disabled, the regression is gone. I tried a build with both options enabled and the rom size bumped up to 512. The hang still happened but maybe bumping rom size causes its own issues. I don't understand the ipxe + qemu interactions much So for now I pushed builds reverting the HTTPS change
Interesting.. upstream bug?
Yes, this is an upstream iPXE regression, which I had bisected to commit d8c500b7945e ("[efi] Drop to TPL_APPLICATION when gathering entropy", 2018-03-12): https://bugs.launchpad.net/ipxe/+bug/1882671/comments/7 Please see the same discussion for Ubuntu in the following launchpad ticket: "unbalanced UEFI TPL manipulations in iPXE with DOWNLOAD_PROTO_HTTPS enabled" https://bugs.launchpad.net/ipxe/+bug/1882671 Note that, meanwhile, Michael Brown has fixed the actual problem in upstream iPXE: https://bugs.launchpad.net/ipxe/+bug/1882671/comments/32 so you can re-enable DOWNLOAD_PROTO_HTTPS if you backport upstream iPXE commit 2ae5d4338661 ("[efi] Raise TPL during driver entry point", 2020-06-30).
Thanks for the info Laszlo! The fix backports cleanly and I confirmed it fixes the edk2 hang. It seems to unconditionally slow down edk2 boot from 5-10 seconds though, I'm guessing since this hits entropy gathering code. Kinda crappy though I'll probably wait until next week to push a build incase any one wants to comment or object
(In reply to Cole Robinson from comment #21) > Thanks for the info Laszlo! > > The fix backports cleanly and I confirmed it fixes the edk2 hang. It seems > to unconditionally slow down edk2 boot from 5-10 seconds though, I'm > guessing since this hits entropy gathering code. Kinda crappy though > > I'll probably wait until next week to push a build incase any one wants to > comment or object You can choose to enable DOWNLOAD_PROTO_HTTPS in *only* the traditional BIOS build of iPXE, and keep it disabled in the UEFI build of iPXE (in addition to using CONFIG=qemu): - In a VM using SeaBIOS, the only option for HTTPS Boot is iPXE. - But in a VM using OVMF, HTTPS Boot is offered by the edk2 infrastructure itself. And because we already build iPXE with "CONFIG=qemu", the iPXE UEFI driver image will not "hijack" control from the platform firmware (iPXE will not behave like a UEFI application), it will only provide the Simple Network Protocol interface for the rest of edk2 to consume. Therefore, the entropy gathering that's enabled with DOWNLOAD_PROTO_HTTPS can be considered pure waste in any "CONFIG=qemu" UEFI build of iPXE. I suggest re-enabling DOWNLOAD_PROTO_HTTPS in the BIOS build of iPXE only, with or without backporting upstream commit 2ae5d4338661. (Upstream commit 2ae5d4338661 is relevant for the UEFI build of iPXE only -- but for that case, I'm already suggesting keeping DOWNLOAD_PROTO_HTTPS disabled, in which case the regression is not observable in the first place.)
Yep, makes sense to only do this for SeaBIOS usage, especially since we already had to special case disable edk2_ia32
Okay that will take some change to the ipxe.spec build infrastructure, maybe it's simple but I can't say for certain since I don't touch ipxe much. If someone else wants to jump in and figure it out be my guest, but otherwise this is moving down my priority list for a bit
The documentation on upstream commit c801cb29d647 ("[build] Allow for named configurations at build time", 2014-08-20) is very helpful. It explains how the config overrides stack. In particular, as of current upstream HEAD (ef2c844d01e7), "src/config/general.h" still ends with the following snippet, coming from commit c801cb29d647: > #include <config/named.h> > #include NAMED_CONFIG(general.h) > #include <config/local/general.h> > #include LOCAL_NAMED_CONFIG(general.h) In addition, "src/config/general.h" *starts* with the following: > #include <config/defaults.h> Putting both facts together, it emerges that we can rely on macros coming from <config/defaults.h> in our overrides. In turn, here's why <config/defaults.h> is interesting: it consumes the PLATFORM macro, and includes a header file matching PLATFORM: > #define CONFIG_DEFAULTS(_platform) <config/defaults/_platform.h> > #include CONFIG_DEFAULTS(PLATFORM) Let's see what the possible values for PLATFORM are: > src/config/defaults/efi.h > src/config/defaults/linux.h > src/config/defaults/pcbios.h So we got "efi", "linux", and "pcbios". (As a side comment, consider the absolutely beautiful PLATFORM safety measure that is commit de37652044d5, "[efi] Prevent EFI code from being linked in to non-EFI builds", 2017-05-19!) As indicated by the message on commit de37652044d5, and also from comparing "src/config/defaults/efi.h", "src/config/defaults/linux.h", and "src/config/defaults/pcbios.h", the macro that we're looking for, coming from <config/defaults.h>, and consumable in our platform override(s), is IMAGE_EFI. --*-- And so we should append the following to "src/config/local/general.h" (in dist-git) [1]: > /* > * If PLATFORM is "efi", then <src/config/defaults/efi.h> gets included, > * and it defines IMAGE_EFI. No other PLATFORM defines IMAGE_EFI. Enable > * HTTPS for PLATFORMs *different* from "efi". > */ > #ifndef IMAGE_EFI > # define DOWNLOAD_PROTO_HTTPS > #endif (In the longer term, we should attempt to upstream this to "src/config/qemu/general.h".) --*-- For testing whether the above patch was effective, I figured I'd look at the *sizes* of the binaries, as including HTTPS must have a nontrivial impact on the binary sizes. I covered the following NICs: - pcnet32: 1022:2000 - ne2k_pci: 10ec:8029 - rtl8139: 10ec:8139 - virtio-net: 1af4:1000 - e1000: 8086:100e - igbvf: 8086:1520 - igb: 8086:1521 * With this patch [1] added, I get the following file sizes: > -rw-------. 1 83456 2020-08-19 18:38:47 +0200 src/bin/10222000.rom > -rw-------. 1 83456 2020-08-19 18:38:47 +0200 src/bin/10ec8029.rom > -rw-------. 1 84480 2020-08-19 18:38:47 +0200 src/bin/10ec8139.rom > -rw-------. 1 84480 2020-08-19 18:38:47 +0200 src/bin/1af41000.rom > -rw-------. 1 83968 2020-08-19 18:38:47 +0200 src/bin/8086100e.rom > -rw-------. 1 83968 2020-08-19 18:38:47 +0200 src/bin/80861520.rom > -rw-------. 1 83968 2020-08-19 18:38:47 +0200 src/bin/80861521.rom > > -rw-------. 1 152224 2020-08-19 18:38:49 +0200 src/bin-x86_64-efi/10222000.efidrv > -rw-------. 1 151744 2020-08-19 18:38:49 +0200 src/bin-x86_64-efi/10ec8029.efidrv > -rw-------. 1 154752 2020-08-19 18:38:49 +0200 src/bin-x86_64-efi/10ec8139.efidrv > -rw-------. 1 154528 2020-08-19 18:38:49 +0200 src/bin-x86_64-efi/1af41000.efidrv > -rw-------. 1 155872 2020-08-19 18:38:49 +0200 src/bin-x86_64-efi/8086100e.efidrv > -rw-------. 1 153056 2020-08-19 18:38:49 +0200 src/bin-x86_64-efi/80861520.efidrv > -rw-------. 1 155872 2020-08-19 18:38:49 +0200 src/bin-x86_64-efi/80861521.efidrv That is, approx. 82K for the BIOS bins, approx. 151K for the EFI drivers. * With the patch [1] removed (= globally disabling DOWNLOAD_PROTO_HTTPS): > -rw-------. 1 68096 2020-08-19 18:43:06 +0200 src/bin/10222000.rom > -rw-------. 1 68096 2020-08-19 18:43:06 +0200 src/bin/10ec8029.rom > -rw-------. 1 69120 2020-08-19 18:43:06 +0200 src/bin/10ec8139.rom > -rw-------. 1 69120 2020-08-19 18:43:06 +0200 src/bin/1af41000.rom > -rw-------. 1 68608 2020-08-19 18:43:06 +0200 src/bin/8086100e.rom > -rw-------. 1 68608 2020-08-19 18:43:06 +0200 src/bin/80861520.rom > -rw-------. 1 68608 2020-08-19 18:43:06 +0200 src/bin/80861521.rom > > -rw-------. 1 152224 2020-08-19 18:43:07 +0200 src/bin-x86_64-efi/10222000.efidrv > -rw-------. 1 151744 2020-08-19 18:43:07 +0200 src/bin-x86_64-efi/10ec8029.efidrv > -rw-------. 1 154752 2020-08-19 18:43:07 +0200 src/bin-x86_64-efi/10ec8139.efidrv > -rw-------. 1 154528 2020-08-19 18:43:07 +0200 src/bin-x86_64-efi/1af41000.efidrv > -rw-------. 1 155872 2020-08-19 18:43:07 +0200 src/bin-x86_64-efi/8086100e.efidrv > -rw-------. 1 153056 2020-08-19 18:43:07 +0200 src/bin-x86_64-efi/80861520.efidrv > -rw-------. 1 155872 2020-08-19 18:43:07 +0200 src/bin-x86_64-efi/80861521.efidrv That is, approx. 67K for the BIOS bins, approx. 151K for the EFI drivers. Meaning that, undoing the patch, the BIOS bins shrink (because they lose HTTPS), but the EFI drivers don't change (because they never got HTTPS). * And with DOWNLOAD_PROTO_HTTPS #defined *without* regard to IMAGE_EFI: > -rw-------. 1 83456 2020-08-19 18:44:03 +0200 src/bin/10222000.rom > -rw-------. 1 83456 2020-08-19 18:44:03 +0200 src/bin/10ec8029.rom > -rw-------. 1 84480 2020-08-19 18:44:03 +0200 src/bin/10ec8139.rom > -rw-------. 1 84480 2020-08-19 18:44:03 +0200 src/bin/1af41000.rom > -rw-------. 1 83968 2020-08-19 18:44:03 +0200 src/bin/8086100e.rom > -rw-------. 1 83968 2020-08-19 18:44:03 +0200 src/bin/80861520.rom > -rw-------. 1 83968 2020-08-19 18:44:03 +0200 src/bin/80861521.rom > > -rw-------. 1 187360 2020-08-19 18:44:03 +0200 src/bin-x86_64-efi/10222000.efidrv > -rw-------. 1 186912 2020-08-19 18:44:03 +0200 src/bin-x86_64-efi/10ec8029.efidrv > -rw-------. 1 189920 2020-08-19 18:44:03 +0200 src/bin-x86_64-efi/10ec8139.efidrv > -rw-------. 1 189728 2020-08-19 18:44:03 +0200 src/bin-x86_64-efi/1af41000.efidrv > -rw-------. 1 190976 2020-08-19 18:44:03 +0200 src/bin-x86_64-efi/8086100e.efidrv > -rw-------. 1 188192 2020-08-19 18:44:03 +0200 src/bin-x86_64-efi/80861520.efidrv > -rw-------. 1 190976 2020-08-19 18:44:03 +0200 src/bin-x86_64-efi/80861521.efidrv That is, approx. 82K for the BIOS bins, approx. 183K for the EFI drivers. The BIOS bins grow back (they get HTTPS again), and now the EFI drivers grow too (because they *too* get the HTTPS stack). This indicates that "#ifndef IMAGE_EFI" is suitable for restricting DOWNLOAD_PROTO_HTTPS to BIOS builds. (Well, also to the "linux" PLATFORM, but we never attempt to build those binaries.)
Created attachment 1712164 [details] proposed dist-git patch (on top of commit f0fdc922ca2e) Scrach build in Koji: https://koji.fedoraproject.org/koji/taskinfo?taskID=49796111 Dusty / Cole, can you please check this? Thanks.
Thanks Laszlo! That fixes things for me rawhide builds are not completing at the moment though but it's not related to this patch. I'll try again in a week
FEDORA-2020-ee0ee6ef0a has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-ee0ee6ef0a
I coupled this with an update to ipxe.git so a re-test of the HTTPS support will help
FEDORA-2020-ee0ee6ef0a has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-ee0ee6ef0a` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-ee0ee6ef0a See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Seems to be working for me with `ipxe-roms-qemu-20200823-1.git4bd064de.fc33` when trying to grab https artifacts from a remote location. Here is my ipxe config: ``` #!ipxe set base-url https://builds.coreos.fedoraproject.org/prod/streams/next/builds/32.20200907.1.0/x86_64 set version 32.20200907.1.0 kernel ${base-url}/fedora-coreos-${version}-live-kernel-x86_64 console=ttyS0 ignition.firstboot ignition.platform.id=metal coreos.inst.install_dev=sda coreos.inst.ignition_url=https://dustymabe.fedorapeople.org/user-systemd.ign initrd ${base-url}/fedora-coreos-${version}-live-initramfs.x86_64.img initrd ${base-url}/fedora-coreos-${version}-live-rootfs.x86_64.img boot ``` and here is the output: ``` iPXE 1.0.0+ (4bd064de) -- Open Source Network Boot Firmware -- http://ipxe.org Features: DNS HTTP HTTPS iSCSI TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT net0: 52:54:00:95:ae:a4 using 82540em on 0000:00:02.0 (open) [Link:up, TX:0 TXE:0 RX:0 RXE:0] Configuring (net0 52:54:00:95:ae:a4).................. ok net0: 192.168.122.150/255.255.255.0 gw 192.168.122.1 net0: fe80::5054:ff:fe95:aea4/64 Next server: 192.168.122.1 Filename: http://192.168.122.1:8000/boot.ipxe http://192.168.122.1:8000/boot.ipxe... ok boot.ipxe : 776 bytes [script] https://builds.coreos.fedoraproject.org/prod/streams/next/builds/32.20200907.1.0/x86_64/fedora-coreos-32.20200907.1.0-live-kernel-x86_64.... ok https://builds.coreos.fedoraproject.org/prod/streams/next/builds/32.20200907.1.0/x86_64/fedora-coreos-32.20200907.1.0-live-initramfs.x86_64.img... ok https://builds.coreos.fedoraproject.org/prod/streams/next/builds/32.20200907.1.0/x86_64/fedora-coreos-32.20200907.1.0-live-rootfs.x86_64.img... ok Probing EDD (edd=off to disable)... ok ```
FEDORA-2020-ee0ee6ef0a has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.