This has been a common problem when writing the iso image to a USB device on a Windows system. Further analysis has shown that for some reason the bachkup LBA pointer in the GPT header are modified as seen in this dump 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000 EFI PART....\... 00000210: 3308 0146 0000 0000 0100 0000 0000 0000 3..F............ ^^^^ ^^^^ CRC-32 of header (offset +0 to +0x5B) in little endian, with this field zeroed during calculation 60d5 17d8 00000220: 3fb9 5100 0000 0000 4000 0000 0000 0000 ?.Q.....@....... ^^^^ ^^^^ Backup LBA (location of the other header copy) == 5355839 d6a3 7007 == 124822486 00000230: 00b9 5100 0000 0000 f3c5 692f f6f1 ab44 ..Q.......i/...D ^^^^ ^^^^ Last usable LBA for partitions (secondary partition table first LBA − 1) == 5355776 97a3 7007 == 124822423 00000240: ba1d da42 3414 f8e7 0200 0000 0000 0000 ...B4........... 00000250: f800 0000 8000 0000 7f10 99b2 0000 0000 ................ 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000270: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000280: 0000 0000 0000 0000 0000 0000 0000 0000 ................ Actually, the backup LBA should be at the end of the USB device. But when creating the iso image, the size of the USB device is unknown so it is impossible to do this correctly. It looks like Windows tries to fix this, and in doing so, makes the iso check invalid. Is it possible, somehow, to make the implantisomd5 and checkisomd5 immune to this modification of the BLBA pointers? Reproducible: Always
More details: https://discussion.fedoraproject.org/t/install-media-sometimes-fail-media-check-at-4-8/108478 https://discussion.fedoraproject.org/t/talk-install-media-sometimes-fail-media-check-at-4-8/109570 https://github.com/FedoraQt/MediaWriter/issues/669
In my opinion the check is doing its job. The goal is to make sure the bits are the same as what was written to the device and they aren't.
From a strict technical point of view, that is true. The problem is that for lots of people, it's *impossible* to validate the install medium. Just because some OS or UEFI firmware decided to "fix invalid GPT table" (note - there might be other modifications, but this is the one we currently identified). I personally tried to install Fedora on one such laptop, and if UEFI firmware behaves this way, there is no alternative. In that case, not only the checksum verification loses its purpose, it actually does more harm than good, because people are scared to run the installer, and we get a lot of negative PR on Fedora channels complaining about broken tooling. If we could verify only the "important bits" and not the whole medium, it would be an improvement for us. If GPT table is broken, the installer won't boot anyway (and especially the backup table is of little concern to us). If we can verify just the actual partition contents, I believe that is completely good enough for us. Let's not forget that the actual full image checksum was already verified by Fedora Media Writer after writing. In other words, users don't ask the question "are written bits the same?", they ask the question "is it safe to proceed with installation?", and we currently can't answer that. Our current official answer is "if the error shows 4.8%, then most likely yes, please proceed", which is quite a lame answer, and I'd like to provide something better.
For reference this was also recently discussed at https://old.reddit.com/r/linux4noobs/comments/1q5nng8/fedora_media_check_always_fails_help_pls/ and, IMO, saying that the check is doing its job and that users should just "know" when to ignore a failure there is doing a massive disservice to all Windows users who want to try Fedora, because they will all see a failed media validation. And if you want to go the "technically correct" route, then the root of the matter is that, the current Fedora ISO does NOT create a valid GPT bootable media, period. The minute you write the ISOHybrid, byte-for-byte, to a media that is not the exact same size as the image is the minute you have produced a broken GPT drive. And there are some platforms out there that, when seeing main and backup GPT in disagreement, may choose not to boot the media altogether. And sure, your check might say that the media is 100% identical to the source image, if you wrote it from a platform (Linux, Mac?) that doesn't automatically fix broken GPT, but that still doesn't mean that your media is valid, if the data that falls outside of the ISO range is supposed, per UEFI specs, to be laid out in a certain way, and the media you created breaks the specs. Which means that, no matter the OS you used to write the media, you end up with a broken media, either because the internal media check will report it as such, or because it is not specs-compliant for UEFI. So, *if* Fedora wants to help its new users, and especially the ones coming from Windows, I believe it has 3 options: 1. Switch the partition scheme to MBR, since UEFI supports GPT and UEFI equally (here too, it's in the UEFI specs that BOTH should be equally supported), and, since you're never going to have >2TB ISOs or tons of separate partitions, you're never going to run into MBR limitations. And if your concern if to ensure that the media cannot be booted in BIOS/Legacy, you can also easily add an MBR bootloader that displays a message alerting the user about legacy boot (Similar to https://raw.githubusercontent.com/wiki/pbatard/rufus/images/legacy_boot.png, from which you can even reuse the GPL licensed code at https://github.com/pbatard/rufus/blob/master/res/mbr/msg.S). 2. Have the check ignore the 3 32-bit fields at address 0x210, 0x220 and 0x230, that get modified by Windows when it automatically fixes the GPT. 3. Follow the UEFI recommendations and work at the file system level instead of (as we are explicitly seeing here) the all too brittle sector level, by parsing an md5sum/sha256sum (similar to what the Ubuntu media check does/used to do) and validating the ISO-9660 files individually. Sure, you are going to lose some partition or file system metadata, but this is actually what you want because this has the much added benefit of then allowing users, especially the ones on Windows (where dd does not come as standard, so any distros that recommends dd-like creation FORCES Windows users to first install a third party utility), to use "File System Transposition" to simply create their boot media by just extracting the ISO files onto a FAT32 GPT media, that they partitioned and formatted themselves (using the OS NATIVE utilities). Personally, with UEFI boot having been designed to work at the file system level rather than the sector level like BIOS/Legacy did, it always seemed to me kind of backwards that Linux distro maintainers focusing on UEFI boot put all their eggs in the ISOHybrid basket and kind of force their users to use sector-by-sector copy, instead of giving them the CHOICE to use either dd or File System Transposition.
(In reply to Pete Batard from comment #4) > ... > > 1. Switch the partition scheme to MBR, since UEFI supports GPT and UEFI > equally (here too, it's in the UEFI specs that BOTH should be equally > supported), and, since you're never going to have >2TB ISOs or tons of > separate partitions, you're never going to run into MBR limitations. And if > your concern if to ensure that the media cannot be booted in BIOS/Legacy, > you can also easily add an MBR bootloader that displays a message alerting > the user about legacy boot (Similar to > https://raw.githubusercontent.com/wiki/pbatard/rufus/images/legacy_boot.png, > from which you can even reuse the GPL licensed code at > https://github.com/pbatard/rufus/blob/master/res/mbr/msg.S). > > ... The SUSE iso images is doing exactly that. It also have a slightly modified checksum utility similar to checkisomd5. If it works for SUSE, it should work for Fedora. This is seen with openSUSE-Tumbleweed-DVD-x86_64-Snapshot20251211-Media.iso.
Well it seems switching to MBR is the simplest path to fixing this, by far. a. UEFI spec says tools that create or modify GPT need to do so correctly, Fedora Media Writer is cloning a GPT, it's not clear to me it's obligated to rewrite the GPT correctly b. Windows rewriting the GPT seems like a misfeature to me because it's not a tool that creates or modifies GPTs in and of itself c. but OK if GPT, then either the cloning tool or OS are likely to modify the GPT so that it's correct (per the spec), therefore d. isomd5sum needs to be modified to exclude all the GPT portions from measurement or else render itself unable to support GPT, which takes us back to using MBR
My vague recollection how isomd5sum works is in chunks. Therefore, in retrospect, I'm not sure this provides the granularity that may be required to exactly exclude only the GPT from the check. We've considered dropping the isomd5sum verification on the devel@ mailing list at least once or twice. The argument at the time was xz and zstd are intolerant to trivial corruptions during decompression, thereby indirectly detecting them. I'm not sure that's entirely reliable. Exploring alternatives or redesign in this bug report is probably out of scope. And I agree there isn't anything to fix in isomd5sum.
(In reply to Chris Murphy from comment #7) > My vague recollection how isomd5sum works is in chunks. Therefore, in > retrospect, I'm not sure this provides the granularity that may be required > to exactly exclude only the GPT from the check. > But it does explicitly exclude byte 0x00008373 through 0x00008572 by zeroing these bytes before doing the calculation. That is where the checksum values are stored on the iso image.
(In reply to Chris Murphy from comment #7) > My vague recollection how isomd5sum works is in chunks. Therefore, in > retrospect, I'm not sure this provides the granularity that may be required > to exactly exclude only the GPT from the check. Chris, I'm not sure if you saw the work already done by TK here? https://discussion.fedoraproject.org/t/109570/45 https://discussion.fedoraproject.org/t/109570/49 I believe we need to do *something* here, this is long-standing pain point for Fedora, I'm just not sure what. Patch isomd5sum, adopt the SUSE approach, ... Or would dropping the ISOHybrid format improve the situation? I don't think we require the image to look like a DVD ISO anymore, optical boot is no longer a release-blocking requirement.
This package has changed maintainer in Fedora. Reassigning to the new maintainer of this component.