Bug 2044108

This is more of a feature request than a bug. I can see the logic of ending at last usable, but as most all file systems now use 4096 byte block sizes anyway, if the last 512 byte sectors don't add up to 4096 bytes, they won't be used anyway.

I could add an option to do this, but it seems to me that this is more of a bug in cryptsetup than in GPT fdisk. I could find nothing in the UEFI/GPT specification that addresses where partitions should end, although there is such information relating to partition start points (on p. 118 of the UEFI 2.8 specification).

FWIW, I just did some tests, and Linux fdisk does the same thing as gdisk -- it ends a partition occupying the whole disk at the last usable sector, as defined in the GPT data structures. GNU parted ends it early, to align the end point on a 1024KiB boundary.

I'll give some thought to how such an endpoint-alignment feature might be implemented in GPT fdisk. The design philosophy of GPT fdisk is to give the user total control over the disk data structures, within the confines of what's valid in GPT (or sometimes even when it's not, but that is well defined outside of it, as in hybrid MBRs). Restricting end points in all cases would be a break from this design philosophy, and could create problems -- for instance, it would then become impossible to re-create a partition table that had set end points later than an aligned-endpoint requirement would dictate. Thus, to implement this feature, I would need to give users some way to specify that they want an aligned end-point rather than to use all available space, and perhaps to specify what alignment value is required. (4096 bytes is, as you say, common, but it's not guaranteed to be the best option in all situations, especially in the future.)

It's a good point that cryptsetup/dm-crypt probably needs to ignore or map out the last 512-byte sectors when it's a fraction of 4096 bytes, and report the resulting (reduced) logical block device accordingly. And for that matter, what's the behavior if the underlying partition ends on a 4096 byte boundary at create time, but subsequently is resized and then isn't? Does cryptsetup fail to open such a LUKS device? Seems not very fail safe. I'll start a thread on the cryptsetup list.

OK so what cryptsetup does by default is if physical sector size is 4096 bytes and the block device starts and ends on 4096 byte boundaries, you get a 4096 byte LUKS sector size. If there's misalignment in any way, including in my case of 7 dangling 512 byte sectors at the end, it falls back to using 512 byte sectors.

The reason it fails in the above case is because I passed --sector-size 4096 explicitly.

So yeah there isn't a bug here per se. But I still think it's suboptimal. The UEFI spec didn't really consider one way or another the fact it's the backup GPT that results in the misalignment in only the last "sector" at the end of the last partition.

Started a thread on the cryptsetup/dm-crypt list
https://marc.info/?l=dm-crypt&m=164306225923513&w=2

The problem is the last partition on the device. The standard partitions are aligned (by default) to 1MiB, but for the last partition fdisk/parted uses all usable space. That's a problem mostly on GPT where after the last partition is backup GPT header and size of this "hidden" area does not have to aligned to physical sectors if the logical sector is 512 bytes.

We can improve it easily, GPT header specifies an area where is possible to create partitions (aka. First and Last usable LBA), size of this area we need to align to physical sector size (or optimal I/O or 1MiB), after that the last partition will be aligned although it will be created by the arbitrary 3rd-party tool.

I'll prepare a patch for this issue for libfdisk.

(In reply to Karel Zak from comment #6)
> We can improve it easily, GPT header specifies an area where is possible to
> create partitions (aka. First and Last usable LBA), size of this area we
> need to align to physical sector size

This is the ideal solution. Unfortunately, useless for libfdisk, because it needs backward 
compatibility with already generated sfdisk dumps/script :-(

I have implemented it in another way. If the size of the last partition is not specified then
libfdisk aligns it to 1MiB (or optimal I/O, etc.) boundary.

Simple reproducer:

old version:

 # scsi_debug dev_size_mb=500 sector_size=512 physblk_exp=3

 # echo ",," | sfdisk --quiet --label gpt /dev/sdc
 # echo "$(lsblk --bytes -no SIZE /dev/sdc1) / (4*1024)" | bc -l
 127739.87500000000000000000
        ^^^^

new version:

 # echo ",," | sfdisk --quiet --label gpt /dev/sdc
 # echo "$(lsblk --bytes -no SIZE /dev/sdc1) / (4*1024)" | bc -l
 127488.00000000000000000000

This change will be available in util-linux v2.38.

Note that Linux fdisk (fdisk, sfdisk, cfdisk; in util-linux package) != GPT fdisk (gdisk, sgdisk, cgdisk; in the gdisk package). This bug report is filed against GPT fdisk, but the patch that Karel Zak describes is to Linux fdisk/util-linux.

Ending the last partition on a $min-io-size boundary seems reasonable. My understanding is that cryptsetup only supports 512-byte and 4096-byte LUKS2 sector size right now, and uses both physical sector size and  partition start and end alignment to that physical sector size to determine which LUKS sector size to use. Not min-io.

I'm not sure there ever really was a per se advantage to 1 MiB alignment.

FYI, 1 MiB alignment is used as a default because many hardware RAID arrays work best with higher alignment values than the 4 KiB alignment that's optimal for Advanced Format disks. Details differ depending on the RAID hardware, and I haven't checked the details recently, but in the past and IIRC, stripe sizes of 32 KiB to 256 KiB were common, with sizes of up to 512 KiB being used in some cases. 1 MiB is not much greater than 512 KiB, so it gives a little wiggle room should stripe sizes increase in the future without wasting all that much space on modern devices. The consequences of not aligning to RAID stripe size are not as great as the consequences of not aligning to 4 KiB boundaries for Advanced Format disks, but they are measurable.

SSDs also have their own optimal alignment issues, many of which are poorly documented. Most are closer to the 4 KiB alignment that's optimal for Advanced Format disks than the tens or hundreds of KiB that's optimal for RAID arrays. I've heard of some weird ones that are optimized on power-of-three boundaries rather than power-of-two boundaries, but I don't know how common they are. Overall, 1 MiB alignment helps with many SSDs without having to dig into the weeds of drive-by-drive specifics.

There are APIs to determine the optimal alignment for a given device, but when I was investigating this issue for GPT fdisk, those APIs returned unreliable results, so I didn't want to rely on them. They may be more reliable now; I haven't checked in quite a while.

Also, and more directly relevant to this bug report, I've implemented an option to align partition end points; however, I haven't yet uploaded this to the GPT fdisk public git repository because I still need to do more testing on it before unleashing a potentially buggy new feature on the world. It's handled differently, in UI terms, in each of the three GPT fdisk programs. In gdisk, the default end point is adjusted, but can be easily overridden just by entering another value (including the maximum one that's still displayed on the screen). In cgdisk, the partition size is adjusted and can also be overridden, but the user would need to do some arithmetic to do so. In sgdisk, the default remains to not align the end point, but it can be aligned by using a new command-line option ("-I", since all the more logical ones were already in use).

This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

FEDORA-2022-14b4ccfa1f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-14b4ccfa1f

FEDORA-2022-f02bc8d566 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f02bc8d566

FEDORA-2022-02a3900f62 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-02a3900f62

FEDORA-2022-02a3900f62 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-02a3900f62`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-02a3900f62

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-14b4ccfa1f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-14b4ccfa1f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-14b4ccfa1f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-f02bc8d566 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-f02bc8d566`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-f02bc8d566

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-f02bc8d566 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-14b4ccfa1f has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-02a3900f62 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.