LUKS2 format uses memory hard pbkdf. By default it aims to use half system memory top (or 1GB whatever hit first) but it may not be enough in your case. You may lower the memory footprint by --pbkdf-memory option (unit is KiB)

For example "cryptsetup luksFormat --pbkdf-memory 32768" will create device with keyslot having pbkdf memory cost set to 32MiBs.

Or you may increase RAM in your VM. I think 2GB would be enough.

Respectfully, I disagree about "not a bug" -- this is something that works in RHEL 8.3, Fedora, Debian, and everywhere else. Requiring 1 GB of RAM by default to do such an operation sounds quite ridiculous -- if --pbkdf-memory 32768 works, why isn't that the default then? It's not that hard to check how much memory is available. Such "unbreak my software" options are highly non-discoverable and quite pointless.

Increasing memory footprint of pbkdf is basically point of memory hard functions. I don't think this bug is proper place to discuss theory (and papers) behind it so I'm going to completely ignore all remarks about it being ridiculous, with all respect.

It seems to me you run VM with available system memory below recommended minimum values (see https://wiki.centos.org/About/Product, which is basically copy/paste from here: https://access.redhat.com/articles/rhel-limits). Exactly for small embedded systems or low memory VMs there's --pbkdf-memory argument. Or you may switch back to pbkdf2 with "--pbkdf pbkdf2". I'm not sure what the VM is supposed to test and what purpose the cryptsetup has in it. But we don't plan to change general defaults for custom tailored VMs.

I do not think the discussion needs to be heated here. It is all about trade-off usability vs. security.

We started to use memory-hard key derivation functions to keep our technology protecting long-term keys in disk encryption better. Mainly because existing algorithms are not acceptable if using massive GPUs cracking power etc. (See all papers about PBKDF2 problems).

That said, there is always a way how to downgrade requested parameters. Or even switch to PBKDF2 that was used in LUKS1 - but it is up to the user's decision.

As a default, resources are tested on minimal distro specs. (And there are some limits, based on experience.)
But still, there are situations where this does not work - the main problem is with the awful design of OOM behavior and various systems like ballooning and overcommitting that are designed for systems
where you allocate a lot of memory, but will never use that in reality.

In the memory-hard function, you WILL use that memory, and it must be physical memory (otherwise, run time increases almost exponentially).

Thanks for the detailed explanations! I don't want to bake a --pbkdf-memory option into cockpit, as that really shouldn't be the place to decide about what's a good default policy. But running with 1.5 GiB RAM works, and the document about the minimum RAM requirement makes sense then. I sent a PR to run the LUKS tests with more memory instead.

I believe this is still a bug in the sense that it should first check for available memory before triggering OOM - i.e. refuse to run with an expplanation message instead of ending up having the kernel invoke OOM. 

btw, I get OOM even on these numbers:

[root@localhost-live ~]# free -mh
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       236Mi       932Mi       0.0Ki       733Mi       919Mi
Swap:          5.8Gi       114Mi       5.7Gi

is that expected?

Reopening this, as this keeps haunting us. This may be possible to do with some /usr/local/sbin/cryptsetup wrapper which injects a --pbkdf-memory argument after a luksFormat. But we mostly consume this through udisks → libblockdev, and that uses libcryptsetup. As cryptsetup also does not have a configuration file, there is currently no way to influence this behaviour, and thus make cryptsetup work on smaller devices (think VMs, ARM boards, embedded devices, etc.)

So either cryptsetup needs to adjust its default PBDKF memory to what the system has actually available (prefered), or this should at least be possible to set in some /etc/cryptsetup.conf.

Thanks!

There were changes in how libcryptsetup handles argon2 benchmark wrt memory requirements. Also we dropped most code that were locking whole process in memory (we lock only key/secretes related memory currently). Those changes were not yet released officially but in general it should affect how the luksFormat operation behaves on systems with low memory. Here is a scratch build if you'd like to test it: https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=2777887. But it's main branch snapshot so there may be other issues.

Thanks Ondrej! I tested that build on current RHEL 9.3, and there isn't much visible difference. udisks still gets OOM-killed in an instance with 768MiB or 1 GiB RAM. It starts working with 1200 MiB RAM (although somewhat brittle), which is about the same threshold as with cryptsetup 2.6.0.

Could you share the debug output for the failing luksFormat operation? It really should not behave the same (see upstream report https://gitlab.com/cryptsetup/cryptsetup/-/issues/802). I've also tested it on a VM with very little memory available (<1GB). If the LUKS2 device is already created with key protected in a keyslot with higher than available memory requirements there's nothing we can do about it.

I'd like to see at least:
1) cryptsetup luksFormat --debug (or whatever service/task that performs failing keyslot add operation)
2) free -M output (in-before the operation)

I'm testing this against `modprobe scsi_debug dev_size_mb=100`, i.e. a 100 MiB RAM disk, in a VM with 800 MiB RAM.

`cryptsetup luksFormat --debug /dev/sda` with cryptsetup-2.6.0-3.el9.x86_64 in RHEL 9.3 proper:


Enter passphrase for /dev/sda: 
Verify passphrase: 
# Checking new password using default pwquality settings.
# Crypto backend (OpenSSL 3.0.7 1 Nov 2022 [default][legacy]) initialized in cryptsetup library version 2.6.0.
# Detected kernel Linux 5.14.0-362.1.1.el9_3.x86_64 x86_64.
# Only 1 active CPUs detected, PBKDF threads decreased from 4 to 1.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 372426kB.
# PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 372426, parallel_threads 1.
Existing 'crypto_LUKS' superblock signature on device /dev/sda will be wiped.
Existing 'crypto_LUKS' superblock signature on device /dev/sda will be wiped.
# Formatting device /dev/sda as type LUKS2.
# Auto-detected optimal encryption sector size for device /dev/sda is 512 bytes.
# Topology: IO (512/524288), offset = 0; Required alignment is 1048576 bytes.
# Checking if cipher aes-xts-plain64 is usable.
# Using userspace crypto wrapper to access keyslot area.
# Formatting LUKS2 with JSON metadata area 12288 bytes and keyslots area 16744448 bytes.
# Creating new digest 0 (pbkdf2).
# Setting PBKDF2 type key digest 0.
# Running pbkdf2(sha256) benchmark.
# PBKDF benchmark: memory cost = 0, iterations = 744727, threads = 0 (took 44 ms)
# PBKDF benchmark: memory cost = 0, iterations = 1028015, threads = 0 (took 510 ms)
# Benchmark returns pbkdf2(sha256) 1028015 iterations, 0 memory, 0 threads (for 512-bits key).
# Segment 0 assigned to digest 0.
# Device size 104857600, offset 16777216.
# Wiping LUKS areas (0x000000 - 0x1000000) with zeroes.
# Wiping keyslots area (0x008000 - 0x1000000) with random data.
# Reusing open rw fd on device /dev/sda
# Device size 104857600, offset 16777216.
# Acquiring write lock for device /dev/sda.
# Opening lock resource file /run/cryptsetup/L_8:0
# Verifying lock handle for /dev/sda.
# Device /dev/sda WRITE lock taken.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Reusing open rw fd on device /dev/sda
# Checksum:d01793e1243e3b4cce711822d6fd87f22f33dd1dcbe228ae754338882eecafa8 (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Reusing open rw fd on device /dev/sda
# Checksum:bbc34fd600612a1fb8d6d81b9b9f5aa5d636d473826ab9fea7638c058225db43 (in-memory)
# Device /dev/sda WRITE lock released.
# Adding new keyslot -1 by passphrase, volume key provided by key (-1).
# Selected keyslot 0.
# Keyslot 0 assigned to digest 0.
# Trying to allocate LUKS2 keyslot 0.
# Found area 32768 -> 290816
# Running argon2id() benchmark.
# PBKDF benchmark: memory cost = 65536, iterations = 4, threads = 1 (took 176 ms)
# PBKDF benchmark: memory cost = 93090, iterations = 4, threads = 1 (took 271 ms)
Killed

(and the OOM is in dmesg).


The OOM also happens with the new version:

    curl -k -O https://kojihub.stream.rdu2.redhat.com/kojifiles/work/tasks/7892/2777892/cryptsetup-2.7.0%7Erc0-1.el9.x86_64.rpm
    curl -k -O https://kojihub.stream.rdu2.redhat.com/kojifiles/work/tasks/7892/2777892/cryptsetup-libs-2.7.0%7Erc0-1.el9.x86_64.rpm
    rpm -U --verbose cryptsetup-*.rpm

... although the PBKDF costs are slightly different:

Enter passphrase for /dev/sda: 
Verify passphrase: 
# Checking new password using default pwquality settings.
# Crypto backend (OpenSSL 3.0.7 1 Nov 2022[default][legacy]) initialized in cryptsetup library version 2.7.0-rc0.
# Detected kernel Linux 5.14.0-362.1.1.el9_3.x86_64 x86_64.
# Only 1 active CPUs detected, PBKDF threads decreased from 4 to 1.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 372426kB.
# PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 372426, parallel_threads 1.
Existing 'crypto_LUKS' superblock signature on device /dev/sda will be wiped.
Existing 'crypto_LUKS' superblock signature on device /dev/sda will be wiped.
# Formatting device /dev/sda as type LUKS2.
# Auto-detected optimal encryption sector size for device /dev/sda is 512 bytes.
# Topology: IO (512/524288), offset = 0; Required alignment is 1048576 bytes.
# Checking if cipher aes-xts-plain64 is usable.
# Using userspace crypto wrapper to access keyslot area.
# Formatting LUKS2 with JSON metadata area 12288 bytes and keyslots area 16744448 bytes.
# Creating new digest 0 (pbkdf2).
# Setting PBKDF2 type key digest 0.
# Running pbkdf2(sha256) benchmark.
# PBKDF benchmark: memory cost = 0, iterations = 1024000, threads = 0 (took 32 ms)
# PBKDF benchmark: memory cost = 0, iterations = 1032062, threads = 0 (took 508 ms)
# Benchmark returns pbkdf2(sha256) 1032062 iterations, 0 memory, 0 threads (for 512-bits key).
# Segment 0 assigned to digest 0.
# Device size 104857600, offset 16777216.
# Wiping LUKS areas (0x000000 - 0x1000000) with zeroes.
# Wiping keyslots area (0x008000 - 0x1000000) with random data.
# Reusing open rw fd on device /dev/sda
# Device size 104857600, offset 16777216.
# Acquiring write lock for device /dev/sda.
# Opening lock resource file /run/cryptsetup/L_8:0
# Verifying lock handle for /dev/sda.
# Device /dev/sda WRITE lock taken.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Reusing open rw fd on device /dev/sda
# Checksum:96b162fdf84942b434dde709ddf8cf95e181b625c777505c71cab589dd1a40b3 (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Reusing open rw fd on device /dev/sda
# Checksum:f17a8bc028236fc275fa2cf060eef6b7abe4c8271fd1fe6da7ee11c97dffffde (in-memory)
# Device /dev/sda WRITE lock released.
# Adding new keyslot -1 by passphrase, volume key provided by key (-1).
# Selected keyslot 0.
# Keyslot 0 assigned to digest 0.
# Trying to allocate LUKS2 keyslot 0.
# Found area 32768 -> 290816
# Running argon2id() benchmark.
# PBKDF benchmark: memory cost = 65536, iterations = 4, threads = 1 (took 164 ms)
# PBKDF benchmark: memory cost = 99902, iterations = 4, threads = 1 (took 269 ms)
Killed


Note that cockpit uses udisks, which links to/uses libcryptsetup directly [1], not the CLI. That seems to have some tweakable PBKDF options, but (1) we don't supply any in Cockpit and thus rely on the defaults, and (2) this happens with the CLI as well, so I figure fixing that should also fix udisks.

[1] https://github.com/storaged-project/libblockdev/blob/master/src/plugins/crypto.c#L930

Argh, sorry - It does work on the CLI if I first install the new cryptsetup. In my attempt in comment #13 there may have been some leftover LUKS signatures from the format attempt with 2.6.0 on /dev/sda ?

Nevertheless, it still fails through udisks, even after a reboot (which is what I originally tested). Does the library have some way of enabling debug messages through environment variables?

`free -m` is originally:
               total        used        free      shared  buff/cache   available
Mem:             727         413         198           3         225         313

... and goes to 'available: 74` before it OOMs.

So I suppose with cryptsetup 2.7 there is some improvement here, and the rest of the issue is then on the libblockdev side?

Just discussed that with Vojtech and Ondrej in a gmeet. Summary:

cryptsetup CLI 2.7 is fine. It uses its internal benchmark to lower the memory cost to min. 64 MiB.

libblockdev explicitly gets the resource defaults (presumably the *static* defaults), and supplies them to the libcryptsetup format API. That of course will use the supplied limits.

The current theory is that libblockdev doesn't call/use the libcryptsetup benchmarks to adjust the default resource costs.

(In reply to Martin Pitt from comment #15)
> The current theory is that libblockdev doesn't call/use the libcryptsetup
> benchmarks to adjust the default resource costs.

Unfortunately (or fortunately) our theory is wrong, libblockdev with cryptsetup works as expected, see the debug log when running crypt_format from libblockdev (with latest cryptsetup from git):

----
[cryptsetup] Only 2 active CPUs detected, PBKDF threads decreased from 4 to 2.
[cryptsetup] Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 82202kB.
[cryptsetup] PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 82202, parallel_threads 2.

....

[cryptsetup] Running argon2id() benchmark.
[cryptsetup] Not enough physical memory detected, PBKDF max memory decreased from 82202kB to 80502kB.
[cryptsetup] PBKDF benchmark: memory cost = 65536, iterations = 4, threads = 2 (took 82 ms)
[cryptsetup] PBKDF benchmark: memory cost = 80502, iterations = 4, threads = 2 (took 100 ms)
[cryptsetup] PBKDF benchmark: memory cost = 80502, iterations = 10, threads = 2 (took 236 ms)
[cryptsetup] PBKDF benchmark: memory cost = 80502, iterations = 84, threads = 2 (took 1771 ms)
[cryptsetup] PBKDF benchmark: memory cost = 80502, iterations = 94, threads = 2 (took 2529 ms)
[cryptsetup] PBKDF benchmark: memory cost = 80502, iterations = 74, threads = 2 (took 1895 ms)
[cryptsetup] PBKDF benchmark: memory cost = 80502, iterations = 78, threads = 2 (took 1730 ms)
[cryptsetup] PBKDF benchmark: memory cost = 80502, iterations = 90, threads = 2 (took 1939 ms)
[cryptsetup] Benchmark returns argon2id() 90 iterations, 80502 memory, 2 threads (for 512-bits key).
----

so the memory is correctly lowered to ~80 MiB and the benchmark runs. So my new theory is that the difference we see between cryptsetup cmdline tool and udisks is most likely caused by udisks consuming more memory.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.