Bug 2193096

+++ This bug was initially created as a clone of Bug #2183437 +++

Description of problem:

In FIPS mode, encrypting partitions configured with (default) LUKS2 requires using PBKDF2 type.
All this is automatically performed by the installer and libcrypt and works fine, unless only 2GB of memory is assigned to the system.
In such case, a crash occurs because of automatic computation of max_memory_kb parameter, which is not a valid option for LUKS2/PBKDF2:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details


  File "/usr/lib64/python3.9/site-packages/pyanaconda/installation_tasks.py", line 496, in start
    self.run_task()
  File "/usr/lib64/python3.9/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()
  File "/usr/lib64/python3.9/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()
  File "/usr/lib64/python3.9/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()
  File "/usr/lib64/python3.9/site-packages/pyanaconda/installation.py", line 399, in run_installation
    queue.start()
  File "/usr/lib64/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.9/site-packages/pyanaconda/threading.py", line 275, in run
    threading.Thread.run(self)
dasbus.error.DBusError: Failed to format device: Invalid argument
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

In the syslog log, we can see:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
05:25:22,059 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:[cryptsetup] PBKDF max memory or parallel threads must not be set with pbkdf2.
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

The source code responsible for the issue is below (/usr/lib/python3.9/site-packages/blivet/formats/luks.py):
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
293     def _create(self, **kwargs):
294         log_method_call(self, device=self.device,
295                         type=self.type, status=self.status)
296         super(LUKS, self)._create(**kwargs)  # set up the event sync
297 
298         if not self.pbkdf_args and self.luks_version == "luks2":
299             if luks_data.pbkdf_args:
300                 self.pbkdf_args = luks_data.pbkdf_args
301             else:
302                 mem_limit = crypto.calculate_luks2_max_memory()
303                 if mem_limit:
304                     self.pbkdf_args = LUKS2PBKDFArgs(max_memory_kb=int(mem_limit.convert_to(KiB)))
305                     luks_data.pbkdf_args = self.pbkdf_args
306                     log.info("PBKDF arguments for LUKS2 not specified, using defaults with memory limit %s", mem_limit)
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Here above, because we don't specify arguments when creating the partition, we end up on line 302, which returns 786432 (KB), causing "luks_data.pbkdf_args" to contain the following:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
{'type': None, 'max_memory_kb': 786432, 'iterations': 0, 'time_ms': 0, 'hash_fn': None}
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Upon executing libcrypt code to encrypt the device, this fails due to invalid parameter "max_memory_kb" being specified.

When the system is configured with 3GB or more, the error doesn't happen because "mem_limit" is set to None (/usr/lib/python3.9/site-packages/blivet/devicelibs/crypto.py):
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
 46 def calculate_luks2_max_memory():
 :
 60     # upper limit from cryptsetup is max(half RAM, 1 GiB)
 61     elif free_mem >= Size("1 GiB") or free_mem >= (total_mem / 2):
 62         return None
 :
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

A workaround consists in specifying encryption parameters, to avoid the computation to occur, e.g.:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
part pv.00 --size=1 --grow --encrypted --passphrase="testpassword" --luks-version=luks2 --pbkdf=pbkdf2
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Version-Release number of selected component (if applicable):

python-blivet-3.4.0-16

How reproducible:

Always

Steps to Reproduce:
1. Configure a VM with 2GB of memory and install it with "fips=1" on kernel command line

   Example of kickstart attached for convenience, but also happens in interactive installation.

Actual results:

Crash when encrypting the partition

Expected results:

No crash and installation continues

Additional info:

Currently the LUKS code is not FIPS aware, which causes the computation of "mem_limit" to always happen, even if libcrypt will "choose" PBKDF2 in the end (due to FIPS).
IMHO blivet needs to be FIPS aware and select the PBKDF type depending on FIPS mode.