The command we run is:

  command: cryptsetup '-d' '/tmp/cryptd2a020.key' 'open' '/dev/sda3' 'cryptsda3' '--type' 'luks'

which exits with code 1 and prints this on stderr:

  command: cryptsetup: stderr:
  Cannot open keyslot (using hash sha256).

This error comes from:

https://gitlab.com/cryptsetup/cryptsetup/-/blob/73cd60b1cc236968d4a60c4ad6772c66cbeb9e1b/lib/luks1/keymanage.c#L997

Is there anything we can do to collect more information?  eg.
Would cryptsetup luksDump of the device help?

Should I be worried that this error comes from the LUKS1 code?
As far as I know nothing should be using that.

Sorry, I'm confusing this bug with
https://bugzilla.redhat.com/show_bug.cgi?id=1977228 .  The
comment above applies to the other bug.  However although the
two error messages are slightly different, they do seem as if
they could be caused by the same underlying problem.

For *this* bug, I am able to reproduce it with libguestfs-1.45.6-8.el9
and not with 1.45.6-1.el9.

Nothing signficant changed between these versions so it's a bit
mysterious what could have caused this.

The command we are running is:

commandrvf: cryptsetup -q --key-slot 0 luksFormat /dev/sda1 /tmp/luksGQ07Jp
Not compatible PBKDF2 options (using hash algorithm sha256).

Is there any extra information we could collect here from
cryptsetup to help with diagnosing this?

(In reply to Richard W.M. Jones from comment #2)

> Is there any extra information we could collect here from
> cryptsetup to help with diagnosing this?

Yes, please run the command with --debug option and post the output here. thx!

$ virt-rescue --scratch 

><rescue> echo 123456 > /tmp/key
><rescue> cryptsetup --key-slot 0 luksFormat /dev/sda /tmp/key

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Not compatible PBKDF2 options (using hash algorithm sha256).

><rescue> cryptsetup --debug --key-slot 0 luksFormat /dev/sda /tmp/key
# cryptsetup 2.3.6 processing "cryptsetup --debug --key-slot 0 luksFormat /dev/sda /tmp/key"
# Running command luksFormat.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/sda.
# Trying to open and read device /dev/sda with direct-io.
# Initialising device-mapper backend library.

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
# File descriptor passphrase entry requested.
# Crypto backend (OpenSSL 3.0.0-alpha16 6 May 2021) initialized in cryptsetup library version 2.3.6.
# Detected kernel Linux 5.12.0-1.el9.x86_64 x86_64.
# Only 1 active CPUs detected, PBKDF threads decreased from 4 to 1.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 624726kB.
# PBKDF argon2i, time_ms 2000 (iterations 0), max_memory_kb 624726, parallel_threads 1.
# Formatting device /dev/sda as type LUKS2.
# Topology: IO (512/0), offset = 0; Required alignment is 1048576 bytes.
# Checking if cipher aes-xts-plain64 is usable.
# Using userspace crypto wrapper to access keyslot area.
# Formatting LUKS2 with JSON metadata area 12288 bytes and keyslots area 16744448 bytes.
# Creating new digest 0 (pbkdf2).
# Setting PBKDF2 type key digest 0.
# Running pbkdf2(sha256) benchmark.
Not compatible PBKDF2 options (using hash algorithm sha256).
# Releasing crypt device /dev/sda context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/sda.
# Unlocking memory.
Command failed with code -1 (wrong or missing parameters).

Thanks, this is very strange - sha256 should be always available in OpenSSL.

There is a patch for OpenSSL3 support upstream (and that should be applied for RHEL9 - decision to skip Fedora and switch to OpenSSL3 alpha in RHEL9 was really not a clever decision).
https://gitlab.com/cryptsetup/cryptsetup/-/merge_requests/178

But it still does not explain this fail here... If you run "cryptsetup benchmark --pbkdf pbkdf2 --hash sha256" - does it works or it is N/A?
Does ti work for other hash? (sha1)

If you use --type luks1 - does it work?

All this seems to me like some OpenSSL incompatibility.

My immediate thoughts are:

 - What other packages apart from cryptsetup might affect this?
   Presumably openssl.  Also the kernel?
   (I have openssl-3.0.0-0.alpha16.4.el9.x86_64)

 - Could optional packages be involved (ie. Suggests/Recommends)?

 - Could an RPM %pre/%post script be required to run in order
   for sha256 to work?  Could it require a kernel module to be loaded?

 - Could it depend on the amount of memory?  The memory inside the
   appliance is restricted.

Here are the results you asked for:

$ virt-rescue --scratch

><rescue> cryptsetup benchmark --pbkdf pbkdf2 --hash sha256 
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha256        N/A

(this command silently exited with code 1)

><rescue> cryptsetup benchmark --pbkdf pbkdf2 --hash sha1   
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1          N/A

(this command also silently exited with code 1)

><rescue> echo 123456 > /tmp/key
><rescue> cryptsetup --key-slot 0 luksFormat /dev/sda /tmp/key --type luks1

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Not compatible PBKDF2 options (using hash algorithm sha256).

(also exited with code 1)

It is either OpenSSL or some change that affects benchmark (broken proces timing, seen this several time mainly for virtual machines).

If you use "--type luks1 ---pbkdf-force-iterations 1000" parameters for format, does it move forward? (This should skip benchmark completelly.)

Anyway, we need to install centos machine and reproduce this... this looks very strange to me.

A different error this time:

$ virt-rescue --scratch
><rescue> echo 123456 > /tmp/key 
><rescue> cryptsetup --debug --key-slot 0 luksFormat /dev/sda /tmp/key --type luks1 --pbkdf-force-iterations=1000 
# cryptsetup 2.3.6 processing "cryptsetup --debug --key-slot 0 luksFormat /dev/sda /tmp/key --type luks1 --pbkdf-force-iterations=1000"
# Running command luksFormat.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/sda.
# Trying to open and read device /dev/sda with direct-io.
# Initialising device-mapper backend library.

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
# File descriptor passphrase entry requested.
# Crypto backend (OpenSSL 3.0.0-alpha16 6 May 2021) initialized in cryptsetup library version 2.3.6.
# Detected kernel Linux 5.13.0-0.rc7.51.el9.x86_64 x86_64.
# PBKDF pbkdf2-sha256, time_ms 0 (iterations 1000).
# Formatting device /dev/sda as type LUKS1.
# Topology: IO (512/0), offset = 0; Required alignment is 1048576 bytes.
# Checking if cipher aes-xts-plain64 is usable.
# Using userspace crypto wrapper to access keyslot area.
# Generating LUKS header version 1 using hash sha256, aes, xts-plain64, MK 64 bytes
# Reusing PBKDF values (no benchmark flag is set).
Cannot create LUKS header: header digest failed (using hash sha256).
# Releasing crypt device /dev/sda context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/sda.
# Unlocking memory.
Command failed with code -1 (wrong or missing parameters).


Without --type luks1 it fails differently again:

><rescue> cryptsetup --debug --key-slot 0 luksFormat /dev/sda /tmp/key --pbkdf-force-iterations=1000 
# cryptsetup 2.3.6 processing "cryptsetup --debug --key-slot 0 luksFormat /dev/sda /tmp/key --pbkdf-force-iterations=1000"
# Running command luksFormat.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/sda.
# Trying to open and read device /dev/sda with direct-io.
# Initialising device-mapper backend library.

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
# File descriptor passphrase entry requested.
# Crypto backend (OpenSSL 3.0.0-alpha16 6 May 2021) initialized in cryptsetup library version 2.3.6.
# Detected kernel Linux 5.13.0-0.rc7.51.el9.x86_64 x86_64.
# Only 1 active CPUs detected, PBKDF threads decreased from 4 to 1.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 620238kB.
# PBKDF argon2i, time_ms 0 (iterations 1000), max_memory_kb 620238, parallel_threads 1.
# Formatting device /dev/sda as type LUKS2.
# Topology: IO (512/0), offset = 0; Required alignment is 1048576 bytes.
# Checking if cipher aes-xts-plain64 is usable.
# Using userspace crypto wrapper to access keyslot area.
# Formatting LUKS2 with JSON metadata area 12288 bytes and keyslots area 16744448 bytes.
# Creating new digest 0 (pbkdf2).
# Setting PBKDF2 type key digest 0.
# Releasing crypt device /dev/sda context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/sda.
# Unlocking memory.
Command failed with code -1 (wrong or missing parameters).
><rescue> echo $?
1

The command works on the same host on baremetal, so it's
likely to be something to do with the minimized appliance
environment (or virtualization).

# nbdkit -U /tmp/sock memory 1G
# modprobe nbd
# nbd-client -unix /tmp/sock /dev/nbd0
Warning: the oldstyle protocol is no longer supported.
This method now uses the newstyle protocol with a default export
Negotiation: ..size = 1024MB
Connected /dev/nbd0

# cryptsetup --key-slot 0 luksFormat /dev/nbd0 /tmp/key 

WARNING!
========
This will overwrite data on /dev/nbd0 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES

(short pause, then exit code 0)

So I tried to "cripple" openssl3 installation as musch as possible, and it still works with cryptsetup (with the same centos9 versions).
No idea what can be wrong there - it must be something really specific to the appliance installation.

Maybe try some openssl commandline test if it works outside libcryptsetup (at least "openssl help" should print supported algorithms and "openssl speed sha256" should run a benchmark),
bu you need to install openssl cli for that.

I think I see what it is.  We save the original configuration files
at build time (since they cannot be reconstructed from the host filesystem
by https://libguestfs.org/supermin.1.html).  This means that if a new
configuration file is added, then it won't be present inside the
appliance.

In this case openssl added a new configuration file (actually a symlink,
/etc/crypto-policies/back-ends/opensslcnf.config).  When this file
is missing openssl fails at start up quite badly:

$ sudo bash -c 'echo openssl > /usr/lib64/guestfs/supermin.d/zz-rwmj-openssl'
$ virt-rescue --scratch
><rescue> openssl help
FATAL: Startup failure (dev note: apps_startup()) for openssl
40B78BB6957F0000:error:80000002:system library:process_include:No such file or directory:crypto/conf/conf_def.c:803:calling stat(/etc/crypto-policies/back-ends/opensslcnf.config)
40B78BB6957F0000:error:07000075:configuration file routines:ssl_module_init:ssl command section empty:crypto/conf/conf_ssl.c:96:name=system_default, value=crypto_policy
40B78BB6957F0000:error:0700006D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:242:module=ssl_conf, value=ssl_module retcode=-1      

So I'm going to rebuild libguestfs now which will pick up the new file
and see if that solves it.

(In reply to Richard W.M. Jones from comment #11)
> In this case openssl added a new configuration file (actually a symlink,
> /etc/crypto-policies/back-ends/opensslcnf.config).  When this file
> is missing openssl fails at start up quite badly:

yes, that's it. I tried to remove openssl config but forgot this one - it is from different package (crypto-policies).

My guess above turned out to be the right one:

>  - Could an RPM %pre/%post script be required [...]

crypto-policies generates these symlinks through a complicated
%post script:

https://gitlab.com/redhat/centos-stream/rpms/crypto-policies/-/blob/7c076748f30246f7f94d6af03de8e4c95f99841c/crypto-policies.spec#L107

supermin doesn't run %post scripts.  Actually nor do a bunch
of dnf-like tools (bazel being another one, used by kubernetes).
So while this is not necessarily a bug, it seems overcomplicated.

Even with dnf this script is not going to work reliably.  It depends on
the exact state of FIPS at RPM install time.  But there are a
couple of problems with that:

(1) The FIPS state might change after install, eg. if the user enables
or disables FIPS.

(2) It won't work for restricted install environments such as
installing into a chroot, or installing in a container with a different
kernel (such as one that does not support FIPS but then moving the
binary container to one which does).

Seems to me like it would be better to do this test at run time; or
have the config files be static; and also make openssl cope better
with the configuration file being missing.

I'm prepared to add this downstream patch to libguestfs which
will work around the issue:

https://github.com/libguestfs/libguestfs/commit/18c910479cb6465bcd3954c5d07f3ab6df5c944a

Verified with package:
libguestfs-1.45.6-9.el9.x86_64

Steps:

1.On rhel9 host with RHEL-9.0.0-20210626.0 compose
$ guestfish -N disk part-disk /dev/sda mbr : luks-format /dev/sda1 0
Enter key or passphrase ("key"):  --- set a passphrase

No error like this "luks_format: Not compatible PBKDF2 options (using hash algorithm sha256)".

2.
$ virt-rescue --scratch
><rescue> echo 123456 > /tmp/key
><rescue> cryptsetup --key-slot 0 luksFormat /dev/sda /tmp/key

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
><rescue> echo $?
0
><rescue>

*** Bug 1977228 has been marked as a duplicate of this bug. ***