Bug 2109402

Summary: [RHEL9] cryptsetup reencrypt failed to read hotzone area starting at x
Product: Red Hat Enterprise Linux 9 Reporter: guazhang <guazhang>
Component: cryptsetupAssignee: Ondrej Kozina <okozina>
Status: CLOSED NOTABUG QA Contact: guazhang <guazhang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.1CC: agk, jbrassow, okozina, prajnoha
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-23 09:42:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description guazhang@redhat.com 2022-07-21 07:15:30 UTC 
Description of problem:
cryptsetup reencrypt random hit the failed, please have a look if hit a bug

Version-Release number of selected component (if applicable):
RHEL-9.1.0-20220718.0 BaseOS x86_64
cryptsetup-2.4.3-4.el9.x86_64


How reproducible:
random

don't reproduce it by manual, just found the failure in beaker job



...snip test log ../

INFO: [2022-07-20 07:34:48] Running: 'echo passwdpasswd | cryptsetup --header '/tmp/testheadersmh'  reencrypt /dev/nvme0n1   -q '...

INFO: [2022-07-20 07:36:15] Running: 'cryptsetup luksDump   --header /tmp/testheadersmh /dev/nvme0n1'...
LUKS header information
Version:       	2
Epoch:         	47
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	9737da31-4130-4a76-98c2-2e777520e903
Label:         	(no label)
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 0 [bytes]
	length: (whole device)
	cipher: aes-xts-plain64
	sector: 512 [bytes]

Keyslots:
  0: luks2
	Key:        512 bits
	Priority:   normal
	Cipher:     aes-xts-plain64
	Cipher key: 512 bits
	PBKDF:      argon2id
	Time cost:  6
	Memory:     1048576
	Threads:    4
	Salt:       76 a1 c1 f0 f2 84 f3 bd 04 74 2f c2 f4 45 66 c7 
	            7a 99 e0 56 f4 04 5c 0d 40 e4 73 8c d0 82 d6 e4 
	AF stripes: 4000
	AF hash:    sha256
	Area offset:32768 [bytes]
	Area length:258048 [bytes]
	Digest ID:  0
Tokens:
Digests:
  0: pbkdf2
	Hash:       sha256
	Iterations: 71389
	Salt:       b7 49 ed cd a6 6d fa 8e 1e 43 bf 05 41 bb dc 75 
	            7d cb b2 d6 e6 b9 42 72 ec ee 4a 1b 35 ee b0 0f 
	Digest:     b7 19 98 29 46 30 ad f1 07 62 a8 fa f4 34 08 8d 
	            02 cf c5 8d a3 c4 4f 8e 99 72 36 3e 46 45 00 07
INFO: [2022-07-20 07:36:15] Running: 'echo passwdpasswd | cryptsetup --type 'luks2' --header '/tmp/testheadersmh'  open /dev/nvme0n1 FGvr  -q  --allow-discards '...

INFO: [2022-07-20 07:36:23] Running: 'lsblk'...
NAME                        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                         7:0    0     5G  0 loop  
loop1                         7:1    0     5G  0 loop  
loop2                         7:2    0     5G  0 loop  
sda                           8:0    0 278.9G  0 disk  
sda1                        8:1    0     1G  0 part  /boot
sda2                        8:2    0 277.9G  0 part  
  rhel_storageqe--60-root 253:0    0    70G  0 lvm   /
  rhel_storageqe--60-swap 253:1    0  15.7G  0 lvm   [SWAP]
  rhel_storageqe--60-home 253:2    0 192.2G  0 lvm   /home
pmem0                       259:0    0     4G  0 disk  
nvme0n1                     259:2    0     5G  0 disk  
FGvr                      253:3    0     5G  0 crypt
INFO: [2022-07-20 07:36:23] Running: 'cryptsetup luksDump   --header /tmp/testheadersmh /dev/nvme0n1'...
LUKS header information
Version:       	2
Epoch:         	47
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	9737da31-4130-4a76-98c2-2e777520e903
Label:         	(no label)
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 0 [bytes]
	length: (whole device)
	cipher: aes-xts-plain64
	sector: 512 [bytes]

Keyslots:
  0: luks2
	Key:        512 bits
	Priority:   normal
	Cipher:     aes-xts-plain64
	Cipher key: 512 bits
	PBKDF:      argon2id
	Time cost:  6
	Memory:     1048576
	Threads:    4
	Salt:       76 a1 c1 f0 f2 84 f3 bd 04 74 2f c2 f4 45 66 c7 
	            7a 99 e0 56 f4 04 5c 0d 40 e4 73 8c d0 82 d6 e4 
	AF stripes: 4000
	AF hash:    sha256
	Area offset:32768 [bytes]
	Area length:258048 [bytes]
	Digest ID:  0
Tokens:
Digests:
  0: pbkdf2
	Hash:       sha256
	Iterations: 71389
	Salt:       b7 49 ed cd a6 6d fa 8e 1e 43 bf 05 41 bb dc 75 
	            7d cb b2 d6 e6 b9 42 72 ec ee 4a 1b 35 ee b0 0f 
	Digest:     b7 19 98 29 46 30 ad f1 07 62 a8 fa f4 34 08 8d 
	            02 cf c5 8d a3 c4 4f 8e 99 72 36 3e 46 45 00 07
INFO: [2022-07-20 07:36:23] Running: 'lsblk'...
NAME                        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                         7:0    0     5G  0 loop  
loop1                         7:1    0     5G  0 loop  
loop2                         7:2    0     5G  0 loop  
sda                           8:0    0 278.9G  0 disk  
sda1                        8:1    0     1G  0 part  /boot
sda2                        8:2    0 277.9G  0 part  
  rhel_storageqe--60-root 253:0    0    70G  0 lvm   /
  rhel_storageqe--60-swap 253:1    0  15.7G  0 lvm   [SWAP]
  rhel_storageqe--60-home 253:2    0 192.2G  0 lvm   /home
pmem0                       259:0    0     4G  0 disk  
nvme0n1                     259:2    0     5G  0 disk  
FGvr                      253:3    0     5G  0 crypt /tmp/libblockdev.rdwu_5scmount_test
INFO: [2022-07-20 07:36:23] FIO Running: 'fio --size='1g' --runtime='60' --filename='/tmp/libblockdev.rdwu_5scmount_test/test.dd' --rw='randrw' --name='fio_test' --direct='1' --iodepth='1' --numjobs='1' --bs='4k' '...
INFO: FIO executed successfully
[2022-07-20 07:36:41] [1;32m INFO: get the mount /tmp/libblockdev.rdwu_5scmount_test [0m
INFO: [2022-07-20 07:37:04] Running: 'lsblk'...
NAME                        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                         7:0    0     5G  0 loop  
loop1                         7:1    0     5G  0 loop  
loop2                         7:2    0     5G  0 loop  
sda                           8:0    0 278.9G  0 disk  
sda1                        8:1    0     1G  0 part  /boot
sda2                        8:2    0 277.9G  0 part  
  rhel_storageqe--60-root 253:0    0    70G  0 lvm   /
  rhel_storageqe--60-swap 253:1    0  15.7G  0 lvm   [SWAP]
  rhel_storageqe--60-home 253:2    0 192.2G  0 lvm   /home
pmem0                       259:0    0     4G  0 disk  
nvme0n1                     259:2    0     5G  0 disk  
FGvr                      253:3    0     5G  0 crypt
INFO: [2022-07-20 07:37:04] Running: 'cryptsetup luksDump   --header /tmp/testheadersmh /dev/nvme0n1'...
LUKS header information
Version:       	2
Epoch:         	47
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	9737da31-4130-4a76-98c2-2e777520e903
Label:         	(no label)
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 0 [bytes]
	length: (whole device)
	cipher: aes-xts-plain64
	sector: 512 [bytes]

Keyslots:
  0: luks2
	Key:        512 bits
	Priority:   normal
	Cipher:     aes-xts-plain64
	Cipher key: 512 bits
	PBKDF:      argon2id
	Time cost:  6
	Memory:     1048576
	Threads:    4
	Salt:       76 a1 c1 f0 f2 84 f3 bd 04 74 2f c2 f4 45 66 c7 
	            7a 99 e0 56 f4 04 5c 0d 40 e4 73 8c d0 82 d6 e4 
	AF stripes: 4000
	AF hash:    sha256
	Area offset:32768 [bytes]
	Area length:258048 [bytes]
	Digest ID:  0
Tokens:
Digests:
  0: pbkdf2
	Hash:       sha256
	Iterations: 71389
	Salt:       b7 49 ed cd a6 6d fa 8e 1e 43 bf 05 41 bb dc 75 
	            7d cb b2 d6 e6 b9 42 72 ec ee 4a 1b 35 ee b0 0f 
	Digest:     b7 19 98 29 46 30 ad f1 07 62 a8 fa f4 34 08 8d 
	            02 cf c5 8d a3 c4 4f 8e 99 72 36 3e 46 45 00 07
INFO: [2022-07-20 07:37:04] Running: 'echo passwdpasswd | cryptsetup --active-name 'FGvr' --resilience 'none' --hotzone-size '100M' --cipher 'serpent-cbc-essiv:sha256' --header '/tmp/testheadersmh'  reencrypt /dev/nvme0n1   -q '...
Failed to read hotzone area starting at 5242880000.
INFO: [2022-07-20 07:37:46] Running: 'echo passwdpasswd | cryptsetup --active-name 'FGvr' --resilience 'none' --hotzone-size '100M' --cipher 'serpent-cbc-essiv:sha256' --header '/tmp/testheadersmh'  reencrypt /dev/nvme0n1   -q  --debug '...
# cryptsetup 2.4.3 processing "cryptsetup --active-name FGvr --resilience none --hotzone-size 100M --cipher serpent-cbc-essiv:sha256 --header /tmp/testheadersmh reencrypt /dev/nvme0n1 -q --debug"
# Running command reencrypt.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating crypt device context by device FGvr.
# Initialising device-mapper backend library.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.46.0.
# Detected dm-crypt version 1.24.0.
# Detected dm-integrity version 1.10.0.
# Device-mapper backend running with UDEV support enabled.
# dm status FGvr  [ opencount noflush ]   [16384] (*1)
# Releasing device-mapper backend.
# Trying to open and read device /dev/mapper/FGvr-overlay with direct-io.
# Trying to open device /dev/mapper/FGvr-overlay without direct-io.
# Allocating context for crypt device /tmp/testheadersmh.
# Trying to open and read device /tmp/testheadersmh with direct-io.
# Initialising device-mapper backend library.
# Setting ciphertext data device to /dev/mapper/FGvr-overlay.
# Trying to open and read device /dev/mapper/FGvr-overlay with direct-io.
# Trying to open device /dev/mapper/FGvr-overlay without direct-io.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table FGvr  [ opencount flush securedata ]   [16384] (*1)
# dm status   (253:6) [ opencount noflush ]   [16384] (*1)
# Trying to open and read device /dev/mapper/FGvr-overlay with direct-io.
# Trying to open device /dev/mapper/FGvr-overlay without direct-io.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm deps FGvr  [ opencount flush ]   [16384] (*1)
# dm table FGvr-overlay  [ opencount flush securedata ]   [16384] (*1)
# dm deps FGvr-overlay  [ opencount flush ]   [16384] (*1)
# dm table FGvr-hotzone-forward  [ opencount flush securedata ]   [16384] (*1)
# dm deps FGvr-hotzone-forward  [ opencount flush ]   [16384] (*1)
# Crypto backend (OpenSSL 3.0.1 14 Dec 2021 [default][legacy]) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.14.0-130.el9.x86_64 x86_64.
# Reloading LUKS2 header (repair disabled).
# Acquiring read lock for device /tmp/testheadersmh.
# Verifying lock handle for /tmp/testheadersmh.
# Device /tmp/testheadersmh READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /tmp/testheadersmh
# Verifying locked device handle (regular file)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:3a1d170d9979089b3a28a4a8a06d226c6cade480f38e624e16046c8d002148e9 (on-disk)
# Checksum:3a1d170d9979089b3a28a4a8a06d226c6cade480f38e624e16046c8d002148e9 (in-memory)
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /tmp/testheadersmh
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:cccbdeabdd716f47d4bb18462d8cc9222f290763ab86a37d8b236dabdffc8477 (on-disk)
# Checksum:cccbdeabdd716f47d4bb18462d8cc9222f290763ab86a37d8b236dabdffc8477 (in-memory)
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Device size 16777216, offset 16777216.
# Device /tmp/testheadersmh READ lock released.
# PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# STDIN descriptor passphrase entry requested.
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Acquiring write lock for device /tmp/testheadersmh.
# Verifying lock handle for /tmp/testheadersmh.
# Device /tmp/testheadersmh WRITE lock taken.
# Checking context sequence id matches value stored on disk.
# Reusing open ro fd on device /tmp/testheadersmh
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 reencryption already initialized.
# Device /tmp/testheadersmh WRITE lock released.
# Loading LUKS2 reencryption context.
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Acquiring nonblocking write lock for resource LUKS2-reencryption-9737da31-4130-4a76-98c2-2e777520e903.
# Opening lock resource file /run/cryptsetup/LN_LUKS2-reencryption-9737da31-4130-4a76-98c2-2e777520e903
# Verifying lock handle for LUKS2-reencryption-9737da31-4130-4a76-98c2-2e777520e903.
# WRITE lock for resource LUKS2-reencryption-9737da31-4130-4a76-98c2-2e777520e903 taken.
# Trying to load LUKS2 crypt type from device /tmp/testheadersmh.
# Reloading LUKS2 header (repair disabled).
# Acquiring read lock for device /tmp/testheadersmh.
# Verifying lock handle for /tmp/testheadersmh.
# Device /tmp/testheadersmh READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Reusing open ro fd on device /tmp/testheadersmh
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:3a1d170d9979089b3a28a4a8a06d226c6cade480f38e624e16046c8d002148e9 (on-disk)
# Checksum:3a1d170d9979089b3a28a4a8a06d226c6cade480f38e624e16046c8d002148e9 (in-memory)
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /tmp/testheadersmh
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:cccbdeabdd716f47d4bb18462d8cc9222f290763ab86a37d8b236dabdffc8477 (on-disk)
# Checksum:cccbdeabdd716f47d4bb18462d8cc9222f290763ab86a37d8b236dabdffc8477 (in-memory)
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Device size 16777216, offset 16777216.
# Device /tmp/testheadersmh READ lock released.
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Device size 5368709120, offset 0.
# Required minimal device size: 0 (0 sectors), real device size: 5368709120 (10485760 sectors) calculated device size: 5368709120 (10485760 sectors)
# Keys are not ready. Unlocking all volume keys.
# Trying to unlock volume key (digest: 0) using keyslot -1.
# Keyslot 0 priority 1 != 2 (required), skipped.
# Keyslot 1 priority 1 != 2 (required), skipped.
# Keyslot 2 priority 0 != 2 (required), skipped.
# Trying to open LUKS2 keyslot 0.
# Running keyslot key derivation.
# Reading keyslot area [0x8000].
# Acquiring read lock for device /tmp/testheadersmh.
# Verifying lock handle for /tmp/testheadersmh.
# Device /tmp/testheadersmh READ lock taken.
# Reusing open ro fd on device /tmp/testheadersmh
# Device /tmp/testheadersmh READ lock released.
# Verifying key from keyslot 0, digest 0.
# Trying to unlock volume key (digest: 1) using keyslot -1.
# Keyslot 0 priority 1 != 2 (required), skipped.
# Keyslot 1 priority 1 != 2 (required), skipped.
# Keyslot 2 priority 0 != 2 (required), skipped.
# Keyslot 0 unusable for digest 1.
# Trying to open LUKS2 keyslot 1.
# Running keyslot key derivation.
# Reading keyslot area [0x47000].
# Acquiring read lock for device /tmp/testheadersmh.
# Verifying lock handle for /tmp/testheadersmh.
# Device /tmp/testheadersmh READ lock taken.
# Reusing open ro fd on device /tmp/testheadersmh
# Device /tmp/testheadersmh READ lock released.
# Verifying key from keyslot 1, digest 1.
# Loading key (32 bytes, type logon) in thread keyring.
# Loading key (64 bytes, type logon) in thread keyring.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table FGvr  [ opencount flush securedata ]   [16384] (*1)
# dm status   (253:6) [ opencount noflush ]   [16384] (*1)
# Trying to open and read device /dev/mapper/FGvr-overlay with direct-io.
# Trying to open device /dev/mapper/FGvr-overlay without direct-io.
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Calculated device size is 10485760 sectors (RW), offset 0.
# segment type mismatch.
# Unlocking WRITE lock for resource LUKS2-reencryption-9737da31-4130-4a76-98c2-2e777520e903.
# Requesting keyring logon key for revoke and unlink.
# Requesting keyring logon key for revoke and unlink.
# Releasing crypt device /tmp/testheadersmh context.
# Releasing device-mapper backend.
# Closing read only fd for /tmp/testheadersmh.
Command failed with code -1 (wrong or missing parameters).
Mismatching parameters on device FGvr.



https://beaker.engineering.redhat.com/recipes/12315408#task147615374
https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2022/07/68342/6834210/12315408/147615374/taskout.log


https://beaker.engineering.redhat.com/recipes/12315406#task147615351