The cryptsetup library interchanges the loop device with its underlying regular file during writing the LUKS header to the regular file. The process fails if the file system is on top of a block device with 4KiB sectors.
In the following examples, the cryptsetup luksFormat command of a detached header fails if /path/to/hdr_file is a regular file in a file system residing on top of a block device with 4KiB sectors.
cryptsetup luksFormat --header /path/to/hdr_file /dev/device
cryptsetup luksFormat /path/to/hdr_file
To work around this problem, the cryptsetup-reencrypt utility needs to be started from a file system on top of a block device with 512-byte sectors. The user can back up the header using the luksHeaderBackup command after formatting the block device using the luksFormat command without a detached header.
Also, the cryptsetup-reencrypt utility fails if the user runs it from a working directory on a file system on top of a block device with 4KiB sectors. The user must start cryptsetup-reencrypt from a working directory on a file system on top of a block device with 512-byte sectors.
Ok finally I got simple reproducer for this issue and it has nothing to do with multipath:
Steps to reproduce:
1) modprobe scsi_debug sector_size=4096 dev_size_mb=50 (let's say you will get device /dev/sdd)
2) mkfs.xfs /dev/sdd (even with ext4, it doesn't matter)
3) mount /dev/sdd /mnt/test
4) fallocate -l $[8*1024*1024] /mnt/test/hdr_file
5) echo aaa | cryptsetup --force-password luksFormat /mnt/test/hdr_file
the command will fail w/ error message about being unable to update the header in file /mnt/test/hdr_file.
It's caused by internal error in cryptsetup library. Trying to write luks header to regular file, It will get block device sector size from loop device (512B) and after that, it will try to write w/ O_DIRECT flag to actual regular file (the loop device backend) instead of to the loop device itself. Naturally, the FS residing on top of 4KiB sector sized block device forbids that operation.
This mainly affects cryptsetup-reencrypt utility (more on that in comments above by Xiao) as it uses headers stored in actual working directory.
Xiao, thank you again for help!
This request was resolved in Red Hat Enterprise Linux 7.0.
Contact your manager or support representative in case you have further questions about the request.
--- Additional comment from XiaoNi on 2013-11-11 00:03:57 EST --- Hi Ondrej I repeat the test and the more information: 1) [root@storageqe-06 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk └─360a98000572d574a4e6f63706772594b 253:4 0 30G 0 mpath ├─360a98000572d574a4e6f63706772594b1 253:9 0 500M 0 part /boot └─360a98000572d574a4e6f63706772594b2 253:10 0 29.5G 0 part ├─rhel_storageqe--06-swap 253:11 0 7.9G 0 lvm [SWAP] └─rhel_storageqe--06-root 253:12 0 21.6G 0 lvm / sdb 8:16 0 5G 0 disk └─360a98000324669436c2b424f41665755 253:0 0 5G 0 mpath └─360a98000324669436c2b424f41665755p1 253:5 0 5G 0 part sdc 8:32 0 5G 0 disk └─360a98000324669436c2b424f41665753 253:1 0 5G 0 mpath └─360a98000324669436c2b424f41665753p1 253:6 0 5G 0 part └─md1 9:1 0 5G 0 raid5 └─test-lv1 253:13 0 5G 0 lvm /test sdd 8:48 0 5G 0 disk └─360a98000324669436c2b424f41665751 253:2 0 5G 0 mpath └─360a98000324669436c2b424f41665751p1 253:7 0 5G 0 part └─md1 9:1 0 5G 0 raid5 └─test-lv1 253:13 0 5G 0 lvm /test sde 8:64 0 5G 0 disk └─360a98000324669436c2b424f4166574f 253:3 0 5G 0 mpath └─360a98000324669436c2b424f4166574f1 253:8 0 5G 0 part sdf 8:80 0 30G 0 disk └─360a98000572d574a4e6f63706772594b 253:4 0 30G 0 mpath ├─360a98000572d574a4e6f63706772594b1 253:9 0 500M 0 part /boot └─360a98000572d574a4e6f63706772594b2 253:10 0 29.5G 0 part ├─rhel_storageqe--06-swap 253:11 0 7.9G 0 lvm [SWAP] └─rhel_storageqe--06-root 253:12 0 21.6G 0 lvm / sdg 8:96 0 5G 0 disk └─360a98000324669436c2b424f41665755 253:0 0 5G 0 mpath └─360a98000324669436c2b424f41665755p1 253:5 0 5G 0 part sdh 8:112 0 5G 0 disk └─360a98000324669436c2b424f41665753 253:1 0 5G 0 mpath └─360a98000324669436c2b424f41665753p1 253:6 0 5G 0 part └─md1 9:1 0 5G 0 raid5 └─test-lv1 253:13 0 5G 0 lvm /test sdi 8:128 0 5G 0 disk └─360a98000324669436c2b424f41665751 253:2 0 5G 0 mpath └─360a98000324669436c2b424f41665751p1 253:7 0 5G 0 part └─md1 9:1 0 5G 0 raid5 └─test-lv1 253:13 0 5G 0 lvm /test sdj 8:144 0 5G 0 disk └─360a98000324669436c2b424f4166574f 253:3 0 5G 0 mpath └─360a98000324669436c2b424f4166574f1 253:8 0 5G 0 part sdk 8:160 0 30G 0 disk └─360a98000572d574a4e6f63706772594b 253:4 0 30G 0 mpath ├─360a98000572d574a4e6f63706772594b1 253:9 0 500M 0 part /boot └─360a98000572d574a4e6f63706772594b2 253:10 0 29.5G 0 part ├─rhel_storageqe--06-swap 253:11 0 7.9G 0 lvm [SWAP] └─rhel_storageqe--06-root 253:12 0 21.6G 0 lvm / sdl 8:176 0 5G 0 disk └─360a98000324669436c2b424f41665755 253:0 0 5G 0 mpath └─360a98000324669436c2b424f41665755p1 253:5 0 5G 0 part sdm 8:192 0 5G 0 disk └─360a98000324669436c2b424f41665753 253:1 0 5G 0 mpath └─360a98000324669436c2b424f41665753p1 253:6 0 5G 0 part └─md1 9:1 0 5G 0 raid5 └─test-lv1 253:13 0 5G 0 lvm /test sdn 8:208 0 5G 0 disk └─360a98000324669436c2b424f41665751 253:2 0 5G 0 mpath └─360a98000324669436c2b424f41665751p1 253:7 0 5G 0 part └─md1 9:1 0 5G 0 raid5 └─test-lv1 253:13 0 5G 0 lvm /test sdo 8:224 0 5G 0 disk └─360a98000324669436c2b424f4166574f 253:3 0 5G 0 mpath └─360a98000324669436c2b424f4166574f1 253:8 0 5G 0 part sdp 8:240 0 30G 0 disk └─360a98000572d574a4e6f63706772594b 253:4 0 30G 0 mpath ├─360a98000572d574a4e6f63706772594b1 253:9 0 500M 0 part /boot └─360a98000572d574a4e6f63706772594b2 253:10 0 29.5G 0 part ├─rhel_storageqe--06-swap 253:11 0 7.9G 0 lvm [SWAP] └─rhel_storageqe--06-root 253:12 0 21.6G 0 lvm / sdq 65:0 0 5G 0 disk └─360a98000324669436c2b424f41665755 253:0 0 5G 0 mpath └─360a98000324669436c2b424f41665755p1 253:5 0 5G 0 part sdr 65:16 0 5G 0 disk └─360a98000324669436c2b424f41665753 253:1 0 5G 0 mpath └─360a98000324669436c2b424f41665753p1 253:6 0 5G 0 part └─md1 9:1 0 5G 0 raid5 └─test-lv1 253:13 0 5G 0 lvm /test sds 65:32 0 5G 0 disk └─360a98000324669436c2b424f41665751 253:2 0 5G 0 mpath └─360a98000324669436c2b424f41665751p1 253:7 0 5G 0 part └─md1 9:1 0 5G 0 raid5 └─test-lv1 253:13 0 5G 0 lvm /test sdt 65:48 0 5G 0 disk └─360a98000324669436c2b424f4166574f 253:3 0 5G 0 mpath └─360a98000324669436c2b424f4166574f1 253:8 0 5G 0 part sr0 11:0 1 1024M 0 rom 2.my test steps: 572 cryptsetup luksFormat /dev/mapper/360a98000324669436c2b424f41665755p1 573 cryptsetup luksOpen /dev/mapper/360a98000324669436c2b424f41665755p1 test 574 ls /dev/mapper/ 575 mkfs.ext4 /dev/mapper/test 576 mount /dev/mapper/test /mnt/ 577 umount /mnt/ 578 cryptsetup luksClose /dev/mapper/test 579 cryptsetup-reencrypt --debug /dev/mapper/360a98000324669436c2b424f41665755p1 3. the output of command cryptsetup-reencrypt are as follows: WARNING: this is experimental code, it can completely break your data. # cryptsetup 1.6.2 processing "cryptsetup-reencrypt --debug /dev/mapper/360a98000324669436c2b424f41665755p1" # Initialising reencryption context. # Initialising UUID. # Allocating crypt device /dev/mapper/360a98000324669436c2b424f41665755p1 context. # Trying to open and read device /dev/mapper/360a98000324669436c2b424f41665755p1. # Initialising device-mapper backend library. # Trying to load LUKS1 crypt type from device /dev/mapper/360a98000324669436c2b424f41665755p1. # Crypto backend (gcrypt 1.5.3) initialized. # Reading LUKS header of size 1024 from device /dev/mapper/360a98000324669436c2b424f41665755p1 # Key length 32, device size 10483712 sectors, header size 2050 sectors. # Releasing crypt device /dev/mapper/360a98000324669436c2b424f41665755p1 context. # Releasing device-mapper backend. # Removing headers. # Allocating crypt device (null) context. # Initialising device-mapper backend library. # Deactivating volume LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.org. # Detected kernel Linux 3.10.0-41.el7.x86_64 x86_64. # dm version OF [16384] (*1) # dm versions OF [16384] (*1) # Detected dm-crypt version 1.12.1, dm-ioctl version 4.26.0. # Device-mapper backend running with UDEV support enabled. # dm status LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.org OF [16384] (*1) Device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.org is not active. # Deactivating volume LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new. # dm status LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new OF [16384] (*1) Device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new is not active. # Releasing crypt device (null) context. # Releasing device-mapper backend. # Created LUKS reencryption log file LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.log. # Log: direction = 0 # Log: offset = 0 # Log: shift = 0 # Running reencryption. # Passhrases initialization. # Allocating crypt device /dev/mapper/360a98000324669436c2b424f41665755p1 context. # Trying to open and read device /dev/mapper/360a98000324669436c2b424f41665755p1. # Initialising device-mapper backend library. # Trying to load LUKS1 crypt type from device /dev/mapper/360a98000324669436c2b424f41665755p1. # Crypto backend (gcrypt 1.5.3) initialized. # Reading LUKS header of size 1024 from device /dev/mapper/360a98000324669436c2b424f41665755p1 # Key length 32, device size 10483712 sectors, header size 2050 sectors. # Setting ciphertext data device to /dev/mapper/360a98000324669436c2b424f41665755p1. # Trying to open and read device /dev/mapper/360a98000324669436c2b424f41665755p1. # Installing SIGINT/SIGTERM handler. # Unblocking interruption on signal. # Interactive passphrase entry requested. Enter passphrase for key slot 0: # Blocking interruption on signal. # Checking volume [keyslot 0] using passphrase. # Trying to open key slot 0 [ACTIVE_LAST]. # Reading key slot 0 area. # Calculated device size is 250 sectors (RW), offset 8. # DM-UUID is CRYPT-TEMP-temporary-cryptsetup-16162 # Udev cookie 0xd4d3ae0 (semid 524288) created # Udev cookie 0xd4d3ae0 (semid 524288) incremented to 1 # Udev cookie 0xd4d3ae0 (semid 524288) incremented to 2 # Udev cookie 0xd4d3ae0 (semid 524288) assigned to CREATE task(0) with flags DISABLE_SUBSYSTEM_RULES DISABLE_DISK_RULES DISABLE_OTHER_RULES (0xe) # dm create temporary-cryptsetup-16162 CRYPT-TEMP-temporary-cryptsetup-16162 OF [16384] (*1) # dm reload temporary-cryptsetup-16162 OFRW [16384] (*1) # dm resume temporary-cryptsetup-16162 OFRW [16384] (*1) # temporary-cryptsetup-16162: Stacking NODE_ADD (253,14) 0:6 0660 [verify_udev] # temporary-cryptsetup-16162: Stacking NODE_READ_AHEAD 256 (flags=1) # Udev cookie 0xd4d3ae0 (semid 524288) decremented to 1 # Udev cookie 0xd4d3ae0 (semid 524288) waiting for zero # Udev cookie 0xd4d3ae0 (semid 524288) destroyed # temporary-cryptsetup-16162: Processing NODE_ADD (253,14) 0:6 0660 [verify_udev] # temporary-cryptsetup-16162: Processing NODE_READ_AHEAD 256 (flags=1) # temporary-cryptsetup-16162 (253:14): read ahead is 256 # temporary-cryptsetup-16162: retaining kernel read ahead of 256 (requested 256) # Udev cookie 0xd4d0c2b (semid 557056) created # Udev cookie 0xd4d0c2b (semid 557056) incremented to 1 # Udev cookie 0xd4d0c2b (semid 557056) incremented to 2 # Udev cookie 0xd4d0c2b (semid 557056) assigned to REMOVE task(2) with flags (0x0) # dm remove temporary-cryptsetup-16162 OFT [16384] (*1) # temporary-cryptsetup-16162: Stacking NODE_DEL [verify_udev] # Udev cookie 0xd4d0c2b (semid 557056) decremented to 1 # Udev cookie 0xd4d0c2b (semid 557056) waiting for zero # Udev cookie 0xd4d0c2b (semid 557056) destroyed # temporary-cryptsetup-16162: Processing NODE_DEL [verify_udev] Key slot 0 unlocked. # Releasing crypt device /dev/mapper/360a98000324669436c2b424f41665755p1 context. # Releasing device-mapper backend. # Creating LUKS header backup for device /dev/mapper/360a98000324669436c2b424f41665755p1. # Allocating crypt device /dev/mapper/360a98000324669436c2b424f41665755p1 context. # Trying to open and read device /dev/mapper/360a98000324669436c2b424f41665755p1. # Initialising device-mapper backend library. # Trying to load LUKS1 crypt type from device /dev/mapper/360a98000324669436c2b424f41665755p1. # Crypto backend (gcrypt 1.5.3) initialized. # Reading LUKS header of size 1024 from device /dev/mapper/360a98000324669436c2b424f41665755p1 # Key length 32, device size 10483712 sectors, header size 2050 sectors. # Crypto backend (gcrypt 1.5.3) initialized. # Requested header backup of device /dev/mapper/360a98000324669436c2b424f41665755p1 (LUKS1) to file LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.org. # Reading LUKS header of size 1024 from device /dev/mapper/360a98000324669436c2b424f41665755p1 # Key length 32, device size 10483712 sectors, header size 2050 sectors. # Storing backup of header (1024 bytes) and keyslot area (1045504 bytes). LUKS header backup of device /dev/mapper/360a98000324669436c2b424f41665755p1 created. # Creating empty file LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new of size 1049600. # Allocating crypt device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new context. # Trying to open and read device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new. # Iteration time set to 1000 miliseconds. # Formatting device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new as type LUKS1. # Crypto backend (gcrypt 1.5.3) initialized. # Trying to open and read device /dev/mapper/360a98000324669436c2b424f41665755p1. # Generating LUKS header version 1 using hash sha1, aes, xts-plain64, MK 32 bytes # Crypto backend (gcrypt 1.5.3) initialized. # KDF pbkdf2, hash sha1: 277989 iterations per second. # Data offset 4096, UUID 5fb88832-dd0e-43c2-8983-8bab3f396cea, digest iterations 33875 # Updating LUKS header of size 1024 on device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new # Key length 32, device size 2050 sectors, header size 2050 sectors. # Reading LUKS header of size 1024 from device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new # Key length 32, device size 2050 sectors, header size 2050 sectors. New LUKS header for device /dev/mapper/360a98000324669436c2b424f41665755p1 created. # Adding new keyslot 0 using volume key. # Calculating data for key slot 0 # Crypto backend (gcrypt 1.5.3) initialized. # KDF pbkdf2, hash sha1: 279471 iterations per second. # Key slot 0 use 136459 password iterations. # Using hash sha1 for AF in key slot 0, 4000 stripes # Updating key slot 0 [0x1000] area. # Allocating a free loop device. # Trying to open and read device /dev/loop0. # Calculated device size is 256 sectors (RW), offset 8. # DM-UUID is CRYPT-TEMP-temporary-cryptsetup-16162 # Udev cookie 0xd4de028 (semid 589824) created # Udev cookie 0xd4de028 (semid 589824) incremented to 1 # Udev cookie 0xd4de028 (semid 589824) incremented to 2 # Udev cookie 0xd4de028 (semid 589824) assigned to CREATE task(0) with flags DISABLE_SUBSYSTEM_RULES DISABLE_DISK_RULES DISABLE_OTHER_RULES (0xe) # dm create temporary-cryptsetup-16162 CRYPT-TEMP-temporary-cryptsetup-16162 OF [16384] (*1) # dm reload temporary-cryptsetup-16162 OFW [16384] (*1) # dm resume temporary-cryptsetup-16162 OFW [16384] (*1) # temporary-cryptsetup-16162: Stacking NODE_ADD (253,14) 0:6 0660 [verify_udev] # temporary-cryptsetup-16162: Stacking NODE_READ_AHEAD 256 (flags=1) # Udev cookie 0xd4de028 (semid 589824) decremented to 1 # Udev cookie 0xd4de028 (semid 589824) waiting for zero # Udev cookie 0xd4de028 (semid 589824) destroyed # temporary-cryptsetup-16162: Processing NODE_ADD (253,14) 0:6 0660 [verify_udev] # temporary-cryptsetup-16162: Processing NODE_READ_AHEAD 256 (flags=1) # temporary-cryptsetup-16162 (253:14): read ahead is 256 # temporary-cryptsetup-16162: retaining kernel read ahead of 256 (requested 256) # Udev cookie 0xd4dc200 (semid 622592) created # Udev cookie 0xd4dc200 (semid 622592) incremented to 1 # Udev cookie 0xd4dc200 (semid 622592) incremented to 2 # Udev cookie 0xd4dc200 (semid 622592) assigned to REMOVE task(2) with flags (0x0) # dm remove temporary-cryptsetup-16162 OFT [16384] (*1) # temporary-cryptsetup-16162: Stacking NODE_DEL [verify_udev] # Udev cookie 0xd4dc200 (semid 622592) decremented to 1 # Udev cookie 0xd4dc200 (semid 622592) waiting for zero # Udev cookie 0xd4dc200 (semid 622592) destroyed # temporary-cryptsetup-16162: Processing NODE_DEL [verify_udev] # Key slot 0 was enabled in LUKS header. # Updating LUKS header of size 1024 on device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new # Key length 32, device size 2050 sectors, header size 2050 sectors. Error during update of LUKS header on device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new. # Releasing crypt device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new context. # Closed loop /dev/loop0 (LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new). # Releasing crypt device /dev/mapper/360a98000324669436c2b424f41665755p1 context. # Releasing device-mapper backend. Creation of LUKS backup headers failed. # Destroying reencryption context. # Closing LUKS reencryption log file LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.log. # Removing headers. # Allocating crypt device (null) context. # Initialising device-mapper backend library. # Deactivating volume LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.org. # dm status LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.org OF [16384] (*1) Device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.org is not active. # Deactivating volume LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new. # dm status LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new OF [16384] (*1) Device LUKS-5fb88832-dd0e-43c2-8983-8bab3f396cea.new is not active. # Releasing crypt device (null) context. # Releasing device-mapper backend 4. The filesystem of my working directory is xfs # # /etc/fstab # Created by anaconda on Thu Nov 7 03:23:17 2013 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/rhel_storageqe--06-root / xfs defaults 1 1 UUID=f96239f4-3039-4310-b7cb-adaf7f638ba1 /boot xfs defaults 1 2 /dev/mapper/rhel_storageqe--06-swap swap swap defaults 0 0 5. The RHEL7 system I provisioned is: RHEL-7.0-20131106.n.0 Server x86_64 Thanks Xiao --- Additional comment from XiaoNi on 2013-11-11 00:35:28 EST --- BTW, the version of cryptsetup, cryptsetup-reencrypt and kernel version are as follows: [root@storageqe-06 ~]# cryptsetup --version cryptsetup 1.6.2 [root@storageqe-06 ~]# cryptsetup-reencrypt --version crypt_reencrypt 1.6.2 [root@storageqe-06 ~]# uname -a Linux storageqe-06.rhts.eng.bos.redhat.com 3.10.0-41.el7.x86_64 #1 SMP Fri Nov 1 19:11:43 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux Thanks Xiao