Bug 2153437

Summary: [RHEL9] tests_lvm_pool_members_scsi_generated.yml failed with "Devices have inconsistent logical block sizes (512 and 4096)"
Product: Red Hat Enterprise Linux 9 Reporter: Zhang Yi <yizhan>
Component: rhel-system-rolesAssignee: Vojtech Trefny <vtrefny>
Status: CLOSED DUPLICATE QA Contact: CS System Management SST QE <rhel-cs-system-management-subsystem-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.2CC: japokorn, nhosoi, okozina, spetrosi
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-12 13:52:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Zhang Yi 2022-12-14 15:09:08 UTC
Description of problem:


Version-Release number of selected component (if applicable):
rhel-system-roles-1.21.0-0.10.storage.el9.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

[root@storageqe-62 rhel-system-roles.storage]# cat inventory 
localhost  ansible_connection=local

[root@storageqe-62 rhel-system-roles.storage]# ansible-playbook -i inventory tests/tests_lvm_pool_members_scsi_generated.yml
--snip--
TASK [Create volume group 'foo' with 3 encrypted PVs] ****************************************************************************************************************************************************************************************************************************

TASK [rhel-system-roles.storage : set platform/version specific variables] *******************************************************************************************************************************************************************************************************
included: /usr/share/ansible/roles/rhel-system-roles.storage/tasks/set_vars.yml for localhost

TASK [rhel-system-roles.storage : Ensure ansible_facts used by role] *************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [rhel-system-roles.storage : Set platform/version specific variables] *******************************************************************************************************************************************************************************************************
skipping: [localhost] => (item=RedHat.yml) 
skipping: [localhost] => (item=RedHat.yml) 
ok: [localhost] => (item=RedHat_9.yml)
skipping: [localhost] => (item=RedHat_9.2.yml) 

TASK [rhel-system-roles.storage : define an empty list of pools to be used in testing] *******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [rhel-system-roles.storage : define an empty list of volumes to be used in testing] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [rhel-system-roles.storage : include the appropriate provider tasks] ********************************************************************************************************************************************************************************************************
included: /usr/share/ansible/roles/rhel-system-roles.storage/tasks/main-blivet.yml for localhost

TASK [rhel-system-roles.storage : get a list of rpm packages installed on host machine] ******************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : make sure blivet is available] *****************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : show storage_pools] ****************************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : show storage_volumes] **************************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : get required packages] *************************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : enable copr repositories if needed] ************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : make sure required packages are installed] *****************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : get service facts] *****************************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : Set storage_cryptsetup_services] ***************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [rhel-system-roles.storage : Mask the systemd cryptsetup services] **********************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [rhel-system-roles.storage : manage the pools and volumes to match the specified state] *************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"actions": [], "changed": false, "crypts": [], "leaves": [], "mounts": [], "msg": "Failed to commit changes to disk: Process reported exit code 5:   Devices have inconsistent logical block sizes (512 and 4096).\n", "packages": ["xfsprogs", "cryptsetup", "lvm2", "dosfstools"], "pools": [], "volumes": []}

TASK [rhel-system-roles.storage : failed message] ********************************************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": {"actions": [], "changed": false, "crypts": [], "failed": true, "invocation": {"module_args": {"disklabel_type": null, "diskvolume_mkfs_option_map": {}, "packages_only": false, "pool_defaults": {"disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "state": "present", "type": "lvm", "volumes": []}, "pools": [{"disks": ["sda", "sdb", "sdd", "sde", "sdf", "sdg", "sdh", "sdi", "sdj", "sdk"], "encryption": true, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": "yabbadabbadoo", "name": "foo", "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "state": "present", "type": "lvm", "volumes": []}], "safe_mode": false, "use_partitions": true, "volume_defaults": {"cache_devices": [], "cache_mode": null, "cache_size": 0, "cached": false, "compression": null, "deduplication": null, "disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "fs_create_options": "", "fs_label": "", "fs_overwrite_existing": true, "fs_type": "xfs", "mount_check": 0, "mount_device_identifier": "uuid", "mount_options": "defaults", "mount_passno": 0, "mount_point": "", "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "size": 0, "state": "present", "thin": null, "thin_pool_name": null, "thin_pool_size": null, "type": "lvm", "vdo_pool_size": null}, "volumes": []}}, "leaves": [], "mounts": [], "msg": "Failed to commit changes to disk: Process reported exit code 5:   Devices have inconsistent logical block sizes (512 and 4096).\n", "packages": ["xfsprogs", "cryptsetup", "lvm2", "dosfstools"], "pools": [], "volumes": []}}

TASK [rhel-system-roles.storage : Unmask the systemd cryptsetup services] ********************************************************************************************************************************************************************************************************
skipping: [localhost]

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=339  changed=10   unreachable=0    failed=0    skipped=261  rescued=2    ignored=0   


[root@storageqe-62 rhel-system-roles.storage]# lsblk 
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                             8:0    0 279.4G  0 disk  
└─sda1                                          8:1    0 279.4G  0 part  
  └─luks-c168df38-9b69-4bf9-ac57-edbc22b088c0 253:12   0 279.4G  0 crypt 
sdb                                             8:16   0 186.3G  0 disk  
└─sdb1                                          8:17   0 186.3G  0 part  
  └─luks-fee93fd9-8d32-4694-959c-8c811c547052 253:11   0 186.3G  0 crypt 
sdc                                             8:32   0 279.4G  0 disk  
├─sdc1                                          8:33   0   600M  0 part  /boot/efi
├─sdc2                                          8:34   0     1G  0 part  /boot
└─sdc3                                          8:35   0 277.8G  0 part  
  ├─rhel_storageqe--62-root                   253:0    0    70G  0 lvm   /
  ├─rhel_storageqe--62-swap                   253:1    0   7.8G  0 lvm   [SWAP]
  └─rhel_storageqe--62-home                   253:2    0   200G  0 lvm   /home
sdd                                             8:48   0 111.8G  0 disk  
└─sdd1                                          8:49   0 111.8G  0 part  
  └─luks-41a22697-ad98-402e-a665-008e3cb8bade 253:10   0 111.8G  0 crypt 
sde                                             8:64   0 931.5G  0 disk  
└─sde1                                          8:65   0 931.5G  0 part  
  └─luks-f7cb09f3-dc8f-4fd6-9967-0385bfb9837f 253:9    0 931.5G  0 crypt 
sdf                                             8:80   0 931.5G  0 disk  
└─sdf1                                          8:81   0 931.5G  0 part  
  └─luks-75b67133-e9d1-42d1-8ba4-1d3c5925f4c5 253:8    0 931.5G  0 crypt 
sdg                                             8:96   0 931.5G  0 disk  
└─sdg1                                          8:97   0 931.5G  0 part  
  └─luks-591e8a87-783c-4b43-88c5-24ef6d7fe6f1 253:7    0 931.5G  0 crypt 
sdh                                             8:112  0 931.5G  0 disk  
└─sdh1                                          8:113  0 931.5G  0 part  
  └─luks-3204ab8d-1b63-4b8d-a70a-0aa611b77f35 253:6    0 931.5G  0 crypt 
sdi                                             8:128  0 279.4G  0 disk  
└─sdi1                                          8:129  0 279.4G  0 part  
  └─luks-b95bf578-11f7-4ef5-b7c3-8e6aed5a88da 253:5    0 279.4G  0 crypt 
sdj                                             8:144  0 931.5G  0 disk  
└─sdj1                                          8:145  0 931.5G  0 part  
  └─luks-0f82daf8-1adf-4eaa-9b09-4bbd0496a0cf 253:4    0 931.5G  0 crypt 
sdk                                             8:160  0 279.4G  0 disk  
└─sdk1                                          8:161  0 279.4G  0 part  
  └─luks-7e438c30-f55d-4c06-b078-460bfc035874 253:3    0 279.4G  0 crypt 
sdl                                             8:176  0 111.8G  0 disk  
nvme0n1                                       259:0    0 894.3G  0 disk  

[root@storageqe-62 rhel-system-roles.storage]# tail /sys/block/sd*/queue/logical_block_size 
==> /sys/block/sda/queue/logical_block_size <==
512

==> /sys/block/sdb/queue/logical_block_size <==
512

==> /sys/block/sdc/queue/logical_block_size <==
512

==> /sys/block/sdd/queue/logical_block_size <==
512

==> /sys/block/sde/queue/logical_block_size <==
512

==> /sys/block/sdf/queue/logical_block_size <==
512

==> /sys/block/sdg/queue/logical_block_size <==
512

==> /sys/block/sdh/queue/logical_block_size <==
512

==> /sys/block/sdi/queue/logical_block_size <==
512

==> /sys/block/sdj/queue/logical_block_size <==
512

==> /sys/block/sdk/queue/logical_block_size <==
512

==> /sys/block/sdl/queue/logical_block_size <==
512
[root@storageqe-62 rhel-system-roles.storage]# tail /sys/block/dm-*/queue/logical_block_size 
==> /sys/block/dm-0/queue/logical_block_size <==
512

==> /sys/block/dm-10/queue/logical_block_size <==
512

==> /sys/block/dm-11/queue/logical_block_size <==
512

==> /sys/block/dm-12/queue/logical_block_size <==
512

==> /sys/block/dm-1/queue/logical_block_size <==
512

==> /sys/block/dm-2/queue/logical_block_size <==
512

==> /sys/block/dm-3/queue/logical_block_size <==
512

==> /sys/block/dm-4/queue/logical_block_size <==
4096

==> /sys/block/dm-5/queue/logical_block_size <==
512

==> /sys/block/dm-6/queue/logical_block_size <==
4096

==> /sys/block/dm-7/queue/logical_block_size <==
4096

==> /sys/block/dm-8/queue/logical_block_size <==
4096

==> /sys/block/dm-9/queue/logical_block_size <==
4096

Comment 1 Rich Megginson 2022-12-14 15:33:50 UTC
Vojtech, any idea?

Comment 2 Vojtech Trefny 2022-12-15 06:52:47 UTC
We let cryptsetup decide what sector size is the best when creating a LUKS device. Cryptsetup uses physical sector size so I guess the disks where cryptsetup decided to create a 4096 sector size LUKS format have 4096 physical sector size. I'll discuss this with the cryptsetup developers, maybe we should change the behaviour in Blivet, or cryptsetup should behave differently on disks with different logical and physical block sizes.

But there are two issues related to this:
1. The blivet issue is tracked separately in https://bugzilla.redhat.com/show_bug.cgi?id=2103800 We might need to change how we call cryptsetup, but at least Blivet should fail sooner with a better error message, 
2. The storage role tests should be able to select disks with same block size for the test run if possible to avoid this problem. LVM doesn't allow mixing different sector sizes in a single VG so the test suite shouldn't try to do that.

Comment 3 Vojtech Trefny 2022-12-15 09:38:06 UTC
Zhang Yi: Can you please add here also /sys/block/<disk>/queue/physical_block_size for the disks? cryptsetup uses physical block size when formatting new LUKS. Also please attach blivet logs from the test run (should be saved to /tmp/blivet.log) if you still have access to the machine, thank you.

Comment 4 Zhang Yi 2022-12-16 14:34:42 UTC
Hi Vojtech

Here is the info:

[root@storageqe-62 rhel-system-roles.storage]#  ansible-playbook -i inventory tests/tests_lvm_pool_members_scsi_generated.yml
--snip--
TASK [rhel-system-roles.storage : manage the pools and volumes to match the specified state] ********************************************************************************************************
fatal: [localhost]: FAILED! => {"actions": [], "changed": false, "crypts": [], "leaves": [], "mounts": [], "msg": "Failed to commit changes to disk: Process reported exit code 5:   Devices have inconsistent logical block sizes (512 and 4096).\n", "packages": ["cryptsetup", "xfsprogs", "lvm2", "dosfstools"], "pools": [], "volumes": []}

TASK [rhel-system-roles.storage : failed message] ***************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": {"actions": [], "changed": false, "crypts": [], "failed": true, "invocation": {"module_args": {"disklabel_type": null, "diskvolume_mkfs_option_map": {}, "packages_only": false, "pool_defaults": {"disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "state": "present", "type": "lvm", "volumes": []}, "pools": [{"disks": ["sdb", "sdc", "sdd", "sde", "sdf", "sdg", "sdh", "sdi", "sdj", "sdk"], "encryption": true, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": "yabbadabbadoo", "name": "foo", "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "state": "present", "type": "lvm", "volumes": []}], "safe_mode": false, "use_partitions": true, "volume_defaults": {"cache_devices": [], "cache_mode": null, "cache_size": 0, "cached": false, "compression": null, "deduplication": null, "disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "fs_create_options": "", "fs_label": "", "fs_overwrite_existing": true, "fs_type": "xfs", "mount_check": 0, "mount_device_identifier": "uuid", "mount_options": "defaults", "mount_passno": 0, "mount_point": "", "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "size": 0, "state": "present", "thin": null, "thin_pool_name": null, "thin_pool_size": null, "type": "lvm", "vdo_pool_size": null}, "volumes": []}}, "leaves": [], "mounts": [], "msg": "Failed to commit changes to disk: Process reported exit code 5:   Devices have inconsistent logical block sizes (512 and 4096).\n", "packages": ["cryptsetup", "xfsprogs", "lvm2", "dosfstools"], "pools": [], "volumes": []}}

TASK [rhel-system-roles.storage : Unmask the systemd cryptsetup services] ***************************************************************************************************************************
skipping: [localhost]

PLAY RECAP ******************************************************************************************************************************************************************************************
localhost                  : ok=339  changed=9    unreachable=0    failed=0    skipped=261  rescued=2    ignored=0   

[root@storageqe-62 rhel-system-roles.storage]# lsblk 
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                             8:0    0 279.4G  0 disk  
├─sda1                                          8:1    0   600M  0 part  /boot/efi
├─sda2                                          8:2    0     1G  0 part  /boot
└─sda3                                          8:3    0 277.8G  0 part  
  ├─rhel_storageqe--62-root                   253:0    0    70G  0 lvm   /
  ├─rhel_storageqe--62-swap                   253:1    0   7.8G  0 lvm   [SWAP]
  └─rhel_storageqe--62-home                   253:2    0   200G  0 lvm   /home
sdb                                             8:16   0 279.4G  0 disk  
└─sdb1                                          8:17   0 279.4G  0 part  
  └─luks-5aef8b6a-1b9f-4fc9-9b49-0a8f18d9966f 253:12   0 279.4G  0 crypt 
sdc                                             8:32   0 186.3G  0 disk  
└─sdc1                                          8:33   0 186.3G  0 part  
  └─luks-d85ee03b-b45a-4e13-97d4-aa66b31ed574 253:11   0 186.3G  0 crypt 
sdd                                             8:48   0 111.8G  0 disk  
└─sdd1                                          8:49   0 111.8G  0 part  
  └─luks-081937fb-172e-4eda-9eb2-ee0675e51eb9 253:10   0 111.8G  0 crypt 
sde                                             8:64   0 111.8G  0 disk  
└─sde1                                          8:65   0 111.8G  0 part  
  └─luks-8b461989-7241-4fc3-9aac-b2df09293a3b 253:9    0 111.8G  0 crypt 
sdf                                             8:80   0 931.5G  0 disk  
└─sdf1                                          8:81   0 931.5G  0 part  
  └─luks-32c73517-eabb-4ac3-a502-06cc4bd1f814 253:8    0 931.5G  0 crypt 
sdg                                             8:96   0 931.5G  0 disk  
└─sdg1                                          8:97   0 931.5G  0 part  
  └─luks-2c108b0c-21b0-41e3-bdde-e9460ca225bf 253:7    0 931.5G  0 crypt 
sdh                                             8:112  0 931.5G  0 disk  
└─sdh1                                          8:113  0 931.5G  0 part  
  └─luks-985e9dfd-dfdb-46b3-8feb-758aa9842d88 253:6    0 931.5G  0 crypt 
sdi                                             8:128  0 931.5G  0 disk  
└─sdi1                                          8:129  0 931.5G  0 part  
  └─luks-5a9f9258-46c0-4b5b-b751-bdeacfbaf63b 253:5    0 931.5G  0 crypt 
sdj                                             8:144  0 279.4G  0 disk  
└─sdj1                                          8:145  0 279.4G  0 part  
  └─luks-78e87830-c31a-4045-bec3-f7e96aa37dcd 253:4    0 279.4G  0 crypt 
sdk                                             8:160  0 279.4G  0 disk  
└─sdk1                                          8:161  0 279.4G  0 part  
  └─luks-3b3a4fd3-0185-4041-a47a-bb52dfc25d40 253:3    0 279.4G  0 crypt 
sdl                                             8:176  0 931.5G  0 disk  
nvme0n1 

[root@storageqe-62 rhel-system-roles.storage]# tail  /sys/block/sd*/queue/physical_block_size 
==> /sys/block/sda/queue/physical_block_size <==
512

==> /sys/block/sdb/queue/physical_block_size <==
512

==> /sys/block/sdc/queue/physical_block_size <==
512

==> /sys/block/sdd/queue/physical_block_size <==
512

==> /sys/block/sde/queue/physical_block_size <==
512

==> /sys/block/sdf/queue/physical_block_size <==
4096

==> /sys/block/sdg/queue/physical_block_size <==
4096

==> /sys/block/sdh/queue/physical_block_size <==
4096

==> /sys/block/sdi/queue/physical_block_size <==
4096

==> /sys/block/sdj/queue/physical_block_size <==
512

==> /sys/block/sdk/queue/physical_block_size <==
512

==> /sys/block/sdl/queue/physical_block_size <==
4096

Comment 6 Vojtech Trefny 2023-01-12 13:52:57 UTC
We don't need a fix in the storage role after all, this will be fixed in blivet only so I am marking this as a duplicate of 2103800

*** This bug has been marked as a duplicate of bug 2103800 ***