Be careful with scsi_debug when specifying multiple devices: unless you add the parameter "per_host_store=1", multiple devices will share the same memory space.  (This is probably not what you want; it will result in an effect similar to a SAN LUN that provides multiple paths from initiator to target: i.e.: both devices will have the same data content.)

Can you try again with the following scsi_debug modprobe statement?

$ sudo modprobe scsi_debug add_host=2 dev_size_mb=5000 sector_size=512 physblk_exp=3 per_host_store=1

I almost forgot one detail: make sure your system has enough RAM to be able to allocate memory for all of the host devices specified (in your case, 10 GiB).

I tried scsi_debug statement from your comment, and it is still reproducible.

# modprobe scsi_debug add_host=2 dev_size_mb=5000 sector_size=512 physblk_exp=3 per_host_store=1
# lsblk
NAME                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                           8:0    0   7.3T  0 disk 
sdb                           8:16   0   4.9G  0 disk 
sdc                           8:32   0   4.9G  0 disk 
nvme1n1                     259:0    0   1.7T  0 disk 
nvme0n1                     259:1    0 465.8G  0 disk 
├─nvme0n1p1                 259:2    0   600M  0 part /boot/efi
├─nvme0n1p2                 259:3    0     1G  0 part /boot
└─nvme0n1p3                 259:4    0 464.2G  0 part 
  ├─rhel_-root 253:0    0    70G  0 lvm  /
  ├─rhel_--76-swap 253:1    0   7.8G  0 lvm  [SWAP]
  └─rhel_--76-home 253:2    0 386.4G  0 lvm  /home
# cat /sys/bus/pseudo/drivers/scsi_debug/per_host_store 
1
# cat /sys/block/sdb/queue/logical_block_size 
512
# cat /sys/block/sdb/queue/physical_block_size
4096
# stratis pool create --key-desc key pool1 /dev/sdb
# stratis pool stop pool1
# stratis pool start --unlock-method keyring --name pool1
# stratis report
{
    "name_to_pool_uuid_map": {},
    "path_to_ids_map": {},
    "pools": [
        {
            "available_actions": "fully_operational",
            "blockdevs": {
                "cachedevs": [],
                "datadevs": [
                    {
                        "blksizes": "BLKSSSZGET: 4096 bytes, BLKPBSZGET: 4096 bytes",
                        "in_use": true,
                        "key_description": "key",
                        "path": "/dev/sdb",
                        "size": "10207232 sectors",
                        "uuid": "089810fc-e6e6-4f90-9c1b-2d13c09c85a9"
                    }
                ]
            },
            "filesystems": [],
            "fs_limit": 100,
            "name": "pool1",
            "uuid": "0b3bc5ff-c154-4472-ab5b-957d526aa189"
        }
    ],
    "stopped_pools": []
}
# stratis pool add-data pool1 /dev/sdc
Execution failed:
stratisd failed to perform the operation that you requested. It returned the following information via the D-Bus: ERROR: The sector sizes of the devices proposed for extending the data tier, BLKSSSZGET: 512 bytes, BLKPBSZGET: 4096 bytes, do not match the effective sector sizes of the existing data devices, BLKSSSZGET: 4096 bytes, BLKPBSZGET: 4096 bytes. 
# free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi        10Gi       3.9Gi        24Mi       1.0Gi       4.5Gi
Swap:          7.8Gi          0B       7.8Gi

OK, I've confirmed the same failure on my system.  For some reason, it seems to be that the data device in the pool changes from this:

            "blockdevs": {
                "cachedevs": [],
                "datadevs": [
                    {
                        "blksizes": "BLKSSSZGET: 512 bytes, BLKPBSZGET: 4096 bytes",
                        "in_use": true,
                        "key_description": "testkey1",
                        "path": "/dev/sdb",
                        "size": "10207232 sectors",
                        "uuid": "56a7be0b-03b9-47a5-9c41-aa5e02561554"
                    }
                ]
            },


...To this:


            "blockdevs": {
                "cachedevs": [],
                "datadevs": [
                    {
                        "blksizes": "BLKSSSZGET: 4096 bytes, BLKPBSZGET: 4096 bytes",
                        "in_use": true,
                        "key_description": "testkey1",
                        "path": "/dev/sdb",
                        "size": "10207232 sectors",
                        "uuid": "56a7be0b-03b9-47a5-9c41-aa5e02561554"
                    }
                ]
            },

...for the scsi_debug devices, which have these values from the "blockdev" command:

(physical block size:)
# blockdev --getpbsz /dev/sdb
4096

(logical sector size)
# blockdev --getss /dev/sdb
512

The same failure is also reproducible with real disks.

# cat /sys/block/sda/queue/logical_block_size 
512
# cat /sys/block/sda/queue/physical_block_size
4096
# stratis pool create --key-desc key pool1 /dev/sda
# stratis report
{
    "name_to_pool_uuid_map": {},
    "path_to_ids_map": {},
    "pools": [
        {
            "available_actions": "fully_operational",
            "blockdevs": {
                "cachedevs": [],
                "datadevs": [
                    {
                        "blksizes": "BLKSSSZGET: 512 bytes, BLKPBSZGET: 4096 bytes",
                        "in_use": true,
                        "key_description": "key",
                        "path": "/dev/sda",
                        "size": "4688398000 sectors",
                        "uuid": "4f384f2c-f199-454f-8e5e-2646ebc76033"
                    }
                ]
            },
            "filesystems": [],
            "fs_limit": 100,
            "name": "pool1",
            "uuid": "bd76ab71-8f5a-4e15-a9b4-26d2ce7a1a9d"
        }
    ],
    "stopped_pools": []
}
# stratis pool stop pool1
# stratis pool start --unlock-method keyring --name pool1
# stratis report
{
    "name_to_pool_uuid_map": {},
    "path_to_ids_map": {},
    "pools": [
        {
            "available_actions": "fully_operational",
            "blockdevs": {
                "cachedevs": [],
                "datadevs": [
                    {
                        "blksizes": "BLKSSSZGET: 4096 bytes, BLKPBSZGET: 4096 bytes",
                        "in_use": true,
                        "key_description": "key",
                        "path": "/dev/sda",
                        "size": "4688398000 sectors",
                        "uuid": "4f384f2c-f199-454f-8e5e-2646ebc76033"
                    }
                ]
            },
            "filesystems": [],
            "fs_limit": 100,
            "name": "pool1",
            "uuid": "bd76ab71-8f5a-4e15-a9b4-26d2ce7a1a9d"
        }
    ],
    "stopped_pools": []
}
# stratis pool add-data pool1 /dev/sdb
Execution failed:
stratisd failed to perform the operation that you requested. It returned the following information via the D-Bus: ERROR: The sector sizes of the devices proposed for extending the data tier, BLKSSSZGET: 512 bytes, BLKPBSZGET: 4096 bytes, do not match the effective sector sizes of the existing data devices, BLKSSSZGET: 4096 bytes, BLKPBSZGET: 4096 bytes.

Do you encrypt the device with dm-crypt? The thing is luksFormat operation selects optimal encryption sector size since it can improve performance. If device size is properly aligned and also reports physical block size = 4K it will increase encryption sector size to 4K and effectively increase also logical block size.

(In reply to Ondrej Kozina from comment #8)
> Do you encrypt the device with dm-crypt? The thing is luksFormat operation
> selects optimal encryption sector size since it can improve performance. If
> device size is properly aligned and also reports physical block size = 4K it
> will increase encryption sector size to 4K and effectively increase also
> logical block size.

Yes, an encrypted Stratis pool is done via dm-crypt.

The physical block size of a "512e" or "Advanced Format" disk is 4096 bytes, while the logical block size (the result of "blockdev --getss") is 512 bytes.  The drive's controller performs 512-byte emulation, but uses 4096-byte physical blocks.

Here's an example with only scsi_debug and cryptsetup:

# modprobe scsi_debug add_host=2 dev_size_mb=5000 sector_size=512 physblk_exp=3 per_host_store=1

# cryptsetup luksFormat /dev/sdb
...
# cryptsetup luksOpen /dev/sdb crypttest1
...

# lsblk -i /dev/sdb
NAME         MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sdb            8:16   0  4.9G  0 disk
`-crypttest1 253:0    0  4.9G  0 crypt

# blockdev --getss /dev/sdb
512
# blockdev --getpbsz /dev/sdb
4096
# blockdev --getss /dev/mapper/crypttest1
4096
# blockdev --getpbsz /dev/mapper/crypttest1
4096

Yup, so to clarify what happens. By setting encryption sector size to 4K, active device dm-crypt device increases logical block size to 4K as well, even though underlying block device exposes 512e compatibility. The issue is dm-crypt (with 4K encryption sector size enabled) can not perform IO not aligned to 4K.

You may enforce encryption sector size during luksFormat operation. On API level you have to explicitly set 'sector_size' member in 'crypt_params_luks2' structure passed in 'crypt_format'.

We expect this to be fixed in the next stratisd release, 3.5.3. See GitHub issue: https://github.com/stratis-storage/stratisd/issues/3290 .

Verified with stratisd-3.5.5-1.el9.

# modprobe scsi_debug add_host=2 dev_size_mb=5000 sector_size=512 physblk_exp=3 per_host_store=1
# stratis --propagate pool create --key-desc key test_pool /dev/sdm
# stratis pool stop test_pool
# stratis pool start --unlock-method keyring --name test_pool
# stratis pool add-data test_pool /dev/sdn
# stratis report
{
    "name_to_pool_uuid_map": {},
    "partially_constructed_pools": [],
    "path_to_ids_map": {},
    "pools": [
        {
            "available_actions": "fully_operational",
            "blockdevs": {
                "cachedevs": [],
                "datadevs": [
                    {
                        "blksizes": "base: BLKSSSZGET: 512 bytes, BLKPBSZGET: 4096 bytes, crypt: BLKSSSZGET: 4096 bytes, BLKPBSZGET: 4096 bytes",
                        "in_use": true,
                        "key_description": "key",
                        "path": "/dev/sdm",
                        "size": "10207232 sectors",
                        "uuid": "35e99c01-7ffe-437f-bc79-539c27862820"
                    },
                    {
                        "blksizes": "base: BLKSSSZGET: 512 bytes, BLKPBSZGET: 4096 bytes, crypt: BLKSSSZGET: 4096 bytes, BLKPBSZGET: 4096 bytes",
                        "in_use": true,
                        "key_description": "key",
                        "path": "/dev/sdn",
                        "size": "10207232 sectors",
                        "uuid": "b8535c9a-ad84-459e-ba83-aa0c1467b310"
                    }
                ]
            },
            "filesystems": [],
            "fs_limit": 100,
            "name": "test_pool",
            "uuid": "b7c7bb55-2ad1-4565-9b79-cce3b0f99519"
        }
    ],
    "stopped_pools": []
}

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (stratisd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6424