Bug 1847180 - mount fails after adding integrity to active LV
Summary: mount fails after adding integrity to active LV
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.3
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: 8.0
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-15 20:25 UTC by David Teigland
Modified: 2022-12-09 13:19 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-26 17:58:45 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description David Teigland 2020-06-15 20:25:55 UTC
Description of problem:

This happens when the original LV has logical block size 512 and physical block size 4096 (because it's created on devs with LBS 512 and PBS 4096), and then integrity is added to the active LV (doesn't happen if integrity is added while the LV is inactive).  LVM selects an integrity block size of 4096 to match the xfs sector size and physical block size.  This causes subsequent mount to fail with kernel errors:

[14442.753210] XFS: xfs_readsb sector_size 512 BTOBB 1
[14442.758096] device-mapper: integrity: Bio not aligned on 8 sectors: 0x0, 0x1
[14442.765135] md/raid1:mdX: dm-4: rescheduling sector 0
[14442.770201] device-mapper: integrity: Bio not aligned on 8 sectors: 0x0, 0x1
[14442.777245] device-mapper: integrity: Bio not aligned on 8 sectors: 0x0, 0x1
[14442.784286] md/raid1:mdX: Disk failure on dm-4, disabling device.
               md/raid1:mdX: Operation continuing on 1 devices.
[14442.796008] md/raid1:mdX: redirecting sector 0 to other mirror: dm-6
[14442.802381] device-mapper: integrity: Bio not aligned on 8 sectors: 0x0, 0x1
[14442.809457] XFS (dm-7): SB validate failed with error -5 sector_size 512.

mount won't succeed until the LBS reported by the device is updated to 4096.  lvchange --refresh doesn't change that, but deactivating the activating the LV does.

Manually selecting an integrity block size of 512 (overriding lvm's select of 4096) avoids the problem.

I will probably change LVM's default block size selection to work around this issue until it can be resolved.


[root@null-04 ~]# lvcreate --type raid1 -n lv1 -L1G test
  Logical volume "lv1" created.
[root@null-04 ~]# blockdev --getss --getpbsz /dev/test/lv1
512
4096
[root@null-04 ~]# mkfs.xfs -f /dev/test/lv1
meta-data=/dev/test/lv1          isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@null-04 ~]# mount /dev/test/lv1 /mnt
[root@null-04 ~]# echo "hi" > /mnt/hi
[root@null-04 ~]# umount /mnt
[root@null-04 ~]# blockdev --getss --getpbsz /dev/test/lv1
512
4096
[root@null-04 ~]# lvconvert --raidintegrity y test/lv1
  Creating integrity metadata LV lv1_rimage_0_imeta with size 12.00 MiB.
  Logical volume "lv1_rimage_0_imeta" created.
  Creating integrity metadata LV lv1_rimage_1_imeta with size 12.00 MiB.
  Logical volume "lv1_rimage_1_imeta" created.
  Using integrity block size 4096 for file system block size 4096.
  Logical volume test/lv1 has added integrity.
[root@null-04 ~]# blockdev --getss --getpbsz /dev/test/lv1
512
4096
[root@null-04 ~]# mount /dev/test/lv1 /mnt
mount: /mnt: can't read superblock on /dev/mapper/test-lv1.
[root@null-04 ~]# blockdev --getss --getpbsz /dev/test/lv1
512
4096
[root@null-04 ~]# lvchange -an test/lv1
[root@null-04 ~]# lvchange -ay test/lv1
[root@null-04 ~]# blockdev --getss --getpbsz /dev/test/lv1
4096
4096
[root@null-04 ~]# mount /dev/test/lv1 /mnt
[root@null-04 ~]# cat /mnt/hi
hi
[root@null-04 ~]# umount /mnt


Using --raidintegrityblocksize 512 avoids the problem:

[root@null-04 ~]# lvcreate --type raid1 -n lv1 -L1G test
  Logical volume "lv1" created.
[root@null-04 ~]# blockdev --getss --getpbsz /dev/test/lv1
512
4096
[root@null-04 ~]# mkfs.xfs -f /dev/test/lv1
meta-data=/dev/test/lv1          isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@null-04 ~]# mount /dev/test/lv1 /mnt
[root@null-04 ~]# echo "hi" > /mnt/hi
[root@null-04 ~]# blockdev --getss --getpbsz /dev/test/lv1
512
4096
[root@null-04 ~]# umount /mnt
[root@null-04 ~]# lvconvert --raidintegrity y --raidintegrityblocksize 512 test/lv1
  Creating integrity metadata LV lv1_rimage_0_imeta with size 12.00 MiB.
  Logical volume "lv1_rimage_0_imeta" created.
  Creating integrity metadata LV lv1_rimage_1_imeta with size 12.00 MiB.
  Logical volume "lv1_rimage_1_imeta" created.
  Using integrity block size 512 for file system block size 4096.
  Logical volume test/lv1 has added integrity.
[root@null-04 ~]# blockdev --getss --getpbsz /dev/test/lv1
512
4096
[root@null-04 ~]# mount /dev/test/lv1 /mnt


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 David Teigland 2020-06-15 20:52:35 UTC
I test this by creating the VG on ramdisk devs which have LBS 512 and PBS 4096:

# blockdev --getss --getpbsz /dev/ram0
512
4096

Comment 2 David Teigland 2020-06-16 16:28:39 UTC
It seems to me that increasing the logical block size under an active LV could be problematic in general, and what we see from xfs may be indicative of other applications.  They expect a certain io size or alignment to be usable while the device remains active.  If the block device allows 512 byte size|alignment at one moment, and then requires 4096 byte size|alignment the next, it could be challenging to adapt to that in the application.

So, my approach to this when it comes to adding integrity is if integrity is added to an active LV, avoid choosing an integrity block size that would result in the logical block size of the device increasing.  In practice the impact would be: if integrity is added to an active LV that has LBS 512 and a file system block size larger than 512, lvm would choose integrity block size of 512 rather than a larger size to match the file system block size.  The --raidintegrityblocksize option would allow overriding this restriction.

Comment 3 Mikuláš Patočka 2020-06-16 17:26:27 UTC
I reproduced this bug and I've found out that if you run this

dmsetup table /dev/test/lv1 | dmsetup load /dev/test/lv1
dmsetup suspend /dev/test/lv1; dmsetup resume /dev/test/lv1

then the sector size will be set to 4k. So, you can reload the raid target like this from lvm.

I wouldn't recommend using 512-byte sector size on integrity and writecache, because it makes them slow.

Comment 4 David Teigland 2020-06-16 18:13:09 UTC
Avoiding the problem is the workaround for now in master:
https://sourceware.org/git/?p=lvm2.git;a=commit;h=48872b0369af5ee048c947586d7da82d9769f5c3

I have it printing a message to give the user an idea that they are being subjected to this limitation:

# lvconvert --raidintegrity y test/lv1
  Creating integrity metadata LV lv1_rimage_0_imeta with size 12.00 MiB.
  Logical volume "lv1_rimage_0_imeta" created.
  Creating integrity metadata LV lv1_rimage_1_imeta with size 12.00 MiB.
  Logical volume "lv1_rimage_1_imeta" created.
  Limiting integrity block size to 512 because the LV is active.
  Using integrity block size 512 for file system block size 4096.
  Logical volume test/lv1 has added integrity.


(I would like to be able to rework the raid activation code to allow this, but that libdm activation code can be very difficult to figure out and get working correctly.  From my limited experience, it can be an exercise in figuring out all the existing special case workarounds, and then adding another special case workaround to work around the others.  We can perhaps use another bz that will let others assess the problem and prioritize the work to fix it.)

Comment 7 Debra Miller 2022-12-09 13:19:56 UTC
I have monitored such exercises which include working with statistics on demographics and other things about population and society. I have noticed that my biggest problem is working with correct data entry. I usually analyze the concepts like national integrity and so on through various mathematical and statistical experiments. Such assignments include math integrity, and I'm having trouble with this chapter too. Fortunately, I found a source that is https://graduateway.com/essay-examples/integrity/ to obtain and input reliable data about social integrity into my exercises. I'm not saying it's facile, but it's much easier to read essay examples with explanations about the experiment I have to analyze and then go through my technical steps. A student must have, at least, some support services during his work.


Note You need to log in before you can comment on or make changes to this bug.