RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2125680 - file system block size detection from libblkid is not working
Summary: file system block size detection from libblkid is not working
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: lvm2
Version: 9.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-09-09 15:59 UTC by Corey Marthaler
Modified: 2023-05-09 10:37 UTC (History)
8 users (show)

Fixed In Version: lvm2-2.03.17-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-09 08:23:45 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CLUSTERQE-6172 0 None None None 2022-11-28 20:42:22 UTC
Red Hat Issue Tracker RHELPLAN-133695 0 None None None 2022-09-09 16:15:11 UTC
Red Hat Product Errata RHBA-2023:2544 0 None None None 2023-05-09 08:24:04 UTC

Description Corey Marthaler 2022-09-09 15:59:13 UTC
Description of problem:
At first this appeared to be a regression of bug 1808012. I noticed that our test scenarios that do this operation were ending up with corrupted file systems almost every time lately. The lvm convert cmd does warn the user, but with the '--yes' flag becoming more and more "default" with all the minor issues that are prompted I fear this warning may be lost.

It's not really a "may", but more like a "most certainly will".


[root@hayes-03 ~]# lvcreate --wipesignatures y  -L 4G -n cworigin writecache_sanity /dev/sdm1
WARNING: xfs signature detected on /dev/writecache_sanity/cworigin at offset 0. Wipe it? [y/n]: y
  Wiping xfs signature on /dev/writecache_sanity/cworigin.
  Logical volume "cworigin" created.
[root@hayes-03 ~]# mkfs.xfs /dev/writecache_sanity/cworigin
meta-data=/dev/writecache_sanity/cworigin isize=512    agcount=4, agsize=262144 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=1048576, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@hayes-03 ~]# mount /dev/writecache_sanity/cworigin /mnt/cworigin/
[root@hayes-03 ~]# lvcreate  -L 2G -n cache_during_fs_io writecache_sanity /dev/sdg1
  Logical volume "cache_during_fs_io" created.
[root@hayes-03 ~]# lvs -a -o +devices,segtype
  LV                 VG                Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices      Type
  cache_during_fs_io writecache_sanity -wi-a----- 2.00g                                                     /dev/sdg1(0) linear
  cworigin           writecache_sanity -wi-ao---- 4.00g                                                     /dev/sdm1(0) linear
[root@hayes-03 ~]# lvchange -an writecache_sanity/cache_during_fs_io
[root@hayes-03 ~]# lvconvert --type writecache --cachevol writecache_sanity/cache_during_fs_io writecache_sanity/cworigin
Erase all existing data on writecache_sanity/cache_during_fs_io? [y/n]: y
  Using writecache block size 4096 for unknown file system block size, logical block size 512, physical block size 512.
  WARNING: unable to detect a file system block size on writecache_sanity/cworigin
  WARNING: using a writecache block size larger than the file system block size may corrupt the file system.
Use writecache block size 4096? [y/n]: y
  Logical volume writecache_sanity/cworigin now has writecache.


Sep  9 10:26:45 hayes-03 kernel: device-mapper: writecache: I/O is not aligned, sector 0, size 512, block size 4096
Sep  9 10:26:45 hayes-03 kernel: XFS (dm-0): metadata I/O error in "xfs_buf_ioend_handle_error+0xb3/0x3d0 [xfs]" at daddr 0x0 len 1 error 5
Sep  9 10:26:45 hayes-03 kernel: device-mapper: writecache: I/O is not aligned, sector 1, size 512, block size 4096
Sep  9 10:26:45 hayes-03 kernel: device-mapper: writecache: I/O is not aligned, sector 2, size 512, block size 4096
Sep  9 10:26:45 hayes-03 kernel: device-mapper: writecache: I/O is not aligned, sector 0, size 512, block size 4096
Sep  9 10:26:45 hayes-03 kernel: device-mapper: writecache: I/O is not aligned, sector 1, size 512, block size 4096
Sep  9 10:26:45 hayes-03 kernel: device-mapper: writecache: I/O is not aligned, sector 2, size 512, block size 4096
Sep  9 10:26:46 hayes-03 kernel: device-mapper: writecache: I/O is not aligned, sector 2097153, size 512, block size 4096
Sep  9 10:26:46 hayes-03 kernel: device-mapper: writecache: I/O is not aligned, sector 2097153, size 512, block size 4096
[...]



Version-Release number of selected component (if applicable):
kernel-5.14.0-161.el9    BUILT: Mon Sep  5 03:44:29 AM CDT 2022
lvm2-2.03.16-3.el9    BUILT: Mon Aug  1 04:42:35 AM CDT 2022
lvm2-libs-2.03.16-3.el9    BUILT: Mon Aug  1 04:42:35 AM CDT 2022


How reproducible:
Almost every time

Comment 1 David Teigland 2022-09-09 16:30:48 UTC
The messages are saying that it can't detect the xfs block size, which could be 512 or 4096.  I think that may be the more important issue here, i.e. is the code for checking the fs block size is not working?  When lvconvert can't detect the fs block size, it defaults to 4096.  If there is actually a file system on the LV, and if the fs uses a 4096 block size, then there should be no corruption.  But, if there's a fs with a 512 block size, then corruption is expected.  So, "may corrupt" is accurate since it depends on the fs block size.  (We originally defaulted to using 512 when the fs block size could not be detected, since that would always be safe, but that's a serious performance penalty for cases where there's no fs on the LV yet.)

To start, let's figure out why the fs block size is not being detected.  If you run that lvconvert with -vvvv, there should be one of two messages in the debug:

log_debug("Found blkid BLOCK_SIZE %u for fs on %s", *fs_block_size, pathname);

or

log_debug("No blkid BLOCK_SIZE for fs on %s", pathname);

and I'm guessing we'll see the second.  I recently discovered that there is a different libblkid function for checking the BLOCK_SIZE that might work in some situations where the current function we use doesn't.  So, the solution might be to switch the libblkid function we're calling.

Comment 2 Corey Marthaler 2022-09-09 17:49:25 UTC
12:47:12.738589 lvconvert[81830] activate/dev_manager.c:857  Getting device info for writecache_sanity-cache_during_fs_io [LVM-f3AFYefGLB1edHjmiLfHMfKbThjYGCBOHycROEwI1FxQrt78TigneTCU8JiryDfY].
12:47:12.738607 lvconvert[81830] device_mapper/ioctl/libdm-iface.c:2081  dm version   [ opencount flush ]   [2048] (*1)
12:47:12.738618 lvconvert[81830] device_mapper/ioctl/libdm-iface.c:2081  dm info  LVM-f3AFYefGLB1edHjmiLfHMfKbThjYGCBOHycROEwI1FxQrt78TigneTCU8JiryDfY [ noopencount flush ]   [2048] (*1)
12:47:12.738628 lvconvert[81830] activate/dev_manager.c:837  Skipping checks for old devices without LVM- dm uuid prefix (kernel vsn 5 >= 3).
12:47:12.738634 lvconvert[81830] activate/dev_manager.c:857  Getting device info for writecache_sanity-cworigin [LVM-f3AFYefGLB1edHjmiLfHMfKbThjYGCBO7Zurnq2QlsuP0PdHclr3uPsfb8iYx3vP].
12:47:12.738639 lvconvert[81830] device_mapper/ioctl/libdm-iface.c:2081  dm info  LVM-f3AFYefGLB1edHjmiLfHMfKbThjYGCBO7Zurnq2QlsuP0PdHclr3uPsfb8iYx3vP [ noopencount flush ]   [2048] (*1)
12:47:12.738649 lvconvert[81830] metadata/lv_manip.c:772  Generating list of PVs that writecache_sanity/cworigin uses:
12:47:12.738655 lvconvert[81830] metadata/lv_manip.c:742    writecache_sanity/cworigin uses /dev/sdm1
12:47:12.744318 lvconvert[81830] device/dev-type.c:836  No blkid BLOCK_SIZE for fs on /dev//writecache_sanity/cworigin
12:47:12.744332 lvconvert[81830] lvconvert.c:6017  Using writecache block size 4096 for unknown file system block size, logical block size 512, physical block size 512.
12:47:12.744342 lvconvert[81830] lvconvert.c:6021  WARNING: unable to detect a file system block size on writecache_sanity/cworigin
12:47:12.744347 lvconvert[81830] lvconvert.c:6022  WARNING: using a writecache block size larger than the file system block size may corrupt the file system.

Comment 3 David Teigland 2022-09-09 18:51:27 UTC
I found the email conversation where Karel Zak recommended avoiding blkid_get_tag_value() since it relies on some unreliable caching.

Also, I'm not sure if this is related, but I'm seeing inconsistent behavior from blkid on two of my test machines, and perhaps the default behavior changed at some point.  The older version is reporting BLOCK_SIZE and the newer isn't.

$ blkid -V
blkid from util-linux 2.32.1  (libblkid 2.32.1, 16-Jul-2018)
$ blkid -c /dev/null /dev/rhel_null-04/home
/dev/rhel_null-04/home: UUID="e97ff9cb-d00a-4d04-bd3e-48a3a20b32b5" BLOCK_SIZE="512" TYPE="xfs"


$ blkid -V
blkid from util-linux 2.38  (libblkid 2.38.0, 28-Mar-2022)
$ blkid -c /dev/null /dev/rhel/root
/dev/rhel/root: UUID="ec346632-551a-449d-8f60-e37250161857" TYPE="xfs"

In any case, I'll switch to the "probe" libblkid functions which seem to be more correct, and I expect that will fix the problem.

Comment 4 David Teigland 2022-09-09 20:56:27 UTC
I believe this is likely to fix the problem of missing fs block size detection, although the failure didn't occur on my test machines to compare before and after.
Fix in the main branch:
https://sourceware.org/git/?p=lvm2.git;a=commit;h=b869a6ff7a3d254a56ad78444e7d7f8af93d6844

Comment 5 David Teigland 2022-11-02 18:34:32 UTC
Interested in finding out if this issue appears on RHEL8 (considering whether to backport the fix.)

Comment 6 Corey Marthaler 2022-11-03 19:01:21 UTC
I have not see this in the latest testing of rhel8.

Comment 9 Corey Marthaler 2022-11-23 02:51:09 UTC
Marking Verified:Tested in the latest rpms.

kernel-5.14.0-176.el9    BUILT: Wed Oct 12 03:57:18 AM CDT 2022
lvm2-2.03.17-1.el9    BUILT: Thu Nov 10 10:02:16 AM CST 2022
lvm2-libs-2.03.17-1.el9    BUILT: Thu Nov 10 10:02:16 AM CST 2022


[root@hayes-02 ~]# lvcreate --wipesignatures y  -L 4G -n cworigin writecache_sanity /dev/sdk1
  Logical volume "cworigin" created.
[root@hayes-02 ~]# mkfs.xfs /dev/writecache_sanity/cworigin
meta-data=/dev/writecache_sanity/cworigin isize=512    agcount=4, agsize=262144 blks
[...]
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@hayes-02 ~]# mount /dev/writecache_sanity/cworigin /mnt/cworigin/
[root@hayes-02 ~]# df -h
Filesystem                              Size  Used Avail Use% Mounted on
/dev/mapper/writecache_sanity-cworigin  4.0G   61M  4.0G   2% /mnt/cworigin
[root@hayes-02 ~]# lvcreate  -L 2G -n cache_during_fs_io writecache_sanity /dev/sdg1
  Logical volume "cache_during_fs_io" created.
[root@hayes-02 ~]# lvs -a -o +devices,segtype
  LV                 VG                Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices      Type  
  cache_during_fs_io writecache_sanity -wi-a----- 2.00g                                                     /dev/sdg1(0) linear
  cworigin           writecache_sanity -wi-ao---- 4.00g                                                     /dev/sdk1(0) linear
[root@hayes-02 ~]# lvchange -an writecache_sanity/cache_during_fs_io
[root@hayes-02 ~]# lvconvert --type writecache --cachevol writecache_sanity/cache_during_fs_io writecache_sanity/cworigin
Erase all existing data on writecache_sanity/cache_during_fs_io? [y/n]: y
  Logical volume writecache_sanity/cworigin now has writecache.



Create xfs filesystem, add data and convert to writecache while it's under an I/O load

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sdp1
*  pool (fast):    /dev/sde1
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin

Checking files on /mnt/cworigin

Quick check to ensure online/active conversion is *NOT* allowed in rhel8.2 1808012
lvcreate --yes  -L 1G -n dummy_pool writecache_sanity @fast
lvconvert --yes --type writecache --cachevol writecache_sanity/dummy_pool writecache_sanity/cworigin
  LV writecache_sanity/dummy_pool must be inactive to attach.

Quick check to ensure active origin and *inactive* pool conversion is *NOW* allowed in rhel8.3 18534712 (8.2 1805532)
lvchange -an writecache_sanity/dummy_pool
lvconvert --test --yes --type writecache --cachevol writecache_sanity/dummy_pool writecache_sanity/cworigin
  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
  WARNING: unable to detect a file system block size on writecache_sanity/cworigin
  WARNING: using a writecache block size larger than the file system block size may corrupt the file system.
Starting an I/O load on the origin volume's FS
<start name="hayes-03"  pid="3016499" time="Tue Nov 22 20:48:31 2022 -0600" type="cmd" />
Sleeping 20 seconds to get some outsanding I/O locks before the conversion
Converting origin to a cache volume...
Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n cache_during_fs_io writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=1 high_watermark=92 writeback_jobs=1377 autocommit_blocks=1009 autocommit_time=1480' --cachevol writecache_sanity/cache_during_fs_io writecache_sanity/cworigin

Stopping the io load (collie/xdoio) on origin
<halt name="hayes-03"  pid="3016499" time="Tue Nov 22 20:49:14 2022 -0600" type="cmd" duration="43" ec="143" />
Checking files on /mnt/cworigin
Uncaching cache origin (lvconvert --yes --uncache writecache_sanity/cworigin) from cache origin

re-check data after uncache/splitcache (potential for bug 2125680)
Checking files on /mnt/cworigin

Quick check to ensure inactive origin and *active* pool conversion is *NOT* allowed in rhel8.2 (and STILL IN 8.3) 1805532
lvchange -ay writecache_sanity/dummy_pool
lvconvert --yes --type writecache --cachesettings --cachevol writecache_sanity/dummy_pool writecache_sanity/cworigin
  LV writecache_sanity/dummy_pool must be inactive to attach.
lvremove -f writecache_sanity/dummy_pool

Comment 14 Corey Marthaler 2022-12-15 18:17:40 UTC
Verified in the latest rpms as well.

kernel-5.14.0-205.el9    BUILT: Fri Dec  2 07:14:37 AM CST 2022
lvm2-2.03.17-3.el9    BUILT: Wed Dec  7 10:41:40 AM CST 2022
lvm2-libs-2.03.17-3.el9    BUILT: Wed Dec  7 10:41:40 AM CST 2022

Comment 16 errata-xmlrpc 2023-05-09 08:23:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2544


Note You need to log in before you can comment on or make changes to this bug.