Bug 2125680
| Summary: | file system block size detection from libblkid is not working | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Corey Marthaler <cmarthal> |
| Component: | lvm2 | Assignee: | David Teigland <teigland> |
| lvm2 sub component: | Cache Logical Volumes | QA Contact: | cluster-qe <cluster-qe> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | agk, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, teigland, zkabelac |
| Version: | 9.1 | Keywords: | Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | lvm2-2.03.17-1.el9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-05-09 08:23:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Corey Marthaler
2022-09-09 15:59:13 UTC
The messages are saying that it can't detect the xfs block size, which could be 512 or 4096. I think that may be the more important issue here, i.e. is the code for checking the fs block size is not working? When lvconvert can't detect the fs block size, it defaults to 4096. If there is actually a file system on the LV, and if the fs uses a 4096 block size, then there should be no corruption. But, if there's a fs with a 512 block size, then corruption is expected. So, "may corrupt" is accurate since it depends on the fs block size. (We originally defaulted to using 512 when the fs block size could not be detected, since that would always be safe, but that's a serious performance penalty for cases where there's no fs on the LV yet.)
To start, let's figure out why the fs block size is not being detected. If you run that lvconvert with -vvvv, there should be one of two messages in the debug:
log_debug("Found blkid BLOCK_SIZE %u for fs on %s", *fs_block_size, pathname);
or
log_debug("No blkid BLOCK_SIZE for fs on %s", pathname);
and I'm guessing we'll see the second. I recently discovered that there is a different libblkid function for checking the BLOCK_SIZE that might work in some situations where the current function we use doesn't. So, the solution might be to switch the libblkid function we're calling.
12:47:12.738589 lvconvert[81830] activate/dev_manager.c:857 Getting device info for writecache_sanity-cache_during_fs_io [LVM-f3AFYefGLB1edHjmiLfHMfKbThjYGCBOHycROEwI1FxQrt78TigneTCU8JiryDfY]. 12:47:12.738607 lvconvert[81830] device_mapper/ioctl/libdm-iface.c:2081 dm version [ opencount flush ] [2048] (*1) 12:47:12.738618 lvconvert[81830] device_mapper/ioctl/libdm-iface.c:2081 dm info LVM-f3AFYefGLB1edHjmiLfHMfKbThjYGCBOHycROEwI1FxQrt78TigneTCU8JiryDfY [ noopencount flush ] [2048] (*1) 12:47:12.738628 lvconvert[81830] activate/dev_manager.c:837 Skipping checks for old devices without LVM- dm uuid prefix (kernel vsn 5 >= 3). 12:47:12.738634 lvconvert[81830] activate/dev_manager.c:857 Getting device info for writecache_sanity-cworigin [LVM-f3AFYefGLB1edHjmiLfHMfKbThjYGCBO7Zurnq2QlsuP0PdHclr3uPsfb8iYx3vP]. 12:47:12.738639 lvconvert[81830] device_mapper/ioctl/libdm-iface.c:2081 dm info LVM-f3AFYefGLB1edHjmiLfHMfKbThjYGCBO7Zurnq2QlsuP0PdHclr3uPsfb8iYx3vP [ noopencount flush ] [2048] (*1) 12:47:12.738649 lvconvert[81830] metadata/lv_manip.c:772 Generating list of PVs that writecache_sanity/cworigin uses: 12:47:12.738655 lvconvert[81830] metadata/lv_manip.c:742 writecache_sanity/cworigin uses /dev/sdm1 12:47:12.744318 lvconvert[81830] device/dev-type.c:836 No blkid BLOCK_SIZE for fs on /dev//writecache_sanity/cworigin 12:47:12.744332 lvconvert[81830] lvconvert.c:6017 Using writecache block size 4096 for unknown file system block size, logical block size 512, physical block size 512. 12:47:12.744342 lvconvert[81830] lvconvert.c:6021 WARNING: unable to detect a file system block size on writecache_sanity/cworigin 12:47:12.744347 lvconvert[81830] lvconvert.c:6022 WARNING: using a writecache block size larger than the file system block size may corrupt the file system. I found the email conversation where Karel Zak recommended avoiding blkid_get_tag_value() since it relies on some unreliable caching. Also, I'm not sure if this is related, but I'm seeing inconsistent behavior from blkid on two of my test machines, and perhaps the default behavior changed at some point. The older version is reporting BLOCK_SIZE and the newer isn't. $ blkid -V blkid from util-linux 2.32.1 (libblkid 2.32.1, 16-Jul-2018) $ blkid -c /dev/null /dev/rhel_null-04/home /dev/rhel_null-04/home: UUID="e97ff9cb-d00a-4d04-bd3e-48a3a20b32b5" BLOCK_SIZE="512" TYPE="xfs" $ blkid -V blkid from util-linux 2.38 (libblkid 2.38.0, 28-Mar-2022) $ blkid -c /dev/null /dev/rhel/root /dev/rhel/root: UUID="ec346632-551a-449d-8f60-e37250161857" TYPE="xfs" In any case, I'll switch to the "probe" libblkid functions which seem to be more correct, and I expect that will fix the problem. I believe this is likely to fix the problem of missing fs block size detection, although the failure didn't occur on my test machines to compare before and after. Fix in the main branch: https://sourceware.org/git/?p=lvm2.git;a=commit;h=b869a6ff7a3d254a56ad78444e7d7f8af93d6844 Interested in finding out if this issue appears on RHEL8 (considering whether to backport the fix.) I have not see this in the latest testing of rhel8. Marking Verified:Tested in the latest rpms. kernel-5.14.0-176.el9 BUILT: Wed Oct 12 03:57:18 AM CDT 2022 lvm2-2.03.17-1.el9 BUILT: Thu Nov 10 10:02:16 AM CST 2022 lvm2-libs-2.03.17-1.el9 BUILT: Thu Nov 10 10:02:16 AM CST 2022 [root@hayes-02 ~]# lvcreate --wipesignatures y -L 4G -n cworigin writecache_sanity /dev/sdk1 Logical volume "cworigin" created. [root@hayes-02 ~]# mkfs.xfs /dev/writecache_sanity/cworigin meta-data=/dev/writecache_sanity/cworigin isize=512 agcount=4, agsize=262144 blks [...] realtime =none extsz=4096 blocks=0, rtextents=0 [root@hayes-02 ~]# mount /dev/writecache_sanity/cworigin /mnt/cworigin/ [root@hayes-02 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/writecache_sanity-cworigin 4.0G 61M 4.0G 2% /mnt/cworigin [root@hayes-02 ~]# lvcreate -L 2G -n cache_during_fs_io writecache_sanity /dev/sdg1 Logical volume "cache_during_fs_io" created. [root@hayes-02 ~]# lvs -a -o +devices,segtype LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices Type cache_during_fs_io writecache_sanity -wi-a----- 2.00g /dev/sdg1(0) linear cworigin writecache_sanity -wi-ao---- 4.00g /dev/sdk1(0) linear [root@hayes-02 ~]# lvchange -an writecache_sanity/cache_during_fs_io [root@hayes-02 ~]# lvconvert --type writecache --cachevol writecache_sanity/cache_during_fs_io writecache_sanity/cworigin Erase all existing data on writecache_sanity/cache_during_fs_io? [y/n]: y Logical volume writecache_sanity/cworigin now has writecache. Create xfs filesystem, add data and convert to writecache while it's under an I/O load *** Writecache info for this scenario *** * origin (slow): /dev/sdp1 * pool (fast): /dev/sde1 ************************************ Adding "slow" and "fast" tags to corresponding pvs Create origin (slow) volume lvcreate --yes --wipesignatures y -L 4G -n cworigin writecache_sanity @slow Placing an xfs filesystem on origin volume Mounting origin volume Writing files to /mnt/cworigin Checking files on /mnt/cworigin Quick check to ensure online/active conversion is *NOT* allowed in rhel8.2 1808012 lvcreate --yes -L 1G -n dummy_pool writecache_sanity @fast lvconvert --yes --type writecache --cachevol writecache_sanity/dummy_pool writecache_sanity/cworigin LV writecache_sanity/dummy_pool must be inactive to attach. Quick check to ensure active origin and *inactive* pool conversion is *NOW* allowed in rhel8.3 18534712 (8.2 1805532) lvchange -an writecache_sanity/dummy_pool lvconvert --test --yes --type writecache --cachevol writecache_sanity/dummy_pool writecache_sanity/cworigin TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated. WARNING: unable to detect a file system block size on writecache_sanity/cworigin WARNING: using a writecache block size larger than the file system block size may corrupt the file system. Starting an I/O load on the origin volume's FS <start name="hayes-03" pid="3016499" time="Tue Nov 22 20:48:31 2022 -0600" type="cmd" /> Sleeping 20 seconds to get some outsanding I/O locks before the conversion Converting origin to a cache volume... Create writecache cvol (fast) volumes lvcreate --yes -L 2G -n cache_during_fs_io writecache_sanity @fast Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347) Create writecached volume by combining the cache pool (fast) and origin (slow) volumes lvconvert --yes --type writecache --cachesettings 'low_watermark=1 high_watermark=92 writeback_jobs=1377 autocommit_blocks=1009 autocommit_time=1480' --cachevol writecache_sanity/cache_during_fs_io writecache_sanity/cworigin Stopping the io load (collie/xdoio) on origin <halt name="hayes-03" pid="3016499" time="Tue Nov 22 20:49:14 2022 -0600" type="cmd" duration="43" ec="143" /> Checking files on /mnt/cworigin Uncaching cache origin (lvconvert --yes --uncache writecache_sanity/cworigin) from cache origin re-check data after uncache/splitcache (potential for bug 2125680) Checking files on /mnt/cworigin Quick check to ensure inactive origin and *active* pool conversion is *NOT* allowed in rhel8.2 (and STILL IN 8.3) 1805532 lvchange -ay writecache_sanity/dummy_pool lvconvert --yes --type writecache --cachesettings --cachevol writecache_sanity/dummy_pool writecache_sanity/cworigin LV writecache_sanity/dummy_pool must be inactive to attach. lvremove -f writecache_sanity/dummy_pool Verified in the latest rpms as well. kernel-5.14.0-205.el9 BUILT: Fri Dec 2 07:14:37 AM CST 2022 lvm2-2.03.17-3.el9 BUILT: Wed Dec 7 10:41:40 AM CST 2022 lvm2-libs-2.03.17-3.el9 BUILT: Wed Dec 7 10:41:40 AM CST 2022 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:2544 |