Bug 1999568

Summary: VDO pool statistics are no longer available in /sys/kvdo with kvdo version 8
Product: Red Hat Enterprise Linux 9 Reporter: Vojtech Trefny <vtrefny>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
lvm2 sub component: VDO QA Contact: cluster-qe <cluster-qe>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: unspecified    
Priority: urgent CC: agk, awalsh, cmarthal, heinzm, jbrassow, lmiksik, mcsontos, prajnoha, tbzatek, zkabelac
Version: 9.0Keywords: Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: lvm2-2.03.13-2.el9_b Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2005029 (view as bug list) Environment:
Last Closed: 2021-12-07 21:42:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1994220, 2005029    

Description Vojtech Trefny 2021-08-31 11:08:13 UTC 
Description of problem:

LVM reads vdo_used_size from /sys/kvdo/<pool>/statistics/data_blocks_used which is no longer available with latest kvdo on RHEL 9, pool statistics are now available in /sys/block/<pool>/vdo/statistics

Relevant part of "lvs -a -o+vdo_saving_percent,vdo_used_size -vvv" output:

----
  Processing LV vpool0 in VG test.
  Getting device info for test-vpool0-vpool [LVM-Owve405pRd5exJx6FULQ3WdQMNd4XsNha3nVYQ7YqHPBjDYQMzQpgaZF3v7zJAi0-vpool].
  dm status  LVM-Owve405pRd5exJx6FULQ3WdQMNd4XsNha3nVYQ7YqHPBjDYQMzQpgaZF3v7zJAi0-vpool [ noopencount noflush ]   [16384] (*1)
  /sys/kvdo/test-vpool0-vpool/statistics/data_blocks_used: open failed: No such file or directory
  Processing LV lvol0 in VG test.
  Getting device info for test-lvol0 [LVM-Owve405pRd5exJx6FULQ3WdQMNd4XsNh5o3cQ1vdsG6lANZbOBJIrYGGhzzMVMRZ].
  dm info  LVM-Owve405pRd5exJx6FULQ3WdQMNd4XsNh5o3cQ1vdsG6lANZbOBJIrYGGhzzMVMRZ [ opencount flush ]   [2048] (*1)
  test-lvol0 (253:4): read ahead is 256
  Getting device info for test-vpool0-vpool [LVM-Owve405pRd5exJx6FULQ3WdQMNd4XsNha3nVYQ7YqHPBjDYQMzQpgaZF3v7zJAi0-vpool].
  dm status  LVM-Owve405pRd5exJx6FULQ3WdQMNd4XsNha3nVYQ7YqHPBjDYQMzQpgaZF3v7zJAi0-vpool [ noopencount noflush ]   [16384] (*1)
  /sys/kvdo/test-vpool0-vpool/statistics/data_blocks_used: open failed: No such file or directory
----


Version-Release number of selected component (if applicable):

# lvm version
  LVM version:     2.03.12(2) (2021-05-07)
  Library version: 1.02.177 (2021-05-07)
  Driver version:  4.45.0

# rpm -qi kmod-kvdo
Name        : kmod-kvdo
Version     : 8.1.0.316
Release     : 2.el9
Comment 4 Zdenek Kabelac 2021-09-09 14:29:32 UTC 
Pushed upstream patch:

https://listman.redhat.com/archives/lvm-devel/2021-September/msg00006.html


Patch is trying first to access new location /sys/block/dm-XXX/vdo/statistics/...
Comment 8 Corey Marthaler 2021-09-15 15:00:01 UTC 
Here is what this bug looks like with the latest rpms:

kernel-5.14.0-1.el9    BUILT: Mon Aug 30 08:07:17 PM CDT 2021
lvm2-2.03.13-1.el9    BUILT: Thu Aug 12 10:44:54 AM CDT 2021
lvm2-libs-2.03.13-1.el9    BUILT: Thu Aug 12 10:44:54 AM CDT 2021
vdo-8.1.0.316-1.el9.1    BUILT: Tue Aug 10 07:14:38 PM CDT 2021
kmod-kvdo-8.1.0.316-3.el9_b    BUILT: Wed Sep  8 03:28:01 PM CDT 2021

[root@hayes-02 ~]# lvcreate --yes --type vdo -n vdo1 -L 20G VG
  Wiping vdo signature on /dev/VG/vpool0.
    Logical blocks defaulted to 4186130 blocks.
    The VDO volume can address 16 GB in 8 data slabs, each 2 GB.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
[ 1339.280897] kvdo1:lvcreate: Detected version mismatch between kernel module and tools kernel: 4, tool: 2
[ 1339.291492] kvdo1:lvcreate: Please consider upgrading management tools to match kernel.
[ 1339.300453] kvdo1:lvcreate: loading device '253:1'
[ 1339.305996] kvdo1:lvcreate: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[ 1339.394705] kvdo1:lvcreate: starting device '253:1'
[ 1339.419460] kvdo1:physQ0: VDO commencing normal operation
[ 1339.425618] kvdo1:lvcreate: Setting UDS index target state to online
[ 1339.425624] uds: kvdo1:dedupeQ: creating index: dev=/dev/dm-0 offset=4096 size=2781704192
[ 1339.432713] kvdo1:lvcreate: device '253:1' started
[ 1339.432715] kvdo1:lvcreate: resuming device '253:1'
[ 1339.458528] kvdo1:lvcreate: device '253:1' resumed
[ 1339.472628] kvdo1:packerQ: compression is enabled
[ 1339.473646] uds: kvdo1:dedupeQ: Using 16 indexing zones for concurrency.
  Logical volume "vdo1" created.

[root@hayes-02 ~]# lvs
  LV     VG Attr       LSize  Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vdo1   VG vwi-XXv-X- 15.96g vpool0                                               
  vpool0 VG dwi-XX--X- 20.00g                                                      
[root@hayes-02 ~]# lvs -a -o +devices
  LV             VG Attr       LSize  Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  vdo1           VG vwi-XXv-X- 15.96g vpool0                                                vpool0(0)      
  vpool0         VG dwi-XX--X- 20.00g                                                       vpool0_vdata(0)
  [vpool0_vdata] VG Dwi-ao---- 20.00g                                                       /dev/sdc1(0)
Comment 13 Corey Marthaler 2021-09-21 15:23:58 UTC 
Verified in the latest build.

kernel-5.14.0-1.2.1.el9    BUILT: Fri Sep 17 02:57:25 AM CDT 2021
lvm2-2.03.13-2.el9_b    BUILT: Fri Sep 17 10:55:56 AM CDT 2021
lvm2-libs-2.03.13-2.el9_b    BUILT: Fri Sep 17 10:55:56 AM CDT 2021


[root@hayes-02 ~]# lvcreate --yes --type vdo -n vdo1 -L 20G VG
[   84.606454] uds: loading out-of-tree module taints kernel.
[   84.612747] uds: module verification failed: signature and/or required key missing - tainting kernel
[   84.627276] uds: modprobe: loaded version 8.1.0.351
[   84.662079] kvdo: modprobe: loaded version 8.1.0.316
  Wiping vdo signature on /dev/VG/vpool0.
    Logical blocks defaulted to 4186130 blocks.
    The VDO volume can address 16 GB in 8 data slabs, each 2 GB.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
[   86.460102] kvdo0:lvcreate: Detected version mismatch between kernel module and tools kernel: 4, tool: 2
[   86.470695] kvdo0:lvcreate: Please consider upgrading management tools to match kernel.
[   86.479642] kvdo0:lvcreate: loading device '253:1'
[   86.485171] kvdo0:lvcreate: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[   86.577114] kvdo0:lvcreate: starting device '253:1'
[   86.595123] kvdo0:physQ0: VDO commencing normal operation
[   86.601276] kvdo0:lvcreate: Setting UDS index target state to online
[   86.601282] uds: kvdo0:dedupeQ: creating index: dev=/dev/dm-0 offset=4096 size=2781704192
[   86.608372] kvdo0:lvcreate: device '253:1' started
[   86.622836] kvdo0:lvcreate: resuming device '253:1'
[   86.633905] kvdo0:lvcreate: device '253:1' resumed
[   86.646009] kvdo0:packerQ: compression is enabled
[   86.649292] uds: kvdo0:dedupeQ: Using 16 indexing zones for concurrency.
  Logical volume "vdo1" created.
[root@hayes-02 ~]# lvs -a -o +devices
  LV             VG Attr       LSize  Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  vdo1           VG vwi-a-v--- 15.96g vpool0        0.00                                    vpool0(0)      
  vpool0         VG dwi------- 20.00g               20.05                                   vpool0_vdata(0)
  [vpool0_vdata] VG Dwi-ao---- 20.00g                                                       /dev/sdb1(0)
Comment 16 Corey Marthaler 2021-09-22 15:06:51 UTC 
Marking VERIFIED in the latest.

kernel-5.14.0-1.2.1.el9    BUILT: Fri Sep 17 02:57:25 AM CDT 2021
lvm2-2.03.13-2.el9_b    BUILT: Fri Sep 17 10:55:56 AM CDT 2021
lvm2-libs-2.03.13-2.el9_b    BUILT: Fri Sep 17 10:55:56 AM CDT 2021

vdo-8.1.0.316-1.el9.1    BUILT: Tue Aug 10 07:14:38 PM CDT 2021
kmod-kvdo-8.1.0.316-3.el9_b    BUILT: Wed Sep  8 03:28:01 PM CDT 2021


[root@hayes-02 ~]# lvs -a -o +devices,vdo_compression,vdo_deduplication,vdo_use_metadata_hints,vdo_minimum_io_size
  LV             VG Attr       LSize  Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices         VDOCompression VDODeduplication VDOMetadataHints VDOMinimumIOSize
  vdo1           VG vwi-a-v--- 15.96g vpool0        0.00                                    vpool0(0)              enabled          enabled          enabled            4.00k
  vpool0         VG dwi------- 20.00g               20.05                                   vpool0_vdata(0)        enabled          enabled          enabled            4.00k
  [vpool0_vdata] VG Dwi-ao---- 20.00g                                                       /dev/sdb1(0)