Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1989253

Summary:	lvcreate can't load new vdo module
Product:	Red Hat Enterprise Linux 9	Reporter:	Andy Walsh <awalsh>
Component:	lvm2	Assignee:	Marian Csontos <mcsontos>
lvm2 sub component:	VDO	QA Contact:	cluster-qe <cluster-qe>
Status:	CLOSED CURRENTRELEASE	Docs Contact:
Severity:	medium
Priority:	urgent	CC:	agk, awalsh, bstinson, cluster-qe, cmarthal, heinzm, jbrassow, jwboyer, mcsontos, msnitzer, prajnoha, rmeggins, zkabelac
Version:	CentOS Stream	Keywords:	Triaged
Target Milestone:	beta	Flags:	pm-rhel: mirror+
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	lvm2-2.03.13-1.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1961890
Clones:	2028636 (view as bug list)		Environment:
Last Closed:	2021-12-07 21:24:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1961890
Bug Blocks:	1978488, 2028636

Description Andy Walsh 2021-08-02 18:12:25 UTC

+++ This bug was initially created as a clone of Bug #1961890 +++

Same basic problem, different release. Contents below updated to account for RHEL9.

Description of problem:
kernel-5.14.0-0.rc3.29.el9    BUILT: Tue 27 Jul 2021 09:35:18 AM EDT

kmod-kvdo-8.1.0.264-1.el9.x86_64    BUILT: Thu 29 Jul 2021 11:45:01 PM EDT
vdo-8.1.0.264-1.el9.x86_64    BUILT: Fri 30 Jul 2021 01:06:12 AM EDT

device-mapper-1.02.177-3.el9.x86_64    BUILT: Mon 24 May 2021 01:29:36 PM EDT
device-mapper-libs-1.02.177-3.el9.x86_64    BUILT: Mon 24 May 2021 01:29:36 PM EDT
device-mapper-event-libs-1.02.177-3.el9.x86_64    BUILT: Mon 24 May 2021 01:29:36 PM EDT
device-mapper-event-1.02.177-3.el9.x86_64    BUILT: Mon 24 May 2021 01:29:36 PM EDT
lvm2-libs-2.03.12-3.el9.x86_64    BUILT: Mon 24 May 2021 01:29:36 PM EDT
lvm2-2.03.12-3.el9.x86_64    BUILT: Mon 24 May 2021 01:29:36 PM EDT

[root@localhost ~]# find /lib/modules/5.14.0-0.rc3.29.el9.x86_64/extra/ -type f
/lib/modules/5.14.0-0.rc3.29.el9.x86_64/extra/kmod-kvdo/uds/uds.ko
/lib/modules/5.14.0-0.rc3.29.el9.x86_64/extra/kmod-kvdo/vdo/kvdo.ko

[root@localhost ~]# vgcreate vdo_base /dev/loop0
  Volume group "vdo_base" successfully created
[root@localhost ~]# lvcreate --type vdo -L 9G -V 100G vdo_base/vdopool
modprobe: FATAL: Module dm-vdo not found in directory /lib/modules/5.14.0-0.rc3.29.el9.x86_64
  /usr/sbin/modprobe failed: 1
  vdo: Required device-mapper target(s) not detected in your kernel.
  Run `lvcreate --help' for more information.

[root@localhost ~]# modprobe kvdo
[root@localhost ~]# lsmod | grep vdo
kvdo                  557056  1
uds                   282624  1 kvdo
dm_mod                172032  12 kvdo,dm_bufio

Comment 1 Andy Walsh 2021-08-02 18:14:47 UTC

I was able to work around this temporarily by creating a symlink and updating the module db.

[root@localhost ~]# cd /lib/modules/5.14.0-0.rc3.29.el9.x86_64/weak-updates/
[root@localhost weak-updates]# ln -snf ../extra/kmod-kvdo/vdo/kvdo.ko dm-vdo.ko
[root@localhost weak-updates]# depmod -a
[root@localhost weak-updates]# lvcreate --type vdo -L 9G -V 100G vdo_base/vdopool                                                                                                                                                                                                                                    
    The VDO volume can address 6 GB in 3 data slabs, each 2 GB.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
  Logical volume "lvol0" created.

Comment 4 Marian Csontos 2021-08-04 09:42:04 UTC

el9 needs rebuilding

Comment 5 Corey Marthaler 2021-08-31 03:12:37 UTC

This doesn't appear to be fixed in the latest rpms.

kernel-5.14.0-0.rc7.54.el9    BUILT: Tue Aug 24 02:54:42 PM CDT 2021
lvm2-2.03.13-1.el9    BUILT: Thu Aug 12 10:44:54 AM CDT 2021
lvm2-libs-2.03.13-1.el9    BUILT: Thu Aug 12 10:44:54 AM CDT 2021
vdo-8.1.0.316-1.el9.1    BUILT: Tue Aug 10 07:14:38 PM CDT 2021
kmod-kvdo-8.1.0.316-2.el9    BUILT: Tue Aug 10 05:34:43 AM CDT 2021


[root@hayes-03 ~]# ls /lib/modules/5.14.0-0.rc7.54.el9.x86_64/weak-updates
[root@hayes-03 ~]# ls /lib/modules/5.14.0-0.rc4.35.el9.x86_64/extra/kmod-kvdo/
uds  vdo


[root@hayes-03 ~]# vgcreate vdo_base /dev/sd[bcd]1
  Physical volume "/dev/sdb1" successfully created.
  Physical volume "/dev/sdc1" successfully created.
  Physical volume "/dev/sdd1" successfully created.
  Volume group "vdo_base" successfully created
[root@hayes-03 ~]# lvcreate --type vdo -L 9G -V 100G vdo_base/vdopool
modprobe: FATAL: Module kvdo not found in directory /lib/modules/5.14.0-0.rc7.54.el9.x86_64
  /usr/sbin/modprobe failed: 1
  vdo: Required device-mapper target(s) not detected in your kernel.
  Run `lvcreate --help' for more information.
[root@hayes-03 ~]# modprobe kvdo
modprobe: FATAL: Module kvdo not found in directory /lib/modules/5.14.0-0.rc7.54.el9.x86_64

Comment 6 Andy Walsh 2021-08-31 09:58:17 UTC

(In reply to Corey Marthaler from comment #5)
> This doesn't appear to be fixed in the latest rpms.

I think you're actually bumping into BZ1997635 now.

> [root@hayes-03 ~]# vgcreate vdo_base /dev/sd[bcd]1
>   Physical volume "/dev/sdb1" successfully created.
>   Physical volume "/dev/sdc1" successfully created.
>   Physical volume "/dev/sdd1" successfully created.
>   Volume group "vdo_base" successfully created
> [root@hayes-03 ~]# lvcreate --type vdo -L 9G -V 100G vdo_base/vdopool
> modprobe: FATAL: Module kvdo not found in directory

If you were still experiencing the problem from this particular BZ, it would say 'dm-vdo' instead of 'kvdo'

Comment 7 Zdenek Kabelac 2021-08-31 13:07:50 UTC

Yep - lvm2 tried to load  'kvdo'  but failed on kernel level to insert module.

Although new kvdo module seems to introduce some incompatibilies with existing lvm2 usage - so futher changes will be necessary
(i.e. BZ 1999568)

Comment 14 Corey Marthaler 2021-09-15 14:39:21 UTC

A lot of confusion in this bug. :) This is a rhel9 bug so the latest rhel9 lvm2 build is lvm2-2.03.13-1.el9. I was using that build in comment #5 but as Marian pointed out, not in comment #10. The only difference then is the kmod-kvdo rpm.

Using the latest kmod-kvdo (from Sept 8), the module load seems to work with a "out-of-tree module taints kernel" message.

kernel-5.14.0-1.el9    BUILT: Mon Aug 30 08:07:17 PM CDT 2021
lvm2-2.03.13-1.el9    BUILT: Thu Aug 12 10:44:54 AM CDT 2021
lvm2-libs-2.03.13-1.el9    BUILT: Thu Aug 12 10:44:54 AM CDT 2021
kmod-kvdo-8.1.0.316-3.el9_b    BUILT: Wed Sep  8 03:28:01 PM CDT 2021

Something doesn't appear correct with the LV attrs:

[root@hayes-02 ~]# lvcreate --yes --type vdo -n vdo1 -L 5G vg
[  282.633627] uds: loading out-of-tree module taints kernel.
[  282.639901] uds: module verification failed: signature and/or required key missing - tainting kernel
[  282.654450] uds: modprobe: loaded version 8.1.0.351
[  282.690789] kvdo: modprobe: loaded version 8.1.0.316
    Logical blocks defaulted to 523108 blocks.
    The VDO volume can address 2 GB in 1 data slab.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
[  284.332687] kvdo0:lvcreate: Detected version mismatch between kernel module and tools kernel: 4, tool: 2
[  284.343279] kvdo0:lvcreate: Please consider upgrading management tools to match kernel.
[  284.352227] kvdo0:lvcreate: loading device '253:1'
[  284.357764] kvdo0:lvcreate: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[  284.448279] kvdo0:lvcreate: starting device '253:1'
[  284.466003] kvdo0:physQ0: VDO commencing normal operation
[  284.472182] kvdo0:lvcreate: Setting UDS index target state to online
[  284.472188] uds: kvdo0:dedupeQ: creating index: dev=/dev/dm-0 offset=4096 size=2781704192
[  284.479277] kvdo0:lvcreate: device '253:1' started
[  284.479280] kvdo0:lvcreate: resuming device '253:1'
[  284.505067] kvdo0:lvcreate: device '253:1' resumed
[  284.520187] uds: kvdo0:dedupeQ: Using 16 indexing zones for concurrency.
[  284.523141] kvdo0:packerQ: compression is enabled
  Logical volume "vdo1" created.
[root@hayes-02 ~]# lvs
  LV     VG Attr       LSize Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vdo1   vg vwi-XXv-X- 1.99g vpool0                                               
  vpool0 vg dwi-XX--X- 5.00g                                                      
[root@hayes-02 ~]# lvs -a -o +devices
  LV             VG Attr       LSize Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  vdo1           vg vwi-XXv-X- 1.99g vpool0                                                vpool0(0)      
  vpool0         vg dwi-XX--X- 5.00g                                                       vpool0_vdata(0)
  [vpool0_vdata] vg Dwi-ao---- 5.00g                                                       /dev/sdc1(0)

Comment 16 Corey Marthaler 2021-09-15 15:04:05 UTC

With the creation and module loading now working in comment #14, we can mark this Verfied:Tested with the latest kmod-kvdo (From Sept 8).

The issues with the invalid lv_attrs are actually bug 1999568, which is not yet fixed in the latest rpms.

kernel-5.14.0-1.el9    BUILT: Mon Aug 30 08:07:17 PM CDT 2021
lvm2-2.03.13-1.el9    BUILT: Thu Aug 12 10:44:54 AM CDT 2021
lvm2-libs-2.03.13-1.el9    BUILT: Thu Aug 12 10:44:54 AM CDT 2021
kmod-kvdo-8.1.0.316-3.el9_b    BUILT: Wed Sep  8 03:28:01 PM CDT 2021

Comment 19 Corey Marthaler 2021-09-21 15:27:23 UTC

Marking Verified in the latest lvm2 and kernel.

kernel-5.14.0-1.2.1.el9    BUILT: Fri Sep 17 02:57:25 AM CDT 2021
lvm2-2.03.13-2.el9_b    BUILT: Fri Sep 17 10:55:56 AM CDT 2021
lvm2-libs-2.03.13-2.el9_b    BUILT: Fri Sep 17 10:55:56 AM CDT 2021
vdo-8.1.0.316-1.el9.1    BUILT: Tue Aug 10 07:14:38 PM CDT 2021
kmod-kvdo-8.1.0.316-3.el9_b    BUILT: Wed Sep  8 03:28:01 PM CDT 2021


[root@hayes-02 ~]# lvcreate --yes --type vdo -n vdo1 -L 20G VG
[   84.606454] uds: loading out-of-tree module taints kernel.
[   84.612747] uds: module verification failed: signature and/or required key missing - tainting kernel
[   84.627276] uds: modprobe: loaded version 8.1.0.351
[   84.662079] kvdo: modprobe: loaded version 8.1.0.316
  Wiping vdo signature on /dev/VG/vpool0.
    Logical blocks defaulted to 4186130 blocks.
    The VDO volume can address 16 GB in 8 data slabs, each 2 GB.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
[   86.460102] kvdo0:lvcreate: Detected version mismatch between kernel module and tools kernel: 4, tool: 2
[   86.470695] kvdo0:lvcreate: Please consider upgrading management tools to match kernel.
[   86.479642] kvdo0:lvcreate: loading device '253:1'
[   86.485171] kvdo0:lvcreate: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[   86.577114] kvdo0:lvcreate: starting device '253:1'
[   86.595123] kvdo0:physQ0: VDO commencing normal operation
[   86.601276] kvdo0:lvcreate: Setting UDS index target state to online
[   86.601282] uds: kvdo0:dedupeQ: creating index: dev=/dev/dm-0 offset=4096 size=2781704192
[   86.608372] kvdo0:lvcreate: device '253:1' started
[   86.622836] kvdo0:lvcreate: resuming device '253:1'
[   86.633905] kvdo0:lvcreate: device '253:1' resumed
[   86.646009] kvdo0:packerQ: compression is enabled
[   86.649292] uds: kvdo0:dedupeQ: Using 16 indexing zones for concurrency.
  Logical volume "vdo1" created.

[root@hayes-02 ~]# lvs -a -o +devices
  LV             VG Attr       LSize  Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  vdo1           VG vwi-a-v--- 15.96g vpool0        0.00                                    vpool0(0)      
  vpool0         VG dwi------- 20.00g               20.05                                   vpool0_vdata(0)
  [vpool0_vdata] VG Dwi-ao---- 20.00g                                                       /dev/sdb1(0)