RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1695330 - Latest kmod-kvdo does not work with latest RHEL-8.1.0 kernel
Summary: Latest kmod-kvdo does not work with latest RHEL-8.1.0 kernel
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: kmod-kvdo
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.1
Assignee: Andy Walsh
QA Contact: vdo-qe
URL:
Whiteboard:
: 1698299 (view as bug list)
Depends On:
Blocks: 1760741
TreeView+ depends on / blocked
 
Reported: 2019-04-02 20:25 UTC by Andy Walsh
Modified: 2023-09-07 19:52 UTC (History)
4 users (show)

Fixed In Version: kmod-kvdo-6.2.1.138-57.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1760741 (view as bug list)
Environment:
Last Closed: 2020-04-28 16:42:57 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-24171 0 None None None 2021-09-03 15:10:49 UTC
Red Hat Product Errata RHBA-2020:1782 0 None None None 2020-04-28 16:43:13 UTC

Description Andy Walsh 2019-04-02 20:25:27 UTC 
Description of problem:
The latest kmod-kvdo package does not work against the latest RHEL-8.1.0 kernel due to kABI compatibility problem.

Version-Release number of selected component (if applicable):
kernel-4.18.0-80.6.el8
kmod-kvdo-6.2.0.293-50.el8

How reproducible:
100%

Steps to Reproduce:
1. Install a kernel starting at 4.18.0-80.6 or greater
2. Reboot into the newly installed kernel
3. Install the latest kmod-kvdo (version 6.2.0.293-50)
4. Attempt to insmod/modprobe the kernel modules
5. Notice failure in dmesg output.

Actual results:
[root@host-8-242-17 ~]# insmod /lib/modules/4.18.0-72.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko 
insmod: ERROR: could not insert module kvdo.ko: Unknown symbol in module
[root@host-8-241-181 ~]# dmesg
[  254.096707] kvdo: Unknown symbol dm_put_device (err 0)
[  254.096843] kvdo: Unknown symbol dm_shift_arg (err 0)
[  254.096851] kvdo: Unknown symbol dm_unregister_target (err 0)
[  254.096880] kvdo: Unknown symbol dm_set_target_max_io_len (err 0)
[  254.096919] kvdo: Unknown symbol dm_register_target (err 0)
[  254.096937] kvdo: Unknown symbol dm_table_get_mode (err 0)
[  254.096952] kvdo: Unknown symbol dm_get_device (err 0)
[  254.096963] kvdo: Unknown symbol dm_consume_args (err 0)

Expected results:
[root@host-8-241-181 ~]# modprobe kvdo
[root@host-8-241-181 ~]# dmesg
[  168.906841] device-mapper: uevent: version 1.0.3
[  168.934245] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel
[  168.937978] uds: loading out-of-tree module taints kernel.
[  168.942918] uds: modprobe: loaded version 6.2.0.77
[  168.971569] kvdo: modprobe: loaded version 6.2.0.293


Additional info:

Full output of trying to load the module:
[root@host-8-242-17 ~]# uname -r
4.18.0-80.6.el8.x86_64
[root@host-8-242-17 ~]# rpm -qa kmod-kvdo
kmod-kvdo-6.2.0.293-50.el8.x86_64
[root@host-8-242-17 ~]# dmesg -c > /dev/null
[root@host-8-242-17 ~]# insmod /lib/modules/4.18.0-72.el8.x86_64/extra/kmod-kvdo/uds/uds.ko 
[root@host-8-242-17 ~]# dmesg
[  240.050464] uds: loading out-of-tree module taints kernel.
[  240.056809] uds: insmod: loaded version 6.2.0.77
[root@host-8-242-17 ~]# insmod /lib/modules/4.18.0-72.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko 
insmod: ERROR: could not insert module kvdo.ko: Unknown symbol in module
[root@host-8-242-17 ~]# dmesg
[  240.050464] uds: loading out-of-tree module taints kernel.
[  240.056809] uds: insmod: loaded version 6.2.0.77
[  254.096707] kvdo: Unknown symbol dm_put_device (err 0)
[  254.096843] kvdo: Unknown symbol dm_shift_arg (err 0)
[  254.096851] kvdo: Unknown symbol dm_unregister_target (err 0)
[  254.096880] kvdo: Unknown symbol dm_set_target_max_io_len (err 0)
[  254.096919] kvdo: Unknown symbol dm_register_target (err 0)
[  254.096937] kvdo: Unknown symbol dm_table_get_mode (err 0)
[  254.096952] kvdo: Unknown symbol dm_get_device (err 0)
[  254.096963] kvdo: Unknown symbol dm_consume_args (err 0)
Comment 1 Andy Walsh 2019-04-10 12:21:01 UTC 
*** Bug 1698299 has been marked as a duplicate of this bug. ***
Comment 2 ChanghuiZhong 2019-05-05 08:41:15 UTC 
Hello,

The same problem occurs in 4.18.0-81.el8.x86_64

4.18.0-81.el8.x86_64
vdo-6.2.0.293-10.el8.x86_64
kmod-kvdo-6.2.0.293-50.el8.x86_64

[root@storageqe-26 vdo]# modprobe kvdo.ko 
modprobe: FATAL: Module kvdo.ko not found in directory /lib/modules/4.18.0-81.el8.x86_64
[root@storageqe-26 vdo]# 

[root@storageqe-26 vdo]# insmod kvdo.ko 
insmod: ERROR: could not insert module kvdo.ko: Unknown symbol in module
[root@storageqe-26 vdo]# 

[ 2713.940929] kvdo: disagrees about version of symbol dm_register_target
[ 2713.969905] kvdo: Unknown symbol dm_register_target (err -22)
[ 2713.996331] kvdo: Unknown symbol getBytesFromBuffer (err 0)
[ 2714.021460] kvdo: Unknown symbol contentLength (err 0)
[ 2714.044165] kvdo: Unknown symbol nowUsec (err 0)
[ 2714.064884] kvdo: Unknown symbol recordBioAlloc (err 0)
[ 2714.088587] kvdo: Unknown symbol registerAllocatingThread (err 0)
[ 2714.116317] kvdo: Unknown symbol hasSameBytes (err 0)
[ 2714.141149] kvdo: Unknown symbol udsCloseIndexSession (err 0)
[ 2714.169614] kvdo: Unknown symbol allocateMemory (err 0)
[ 2714.193048] kvdo: Unknown symbol udsInitializeConfiguration (err 0)
[ 2714.221428] kvdo: Unknown symbol allocSprintf (err 0)
[ 2714.244431] kvdo: Unknown symbol copyBytes (err 0)
[ 2714.266087] kvdo: disagrees about version of symbol dm_get_device
[ 2714.293471] kvdo: Unknown symbol dm_get_device (err -22)
[ 2714.317292] kvdo: Unknown symbol currentTime (err 0)
[ 2714.339674] kvdo: Unknown symbol resetBufferEnd (err 0)
[ 2714.363263] kvdo: Unknown symbol getUInt32LEFromBuffer (err 0)
Comment 4 Andy Walsh 2019-05-05 21:41:55 UTC 
I have just posted an updated version that should work with 4.18.0-81 and newer.  Updated NVR is kmod-kvdo-6.2.0.293-51.el8.
Comment 5 ChanghuiZhong 2019-05-06 03:32:07 UTC 
Hi,Andy

Confirm that kmod-kvdo-6.2.0.293-51.el8 work well with 4.18.0-81.el8,thanks for your help

[root@storageqe-26 vdo]# modprobe kvdo
[root@storageqe-26 vdo]# 

[ 3234.499452] uds: modprobe: loaded version 6.2.0.77
[ 3234.521266] kvdo: modprobe: loaded version 6.2.0.293


Thanks
Changhui
Comment 7 ChanghuiZhong 2019-05-22 02:48:44 UTC 
Hello,

The same problem occurs in 4.18.0-92.el8.x86_64.


vdo-6.2.1.48-10.el8.x86_64
kmod-kvdo-6.2.1.48-51.el8.x86_64

Creating VDO vPV1
STDERR:vdo: ERROR - Kernel module kvdo not installed
vdo: ERROR - modprobe: FATAL: Module kvdo not found in directory /lib/modules/4.18.0-92.el8.x86_64
RETURN:1
Comment 8 Andy Walsh 2019-05-22 13:33:19 UTC 
(In reply to ChanghuiZhong from comment #7)
> Hello,
> 
> The same problem occurs in 4.18.0-92.el8.x86_64.
> 
> 
> vdo-6.2.1.48-10.el8.x86_64
> kmod-kvdo-6.2.1.48-51.el8.x86_64
> 
> Creating VDO vPV1
> STDERR:vdo: ERROR - Kernel module kvdo not installed
> vdo: ERROR - modprobe: FATAL: Module kvdo not found in directory
> /lib/modules/4.18.0-92.el8.x86_64
> RETURN:1

I built kmod-kvdo-6.2.1.48-52.el8 yesterday to fix this problem.
Comment 9 Andy Walsh 2019-05-30 13:26:54 UTC 
It looks like kernel-4.18.0-97.el8 causes kmod-kvdo-6.2.1.48-52.el8 to stop working on aarch64.  I have found that it does work on x86_64 in my testing, but tests are still running for ppc64le and s390x, so I'm not sure about those.  One failure is enough for me to rebuild, though.

[root@localhost ~]# rpm -qa kernel kmod-kvdo vdo lvm2
lvm2-2.03.02-6.el8.aarch64
kernel-4.18.0-94.el8.aarch64
kernel-4.18.0-97.el8.aarch64
kmod-kvdo-6.2.1.48-52.el8.aarch64
vdo-6.2.1.48-10.el8.aarch64
[root@localhost ~]# find /lib/modules | grep vdo
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo/uds
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo/vdo
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo/vdo/kvdo.ko
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/uds
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/vdo
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/vdo/kvdo.ko
[root@localhost ~]# uname -r
4.18.0-97.el8.aarch64
[root@localhost ~]# dmesg -c > /dev/null
[root@localhost ~]# dmesg
[root@localhost ~]# insmod /lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/uds/uds.ko 
insmod: ERROR: could not insert module /lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/uds/uds.ko: Invalid parameters
[root@localhost ~]# dmesg
[  476.014767] uds: loading out-of-tree module taints kernel.
[  476.085777] uds: module verification failed: signature and/or required key missing - tainting kernel
[  476.195511] uds: disagrees about version of symbol cpu_hwcap_keys
[  476.268653] uds: Unknown symbol cpu_hwcap_keys (err -22)
[  476.332425] uds: disagrees about version of symbol cpu_hwcaps
[  476.401394] uds: Unknown symbol cpu_hwcaps (err -22)
Comment 10 Andy Walsh 2019-07-12 15:07:38 UTC 
Just like in Comment#9, kernel-4.18.0-115.el8 causes kmod-kvdo-6.2.1.102-54.el8 to stop working on aarch64.  It does work on ppc64le, s390x, and x86_64, however.  I will be working to rebuild.

[root@localhost ~]# rpm -qa kernel kmod-kvdo vdo lvm2
lvm2-2.03.05-1.el8.aarch64
kernel-4.18.0-107.el8.aarch64
kernel-4.18.0-115.el8.aarch64
kmod-kvdo-6.2.1.102-54.el8.aarch64
vdo-6.2.1.102-11.el8.aarch64
[root@localhost ~]# find /lib/modules | grep vdo
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo/uds
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo/vdo
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo/vdo/kvdo.ko
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/uds
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/vdo
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/vdo/kvdo.ko
[root@localhost ~]# uname -r
4.18.0-115.el8.aarch64
[root@localhost ~]# dmesg -c > /dev/null
[root@localhost ~]# dmesg
[root@localhost ~]# insmod /lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/uds/uds.ko 
insmod: ERROR: could not insert module /lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/uds/uds.ko: Invalid parameters
[root@localhost ~]# dmesg
[  798.628235] uds: loading out-of-tree module taints kernel.
[  798.698658] uds: disagrees about version of symbol cpu_hwcap_keys
[  798.771808] uds: Unknown symbol cpu_hwcap_keys (err -22)
[  798.835561] uds: disagrees about version of symbol cpu_hwcaps
[  798.904525] uds: Unknown symbol cpu_hwcaps (err -22)
Comment 11 Andy Walsh 2019-07-29 18:13:38 UTC 
kernel-4.18.0-123.el8 causes kmod-kvdo-6.2.1.102-55.el8 to stop working on aarch64.  It does appear to work on ppc64le, and x86_64.  It is unknown at this point whether S390x continues to work.

These are the errors when attempting to insmod the uds module on aarch64:
[  580.593230] uds: loading out-of-tree module taints kernel.
[  580.664035] uds: disagrees about version of symbol cpu_hwcap_keys
[  580.737183] uds: Unknown symbol cpu_hwcap_keys (err -22)
[  580.800934] uds: disagrees about version of symbol cpu_hwcaps
[  580.869893] uds: Unknown symbol cpu_hwcaps (err -22)

I have built an updated version of kmod-kvdo that will resolve this issue.  kmod-kvdo-6.2.1.102-56.el8 is being tested to fix the issue noted above.
Comment 16 Andy Walsh 2019-09-13 15:59:25 UTC 
kernel-4.18.0-144.el8 causes kmod-kvdo-6.2.1.138-56.el8 to stop working.

These are the errors when attempting to insmod kvdo:
:: [ 11:39:52 ] :: [  BEGIN   ] :: Running 'modinfo kvdo'
modinfo: ERROR: Module kvdo not found.
:: [ 11:39:52 ] :: [   FAIL   ] :: Command 'modinfo kvdo' (Expected 0, got 1)
:: [ 11:39:52 ] :: [  BEGIN   ] :: Running 'dmesg -c > /dev/null'
:: [ 11:39:52 ] :: [   PASS   ] :: Command 'dmesg -c > /dev/null' (Expected 0, got 0)
:: [ 11:39:52 ] :: [  BEGIN   ] :: Running 'insmod /lib/modules/4.18.0-129.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko'
insmod: ERROR: could not insert module /lib/modules/4.18.0-129.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko: Unknown symbol in module
:: [ 11:39:52 ] :: [   FAIL   ] :: Command 'insmod /lib/modules/4.18.0-129.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko' (Expected 0, got 1)
:: [ 11:39:52 ] :: [  BEGIN   ] :: Running 'dmesg'
[  194.579895] kvdo: Unknown symbol dm_noflush_suspending (err 0)
[  194.579902] kvdo: Unknown symbol dm_put_device (err 0)
[  194.580107] kvdo: Unknown symbol dm_shift_arg (err 0)
[  194.580119] kvdo: Unknown symbol dm_unregister_target (err 0)
[  194.580154] kvdo: Unknown symbol dm_set_target_max_io_len (err 0)
[  194.580213] kvdo: Unknown symbol dm_register_target (err 0)
[  194.580238] kvdo: Unknown symbol dm_table_get_mode (err 0)
[  194.580263] kvdo: Unknown symbol dm_get_device (err 0)
[  194.580279] kvdo: Unknown symbol dm_consume_args (err 0)
:: [ 11:39:52 ] :: [   PASS   ] :: Command 'dmesg' (Expected 0, got 0)

I am building an updated version of kmod-kvdo that will resolve this issue.  kmod-kvdo-6.2.1.138-57.el8 is the new package NVR.
Comment 17 Andy Walsh 2019-09-13 16:47:35 UTC 
I will note that comment#16 is not an unexpected event.  BZ1749929 triggered this one.
Comment 23 Jakub Krysl 2019-10-15 08:19:55 UTC 
# uname -r
4.18.0-147.el8.x86_64
# modprobe kvdo
# echo $?
0
Comment 26 errata-xmlrpc 2020-04-28 16:42:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1782
Note You need to log in before you can comment on or make changes to this bug.