Bug 1695330

Summary: Latest kmod-kvdo does not work with latest RHEL-8.1.0 kernel
Product: Red Hat Enterprise Linux 8 Reporter: Andy Walsh <awalsh>
Component: kmod-kvdoAssignee: Andy Walsh <awalsh>
Status: CLOSED ERRATA QA Contact: vdo-qe
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.1CC: awalsh, mmarusak, toneata, wchadwic
Target Milestone: rcKeywords: ZStream
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kmod-kvdo-6.2.1.138-57.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1760741 (view as bug list) Environment:
Last Closed: 2020-04-28 16:42:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1760741    

Description Andy Walsh 2019-04-02 20:25:27 UTC
Description of problem:
The latest kmod-kvdo package does not work against the latest RHEL-8.1.0 kernel due to kABI compatibility problem.

Version-Release number of selected component (if applicable):
kernel-4.18.0-80.6.el8
kmod-kvdo-6.2.0.293-50.el8

How reproducible:
100%

Steps to Reproduce:
1. Install a kernel starting at 4.18.0-80.6 or greater
2. Reboot into the newly installed kernel
3. Install the latest kmod-kvdo (version 6.2.0.293-50)
4. Attempt to insmod/modprobe the kernel modules
5. Notice failure in dmesg output.

Actual results:
[root@host-8-242-17 ~]# insmod /lib/modules/4.18.0-72.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko 
insmod: ERROR: could not insert module kvdo.ko: Unknown symbol in module
[root@host-8-241-181 ~]# dmesg
[  254.096707] kvdo: Unknown symbol dm_put_device (err 0)
[  254.096843] kvdo: Unknown symbol dm_shift_arg (err 0)
[  254.096851] kvdo: Unknown symbol dm_unregister_target (err 0)
[  254.096880] kvdo: Unknown symbol dm_set_target_max_io_len (err 0)
[  254.096919] kvdo: Unknown symbol dm_register_target (err 0)
[  254.096937] kvdo: Unknown symbol dm_table_get_mode (err 0)
[  254.096952] kvdo: Unknown symbol dm_get_device (err 0)
[  254.096963] kvdo: Unknown symbol dm_consume_args (err 0)

Expected results:
[root@host-8-241-181 ~]# modprobe kvdo
[root@host-8-241-181 ~]# dmesg
[  168.906841] device-mapper: uevent: version 1.0.3
[  168.934245] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel
[  168.937978] uds: loading out-of-tree module taints kernel.
[  168.942918] uds: modprobe: loaded version 6.2.0.77
[  168.971569] kvdo: modprobe: loaded version 6.2.0.293


Additional info:

Full output of trying to load the module:
[root@host-8-242-17 ~]# uname -r
4.18.0-80.6.el8.x86_64
[root@host-8-242-17 ~]# rpm -qa kmod-kvdo
kmod-kvdo-6.2.0.293-50.el8.x86_64
[root@host-8-242-17 ~]# dmesg -c > /dev/null
[root@host-8-242-17 ~]# insmod /lib/modules/4.18.0-72.el8.x86_64/extra/kmod-kvdo/uds/uds.ko 
[root@host-8-242-17 ~]# dmesg
[  240.050464] uds: loading out-of-tree module taints kernel.
[  240.056809] uds: insmod: loaded version 6.2.0.77
[root@host-8-242-17 ~]# insmod /lib/modules/4.18.0-72.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko 
insmod: ERROR: could not insert module kvdo.ko: Unknown symbol in module
[root@host-8-242-17 ~]# dmesg
[  240.050464] uds: loading out-of-tree module taints kernel.
[  240.056809] uds: insmod: loaded version 6.2.0.77
[  254.096707] kvdo: Unknown symbol dm_put_device (err 0)
[  254.096843] kvdo: Unknown symbol dm_shift_arg (err 0)
[  254.096851] kvdo: Unknown symbol dm_unregister_target (err 0)
[  254.096880] kvdo: Unknown symbol dm_set_target_max_io_len (err 0)
[  254.096919] kvdo: Unknown symbol dm_register_target (err 0)
[  254.096937] kvdo: Unknown symbol dm_table_get_mode (err 0)
[  254.096952] kvdo: Unknown symbol dm_get_device (err 0)
[  254.096963] kvdo: Unknown symbol dm_consume_args (err 0)

Comment 1 Andy Walsh 2019-04-10 12:21:01 UTC
*** Bug 1698299 has been marked as a duplicate of this bug. ***

Comment 2 ChanghuiZhong 2019-05-05 08:41:15 UTC
Hello,

The same problem occurs in 4.18.0-81.el8.x86_64

4.18.0-81.el8.x86_64
vdo-6.2.0.293-10.el8.x86_64
kmod-kvdo-6.2.0.293-50.el8.x86_64

[root@storageqe-26 vdo]# modprobe kvdo.ko 
modprobe: FATAL: Module kvdo.ko not found in directory /lib/modules/4.18.0-81.el8.x86_64
[root@storageqe-26 vdo]# 

[root@storageqe-26 vdo]# insmod kvdo.ko 
insmod: ERROR: could not insert module kvdo.ko: Unknown symbol in module
[root@storageqe-26 vdo]# 

[ 2713.940929] kvdo: disagrees about version of symbol dm_register_target
[ 2713.969905] kvdo: Unknown symbol dm_register_target (err -22)
[ 2713.996331] kvdo: Unknown symbol getBytesFromBuffer (err 0)
[ 2714.021460] kvdo: Unknown symbol contentLength (err 0)
[ 2714.044165] kvdo: Unknown symbol nowUsec (err 0)
[ 2714.064884] kvdo: Unknown symbol recordBioAlloc (err 0)
[ 2714.088587] kvdo: Unknown symbol registerAllocatingThread (err 0)
[ 2714.116317] kvdo: Unknown symbol hasSameBytes (err 0)
[ 2714.141149] kvdo: Unknown symbol udsCloseIndexSession (err 0)
[ 2714.169614] kvdo: Unknown symbol allocateMemory (err 0)
[ 2714.193048] kvdo: Unknown symbol udsInitializeConfiguration (err 0)
[ 2714.221428] kvdo: Unknown symbol allocSprintf (err 0)
[ 2714.244431] kvdo: Unknown symbol copyBytes (err 0)
[ 2714.266087] kvdo: disagrees about version of symbol dm_get_device
[ 2714.293471] kvdo: Unknown symbol dm_get_device (err -22)
[ 2714.317292] kvdo: Unknown symbol currentTime (err 0)
[ 2714.339674] kvdo: Unknown symbol resetBufferEnd (err 0)
[ 2714.363263] kvdo: Unknown symbol getUInt32LEFromBuffer (err 0)

Comment 4 Andy Walsh 2019-05-05 21:41:55 UTC
I have just posted an updated version that should work with 4.18.0-81 and newer.  Updated NVR is kmod-kvdo-6.2.0.293-51.el8.

Comment 5 ChanghuiZhong 2019-05-06 03:32:07 UTC
Hi,Andy

Confirm that kmod-kvdo-6.2.0.293-51.el8 work well with 4.18.0-81.el8,thanks for your help

[root@storageqe-26 vdo]# modprobe kvdo
[root@storageqe-26 vdo]# 

[ 3234.499452] uds: modprobe: loaded version 6.2.0.77
[ 3234.521266] kvdo: modprobe: loaded version 6.2.0.293


Thanks
Changhui

Comment 7 ChanghuiZhong 2019-05-22 02:48:44 UTC
Hello,

The same problem occurs in 4.18.0-92.el8.x86_64.


vdo-6.2.1.48-10.el8.x86_64
kmod-kvdo-6.2.1.48-51.el8.x86_64

Creating VDO vPV1
STDERR:vdo: ERROR - Kernel module kvdo not installed
vdo: ERROR - modprobe: FATAL: Module kvdo not found in directory /lib/modules/4.18.0-92.el8.x86_64
RETURN:1

Comment 8 Andy Walsh 2019-05-22 13:33:19 UTC
(In reply to ChanghuiZhong from comment #7)
> Hello,
> 
> The same problem occurs in 4.18.0-92.el8.x86_64.
> 
> 
> vdo-6.2.1.48-10.el8.x86_64
> kmod-kvdo-6.2.1.48-51.el8.x86_64
> 
> Creating VDO vPV1
> STDERR:vdo: ERROR - Kernel module kvdo not installed
> vdo: ERROR - modprobe: FATAL: Module kvdo not found in directory
> /lib/modules/4.18.0-92.el8.x86_64
> RETURN:1

I built kmod-kvdo-6.2.1.48-52.el8 yesterday to fix this problem.

Comment 9 Andy Walsh 2019-05-30 13:26:54 UTC
It looks like kernel-4.18.0-97.el8 causes kmod-kvdo-6.2.1.48-52.el8 to stop working on aarch64.  I have found that it does work on x86_64 in my testing, but tests are still running for ppc64le and s390x, so I'm not sure about those.  One failure is enough for me to rebuild, though.

[root@localhost ~]# rpm -qa kernel kmod-kvdo vdo lvm2
lvm2-2.03.02-6.el8.aarch64
kernel-4.18.0-94.el8.aarch64
kernel-4.18.0-97.el8.aarch64
kmod-kvdo-6.2.1.48-52.el8.aarch64
vdo-6.2.1.48-10.el8.aarch64
[root@localhost ~]# find /lib/modules | grep vdo
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo/uds
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo/vdo
/lib/modules/4.18.0-94.el8.aarch64/weak-updates/kmod-kvdo/vdo/kvdo.ko
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/uds
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/vdo
/lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/vdo/kvdo.ko
[root@localhost ~]# uname -r
4.18.0-97.el8.aarch64
[root@localhost ~]# dmesg -c > /dev/null
[root@localhost ~]# dmesg
[root@localhost ~]# insmod /lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/uds/uds.ko 
insmod: ERROR: could not insert module /lib/modules/4.18.0-92.el8.aarch64/extra/kmod-kvdo/uds/uds.ko: Invalid parameters
[root@localhost ~]# dmesg
[  476.014767] uds: loading out-of-tree module taints kernel.
[  476.085777] uds: module verification failed: signature and/or required key missing - tainting kernel
[  476.195511] uds: disagrees about version of symbol cpu_hwcap_keys
[  476.268653] uds: Unknown symbol cpu_hwcap_keys (err -22)
[  476.332425] uds: disagrees about version of symbol cpu_hwcaps
[  476.401394] uds: Unknown symbol cpu_hwcaps (err -22)

Comment 10 Andy Walsh 2019-07-12 15:07:38 UTC
Just like in Comment#9, kernel-4.18.0-115.el8 causes kmod-kvdo-6.2.1.102-54.el8 to stop working on aarch64.  It does work on ppc64le, s390x, and x86_64, however.  I will be working to rebuild.

[root@localhost ~]# rpm -qa kernel kmod-kvdo vdo lvm2
lvm2-2.03.05-1.el8.aarch64
kernel-4.18.0-107.el8.aarch64
kernel-4.18.0-115.el8.aarch64
kmod-kvdo-6.2.1.102-54.el8.aarch64
vdo-6.2.1.102-11.el8.aarch64
[root@localhost ~]# find /lib/modules | grep vdo
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo/uds
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo/vdo
/lib/modules/4.18.0-107.el8.aarch64/weak-updates/kmod-kvdo/vdo/kvdo.ko
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/uds
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/vdo
/lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/vdo/kvdo.ko
[root@localhost ~]# uname -r
4.18.0-115.el8.aarch64
[root@localhost ~]# dmesg -c > /dev/null
[root@localhost ~]# dmesg
[root@localhost ~]# insmod /lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/uds/uds.ko 
insmod: ERROR: could not insert module /lib/modules/4.18.0-109.el8.aarch64/extra/kmod-kvdo/uds/uds.ko: Invalid parameters
[root@localhost ~]# dmesg
[  798.628235] uds: loading out-of-tree module taints kernel.
[  798.698658] uds: disagrees about version of symbol cpu_hwcap_keys
[  798.771808] uds: Unknown symbol cpu_hwcap_keys (err -22)
[  798.835561] uds: disagrees about version of symbol cpu_hwcaps
[  798.904525] uds: Unknown symbol cpu_hwcaps (err -22)

Comment 11 Andy Walsh 2019-07-29 18:13:38 UTC
kernel-4.18.0-123.el8 causes kmod-kvdo-6.2.1.102-55.el8 to stop working on aarch64.  It does appear to work on ppc64le, and x86_64.  It is unknown at this point whether S390x continues to work.

These are the errors when attempting to insmod the uds module on aarch64:
[  580.593230] uds: loading out-of-tree module taints kernel.
[  580.664035] uds: disagrees about version of symbol cpu_hwcap_keys
[  580.737183] uds: Unknown symbol cpu_hwcap_keys (err -22)
[  580.800934] uds: disagrees about version of symbol cpu_hwcaps
[  580.869893] uds: Unknown symbol cpu_hwcaps (err -22)

I have built an updated version of kmod-kvdo that will resolve this issue.  kmod-kvdo-6.2.1.102-56.el8 is being tested to fix the issue noted above.

Comment 16 Andy Walsh 2019-09-13 15:59:25 UTC
kernel-4.18.0-144.el8 causes kmod-kvdo-6.2.1.138-56.el8 to stop working.

These are the errors when attempting to insmod kvdo:
:: [ 11:39:52 ] :: [  BEGIN   ] :: Running 'modinfo kvdo'
modinfo: ERROR: Module kvdo not found.
:: [ 11:39:52 ] :: [   FAIL   ] :: Command 'modinfo kvdo' (Expected 0, got 1)
:: [ 11:39:52 ] :: [  BEGIN   ] :: Running 'dmesg -c > /dev/null'
:: [ 11:39:52 ] :: [   PASS   ] :: Command 'dmesg -c > /dev/null' (Expected 0, got 0)
:: [ 11:39:52 ] :: [  BEGIN   ] :: Running 'insmod /lib/modules/4.18.0-129.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko'
insmod: ERROR: could not insert module /lib/modules/4.18.0-129.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko: Unknown symbol in module
:: [ 11:39:52 ] :: [   FAIL   ] :: Command 'insmod /lib/modules/4.18.0-129.el8.x86_64/extra/kmod-kvdo/vdo/kvdo.ko' (Expected 0, got 1)
:: [ 11:39:52 ] :: [  BEGIN   ] :: Running 'dmesg'
[  194.579895] kvdo: Unknown symbol dm_noflush_suspending (err 0)
[  194.579902] kvdo: Unknown symbol dm_put_device (err 0)
[  194.580107] kvdo: Unknown symbol dm_shift_arg (err 0)
[  194.580119] kvdo: Unknown symbol dm_unregister_target (err 0)
[  194.580154] kvdo: Unknown symbol dm_set_target_max_io_len (err 0)
[  194.580213] kvdo: Unknown symbol dm_register_target (err 0)
[  194.580238] kvdo: Unknown symbol dm_table_get_mode (err 0)
[  194.580263] kvdo: Unknown symbol dm_get_device (err 0)
[  194.580279] kvdo: Unknown symbol dm_consume_args (err 0)
:: [ 11:39:52 ] :: [   PASS   ] :: Command 'dmesg' (Expected 0, got 0)

I am building an updated version of kmod-kvdo that will resolve this issue.  kmod-kvdo-6.2.1.138-57.el8 is the new package NVR.

Comment 17 Andy Walsh 2019-09-13 16:47:35 UTC
I will note that comment#16 is not an unexpected event.  BZ1749929 triggered this one.

Comment 23 Jakub Krysl 2019-10-15 08:19:55 UTC
# uname -r
4.18.0-147.el8.x86_64
# modprobe kvdo
# echo $?
0

Comment 26 errata-xmlrpc 2020-04-28 16:42:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1782