Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1907968

Summary: kmod-kvdo does not work with the latest CentOS 8 Stream kernel
Product: Red Hat Enterprise Linux 8 Reporter: Martin Pitt <mpitt>
Component: kmod-kvdoAssignee: Andy Walsh <awalsh>
Status: CLOSED CURRENTRELEASE QA Contact: vdo-qe
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: awalsh, bigpanther, bstinson, carl, christopher.roskelley, cwei, jeremey.wise, jwboyer
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-12 15:58:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Pitt 2020-12-15 15:32:48 UTC
This bug was initially created as a copy of Bug #1894978

I am copying this bug because: This affects curent CentOS 8 stream as well [1]. 

kmod-kvdo                                    x86_64  6.2.3.91-73.el8

vdo: ERROR - modprobe: FATAL: Module kvdo not found in directory /lib/modules/4.18.0-257.el8.x86_64


[1] https://github.com/cockpit-project/bots/pull/1465



Description of problem:
The latest available build of kmod-kvdo depends on kernel-4.18.0-187.el8, but that is not available in the 8.3.0 environment.  Additionally, this Bug will be used for any kABI compatibility issues that arise through the 8.3.0 release.

Version-Release number of selected component (if applicable):
kmod-kvdo-6.2.4.26-74.el8
kernel-4.18.0-242.el8

How reproducible:
100%

Steps to Reproduce:
1. Install rhel-8.4.0
2. Try to install kmod-kvdo

Actual results:
:: [ 04:00:23 ] :: [  BEGIN   ] :: Running 'insmod /lib/modules/4.18.0-240.10.el8.aarch64/extra/kmod-kvdo/uds/uds.ko'
insmod: ERROR: could not insert module /lib/modules/4.18.0-240.10.el8.aarch64/extra/kmod-kvdo/uds/uds.ko: Unknown symbol in module
:: [ 04:00:23 ] :: [   FAIL   ] :: Command 'insmod /lib/modules/4.18.0-240.10.el8.aarch64/extra/kmod-kvdo/uds/uds.ko' (Expected 0, got 1)
:: [ 04:00:23 ] :: [  BEGIN   ] :: Running 'dmesg'
[  246.063396] uds: Unknown symbol dm_bufio_prefetch (err 0)
[  246.152076] uds: Unknown symbol dm_bufio_mark_buffer_dirty (err 0)
[  246.153421] uds: Unknown symbol dm_bufio_read (err 0)
[  246.154564] uds: Unknown symbol dm_bufio_client_destroy (err 0)
[  246.155903] uds: Unknown symbol dm_bufio_get_block_data (err 0)
[  246.157185] uds: Unknown symbol dm_bufio_client_create (err 0)
[  246.158491] uds: Unknown symbol dm_bufio_release (err 0)
[  246.159878] uds: Unknown symbol dm_bufio_write_dirty_buffers (err 0)
[  246.161269] uds: Unknown symbol dm_bufio_new (err 0)
[  246.162334] uds: Unknown symbol dm_bufio_set_sector_offset (err 0)

Expected results:
uds and kvdo modules load via modprobe.

Additional info:
This is the 8.4 version of 1812069.

Comment 1 Andy Walsh 2020-12-16 03:39:19 UTC
I am going to look into why the current version is not posted.  This generally gets addressed by 1894978.  kmod-kvdo-6.2.4.26-75.el8 should be in place on CentOS Stream for kernel-4.18.0-242.el8 and newer.

Comment 3 redbug888 2020-12-20 18:51:23 UTC
Got a similar issue on by upgrading Centos 8.2 to 8.3 - did report it on -->

https://bugs.centos.org/view.php?id=17928#c38067


0017928: ERROR - modprobe: FATAL: Module kvdo not found
Description	Choose (start, stop, restart, try-restart, reload, force-reload, status) : status
● vdo.service - VDO volume services
   Loaded: loaded (/usr/lib/systemd/system/vdo.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-12-09 22:48:20 CET; 33min ago
  Process: 1394 ExecStart=/usr/bin/vdo start --all --confFile /etc/vdoconf.yml (code=exited, status=1/FAILURE)
 Main PID: 1394 (code=exited, status=1/FAILURE)

Comment 4 Andy Walsh 2021-01-04 19:51:36 UTC
*** Bug 1911493 has been marked as a duplicate of this bug. ***

Comment 5 penguinpages 2021-01-04 21:50:36 UTC
 Per note that I am / was missing uuid module.

# Missing module for UUID for VDO to see drive
[root@thor ~]# yum install libuuid-devel
<snip>                                                                                                                                                                  1/1
Installed products updated.

Installed:
  libuuid-devel-2.32.1-24.el8.x86_64

Complete!
[root@thor ~]# grubby --default-kernel
/boot/vmlinuz-4.18.0-240.1.1.el8_3.x86_64
[root@thor ~]# grubby --set-default /boot/vmlinuz-4.18.0-2
vmlinuz-4.18.0-240.1.1.el8_3.x86_64  vmlinuz-4.18.0-259.el8.x86_64
[root@thor ~]# grubby --set-default /boot/vmlinuz-4.18.0-259.el8.x86_64
The default is /boot/loader/entries/3a42d0819f2b47b386f5aa18b553e506-4.18.0-259.el8.x86_64.conf with index 0 and kernel /boot/vmlinuz-4.18.0-259.el8.x86_64
[root@thor ~]# reboot
[root@thor ~]#
Last login: Sun Jan  3 23:31:45 2021 from 172.16.100.138
[root@thor ~]# uname -a
Linux thor.penguinpages.local 4.18.0-259.el8.x86_64 #1 SMP Mon Dec 21 21:05:36 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@thor ~]# systemctl status vdo
● vdo.service - VDO volume services
   Loaded: loaded (/usr/lib/systemd/system/vdo.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2021-01-04 16:45:32 EST; 1min 20s ago
  Process: 1431 ExecStart=/usr/bin/vdo start --all --confFile /etc/vdoconf.yml (code=exited, status=1/FAILURE)
 Main PID: 1431 (code=exited, status=1/FAILURE)

Jan 04 16:45:30 thor.penguinpages.local vdo[1431]: ERROR - Kernel module kvdo not installed
Jan 04 16:45:30 thor.penguinpages.local vdo[1431]: vdo: ERROR - Kernel module kvdo not installed
Jan 04 16:45:30 thor.penguinpages.local vdo[1431]: ERROR - Kernel module kvdo not installed
Jan 04 16:45:30 thor.penguinpages.local vdo[1431]: vdo: ERROR - modprobe: FATAL: Module kvdo not found in directory /lib/modules/4.18.0-259.el8.x86_64
Jan 04 16:45:30 thor.penguinpages.local vdo[1431]: ERROR - modprobe: FATAL: Module kvdo not found in directory /lib/modules/4.18.0-259.el8.x86_64
Jan 04 16:45:30 thor.penguinpages.local vdo[1431]: Starting VDO vdo_2926
Jan 04 16:45:30 thor.penguinpages.local vdo[1431]: Starting VDO vdo_sdc
Jan 04 16:45:32 thor.penguinpages.local systemd[1]: vdo.service: Main process exited, code=exited, status=1/FAILURE
Jan 04 16:45:32 thor.penguinpages.local systemd[1]: vdo.service: Failed with result 'exit-code'.
Jan 04 16:45:32 thor.penguinpages.local systemd[1]: Failed to start VDO volume services.
[root@thor ~]#

# Looked at other bug posting https://bugzilla.redhat.com/show_bug.cgi?id=1907968
[root@thor ~]# yum install kmod-kvdo
Last metadata expiration check: 4:15:08 ago on Mon 04 Jan 2021 12:33:28 PM EST.
Package kmod-kvdo-6.2.3.91-73.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@thor ~]# yum update -y
Last metadata expiration check: 4:15:54 ago on Mon 04 Jan 2021 12:33:28 PM EST.
Dependencies resolved.
Nothing to do.
Complete!
[root@thor ~]#


Not sure what else I am missing here.  I guess I will go back to 

[root@thor ~]#  grubby --set-default /boot/vmlinuz-4.18.0-240.1.1.el8_3.x86_64
The default is /boot/loader/entries/3a42d0819f2b47b386f5aa18b553e506-4.18.0-240.1.1.el8_3.x86_64.conf with index 1 and kernel /boot/vmlinuz-4.18.0-240.1.1.el8_3.x86_64
[root@thor ~]# reboot

Let me know if you need any more details on this bug.

Comment 6 Andy Walsh 2021-01-04 21:58:01 UTC
Thanks.  The libuuid-devel should have allowed you to build again, based on your comment in bz1911493.

The kmod-kvdo/kernel package mismatch issue is still present.  I am working on getting in touch with the others involved with the kernel and CentOS Stream to get the update issue figured out.  I will post an update when I have one.

Comment 7 Christopherr 2021-01-08 22:51:20 UTC
I can confirm the kmod-kvdo/kernel package mismatch issue is still present.

I am trying out stream for the first time, and the install is new enough that I have never had a working VDO on this system.  Is there a workaround that I can apply while we wait for official fix?

I'm guessing if I install the correct kernel and then reboot into that I should be able to finish setting up my server VDO volume?  Hopefully?

dnf install kernel-4.18.0-XXX.el8.x86_64

If you think that command would let me resume my server VDO setup, do you know what kernel version I should install to work with the kmod-kvdo-6.2.3.91-73.el8.x86_64 that I have installed?

Thanks!
Chris

Comment 8 redbug888 2021-01-08 22:59:14 UTC
I did install this workaround & VDO seems to work again.

i'm using this kvdo Modules --> 

Module Info    :  kmod-kvdo
----------------------------
/etc/depmod.d/kvdo.conf
/lib/modules/4.18.0-240.1.1.el8_3.x86_64
/lib/modules/4.18.0-240.1.1.el8_3.x86_64/extra
/lib/modules/4.18.0-240.1.1.el8_3.x86_64/extra/kmod-kvdo
/lib/modules/4.18.0-240.1.1.el8_3.x86_64/extra/kmod-kvdo/uds
/lib/modules/4.18.0-240.1.1.el8_3.x86_64/extra/kmod-kvdo/uds/uds.ko
/lib/modules/4.18.0-240.1.1.el8_3.x86_64/extra/kmod-kvdo/vdo
/lib/modules/4.18.0-240.1.1.el8_3.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
/usr/lib/.build-id
/usr/lib/.build-id/52
/usr/lib/.build-id/52/0f38f25eed33c7acf18738a00afec0b4ff2399
/usr/lib/.build-id/a2
/usr/lib/.build-id/a2/2843e7da0a554ed1066f29046b435de3c5ef8d
/usr/share/doc/kmod-kvdo/greylist.txt


and i built this packages (did manually downlad them from --> http://ftp.halifax.rwth-aachen.de/puias/8.3/x86_64/os/BaseOS/Packages/


Version Info   : *vdo*
----------------------------
vdo-support-6.2.3.114-14.el8.x86_64
vdo-6.2.3.114-14.el8.x86_64
kmod-kvdo-6.2.3.114-74.el8.x86_64

Comment 9 Andy Walsh 2021-01-11 21:09:22 UTC
I apologize for the delay on this.

An updated build of kmod-kvdo is now posted on the 8-stream mirror.  The updated NVR is kmod-kvdo-6.2.4.26-76.el8, which works with kernel-4.18.0-259.el8.

You can see that reflected here: http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/

Comment 10 redbug888 2021-01-11 21:35:50 UTC
Thanks Andy

Comment 11 penguinpages 2021-01-11 21:58:42 UTC
Yes.. Thanks Andy and team for hard work.

Most people underestimate VDO .. but if your in trenches with data..  it is critical to ROI of storage and HCI stacks in particular.

Comment 12 Andy Walsh 2021-01-12 15:58:29 UTC
Thanks for the confirmation and feedback!

I am closing this ticket out.  Please feel free to open a new one if you encounter further issues.