Lukas,
this looks simalr to the third party issue we once had, please can you look into it?

Not sure if this is useful...

I think I tracked the affected symbols down to sysfs_create_bin_file and sysfs_remove_bin_file, but when I looked at the actual source code for these symbols in the -257 kernel, I could see that they'd changed from that in the RHEL8.3 kernel (but I'm not a kernel developer)

To be clear, I'm not expecting this to be something Red Hat can (or should) fix, or it to be something that shouldn't break, just trying to highlight that by releasing a constant Stream of development kernels to CentOS Stream means that the kernel in Stream is now no longer compatible with those kernels released to RHEL at the same point in time.

A solution would be to make regular RHEL kernels also available in Stream, either on an opt-in or opt-out basis, so those users for whom kernel ABI compatibility is critical can continue to contribute to, and benefit from, CentOS Stream.

I have other examples of similar breakage if you need more examples (kmod-aacraid and kmod-qla2xxx within the storage space) but I've not traced specific symbols for these.

(In reply to Phil Perry from comment #3)
> Not sure if this is useful...

Thanks, I have missed the part with changes in kernel symbols.

> I think I tracked the affected symbols down to sysfs_create_bin_file and
> sysfs_remove_bin_file, but when I looked at the actual source code for these
> symbols in the -257 kernel, I could see that they'd changed from that in the
> RHEL8.3 kernel (but I'm not a kernel developer)

Cestmir,
I don't see sysfs_create_bin_file and sysfs_remove_bin_file in KABI protected list but also have difficulties to find when we changed these symbols. Please look at the symbols used by 3w-sas, aacraid, qla2xxx between RHEL8.3 and kernel-257, (actually the first one breaking a symbol is sufficient).

 
> To be clear, I'm not expecting this to be something Red Hat can (or should)
> fix, or it to be something that shouldn't break, just trying to highlight
> that by releasing a constant Stream of development kernels to CentOS Stream
> means that the kernel in Stream is now no longer compatible with those
> kernels released to RHEL at the same point in time.
> 
> A solution would be to make regular RHEL kernels also available in Stream,
> either on an opt-in or opt-out basis, so those users for whom kernel ABI
> compatibility is critical can continue to contribute to, and benefit from,
> CentOS Stream.
>
> I have other examples of similar breakage if you need more examples
> (kmod-aacraid and kmod-qla2xxx within the storage space) but I've not traced
> specific symbols for these.

Hi Tomas,

Correct, sysfs_create_bin_file and sysfs_remove_bin_file are not on the kABI whitelist.

The process I used to identify those two symbols was to cross reference those symbols listed in /usr/share/doc/kmod-3w-sas-3.26.02.000/greylist.txt against a diff of symbols that changed between kernel-4.18.0-240.el8.x86_64 and kernel-4.18.0-257.el8.x86_64 from the output of 'rpm -q --provides' for each kernel (some 3595 of them).

Hi Phil,
I know that the KABI is a problem, how it looks from my angle you'll need to detect changes in kernel for modules you maintan and rebuild them I only hope that it doesn't happen often.

I'll reassign the bz to CentOs technical lead so you can discuss the other option - 'make regular RHEL kernels also available in Stream'.

Hi Brian,

CentOs stream suffers from KABI changes see comment#3 (part of CentOs are I think drivers we do not build but CentOs does)

"To be clear, I'm not expecting this to be something Red Hat can (or should) fix, or it to be something that shouldn't break, just trying to highlight that by releasing a constant Stream of development kernels to CentOS Stream means that the kernel in Stream is now no longer compatible with those kernels released to RHEL at the same point in time.

A solution would be to make regular RHEL kernels also available in Stream, either on an opt-in or opt-out basis, so those users for whom kernel ABI compatibility is critical can continue to contribute to, and benefit from, CentOS Stream."

I don't know how to help further so I'm reassigning the bz.

(In reply to Tomas Henzl from comment #9)
> Hi Phil,
> I know that the KABI is a problem, how it looks from my angle you'll need to
> detect changes in kernel for modules you maintan and rebuild them I only
> hope that it doesn't happen often.
> 
> I'll reassign the bz to CentOs technical lead so you can discuss the other
> option - 'make regular RHEL kernels also available in Stream'.

Thanks Thomas,

For the first Stream kernel update (kernel-4.18.0-257.el8.x86_64.rpm), 13 out of 44 packages I maintain were broken as a result of the update, so unfortunately it's a significant number, and not something we are able to maintain (for Stream) going forward, hence the request for Stream to provide an in situ solution.

This bug report was once opened to the public but then made private again. I do not see anything related to security / private stuff. Can someone explain?

(In reply to Akemi Yagi from comment #12)
> This bug report was once opened to the public but then made private again. I
> do not see anything related to security / private stuff. Can someone explain?

There are many layers of Bugzilla automation especially when dealing with the kernel. We're working on a longer-term solution to this problem.

Thanks, Brian, for making the change so quickly.

I want to quote quite an extensive analysis on the kABI of CentOS Stream kernels, given in a recent presentation by Pat Riehecky, "Tracking kernel rate of change", on the CentOS Dojo, 2022.

https://wiki.centos.org/Events/Dojo/FOSDEM2022?action=AttachFile&do=view&target=kernel+rate+of+change.pdf

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.