Bug 1443217

Hello Gary,

> Description of problem:
> 
> NCSU can't use "out of tree" kernel modules such as the openafs file system
> built in using DKMS because /usr/sbin/weak-modules incorrectly assumes every
> thing is a pre-compiled "kmod" and it incorrectly makes symlinks to
> /usr/lib/modules/*/weak-updates/openafs.ko on every kernel update.

RHEL doesn't support dkms, in fact dkms is not even included in the RHEL release. Can you please provide more information about which version of dkms and from which source did you use to build the modules?

We understand that the using the dkms might be important for your use case, can you please open a standard support ticket for RHEL and mention this bug in the ticket?

> Install openafs rpms from our third party yum repository at:
> http://install.linux.ncsu.edu/pub/yum/itecs/public/openafs/7/noarch/openafs-
> release-1.0-1.noarch.rpm
> 
> Or build yourself from sources here:
> 
> https://github.com/gsgatlin/openafs-rpms

I see that you did also build the -kmod version of the openafs modules. I see that your .spec file is based on the old RHEL-6 template which might not work correctly on RHEL-7.
Recently we've published a new tool for packaging kmods for RHEL:
https://github.com/orosp/ddiskit

Do you think you could use this tool to package the openafs modules?

> For more info on why this problem even exits please see:
> 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/
> thread/S7HILMMIKOIIWFAZLCXT6YV3Q6KPIOPB/

Thanks for pointing out this thread. It nicely describes one of the differencies between Fedora and RHEL. Since Fedora doesn't have stable kernel interface the dkms is the only option for 3rd party modules. RHEL with its stable kernel ABI is in different position and kmods are the main target.

> So I would like to suggest that broken scripts either be removed or fixed to
> work with DKMS system since Linux has no stable ABI...
> 
> http://www.kroah.com/log/linux/stable_api_nonsense.html

RHEL does have stable kernel ABI.

> We have to use openafs where I work for at least a while. Moving off openafs
> is not an option for us at this time. We have no interest in trying to
> maintain a pre-compiled binary kmod every few weeks or months. We must use
> DKMS until we switch to auristorFS kmods which auristor INC. will be
> compiling for us.

I'm surprised that on RHEL support channel the openafs modules would require recompilation every few weeks. Can you please provide list of the kernel symbols used by the openafs modules and which kernel releases changed one of these symbols?

Thank you and Best Regards,
-Stanislav

It broke when kernel 2.6.32-431.el6.x86_64 came out and again when the kernel with 6.8 came out. (I don't remember the version) It also broke when the kernel in 7.3 came out upgraded from 7.2. 

The dkms package comes from EPEL. Should that be removed from EPEL?

Why does every other distro support DKMS except for RHEL and its clones? on debian openafs-dkms works as expected. Same on Arch. Same on gentoo.

I will open a support ticket as soon as some ex employees of yours show me how on thursday afternoon.

Thank you very much.

Let me see if I can narrow this down better. I'll try some other kernel modules before I open a standard support ticket for RHEL. Thanks.

(In reply to Gary Gatling from comment #8)
> Let me see if I can narrow this down better. I'll try some other kernel
> modules before I open a standard support ticket for RHEL. Thanks.

Thanks Gary!
We have done some testing with the latest RHEL-7 releases and did not manage to trigger the problem. The best description we have so far is:

"Eventually after enough kernel updates happen all the symlinks are broken and the whole thing is ruined until you un-install and re-install again."

It would be very helpful to find a more reliable reproducer.

If you open the support ticket please make sure to mention this bug there. Thank you!

Hello. Sorry for the delay in getting back to you.

I set up a RHEL 7.2 box and updated through 17 kernels I downloaded with 3 different modules and the bug appears to be fixed now. I guess either some package got updated or perhaps things went south on some customers machines due to too small of a /boot partition. (tested openafs, bbswitch, and xpad)

weak-modules is making symlinks sometimes but it doesn't seem to hurt anything and the symlinks don't break with my latest tests. So I'm sorry for the trouble. Thanks for your patience. It is odd that the script is only called in RHEL kernel package and not fedora kernel package. Thanks.

You can close this ticket as NOTABUG thanks.

Gary,

No worries, thank you very much for following up on this issue, really appreciated.

> I set up a RHEL 7.2 box and updated through 17 kernels I downloaded with 3
> different modules and the bug appears to be fixed now. I guess either some
> package got updated or perhaps things went south on some customers machines
> due to too small of a /boot partition. (tested openafs, bbswitch, and xpad)

We recently did significant update to the weak-modules scripts, which is part of the kmod package.

> weak-modules is making symlinks sometimes

The 'sometimes' here means that script creates symlinks for (and only for) the modules which are compatible with the given kernel. That's it's purpose.

> but it doesn't seem to hurt
> anything and the symlinks don't break with my latest tests.

Right, that's how it should work. Thanks for confirmation.

> So I'm sorry for
> the trouble. Thanks for your patience. It is odd that the script is only
> called in RHEL kernel package and not fedora kernel package. Thanks.

Thank you very much for opening this bug and working on the thorough testing. The script is called only for RHEL because Fedora doesn't have any stable kernel interface.

Thanks!