Bug 1411540 - New and updated drivers fail to install on the initrd image
Summary: New and updated drivers fail to install on the initrd image
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dracut
Version: 7.2
Hardware: x86_64
OS: Linux
Target Milestone: rc
: ---
Assignee: Lukáš Nykrýn
QA Contact: Release Test Team
Depends On:
Blocks: 1465901
TreeView+ depends on / blocked
Reported: 2017-01-09 23:15 UTC by Matthias Idemundia
Modified: 2021-02-15 10:02 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-01-12 18:22:10 UTC
Target Upstream Version:

Attachments (Terms of Use)
The module-setup.sh script (557 bytes, application/x-shellscript)
2017-01-09 23:15 UTC, Matthias Idemundia
no flags Details

Description Matthias Idemundia 2017-01-09 23:15:52 UTC
Created attachment 1238924 [details]
The module-setup.sh script

Description of problem:
This looks like a long-standing issue in the driver installation and initrd rebuild process.  The RHEL initrd image does not include the /lib/modules/<kernel>/extra/ directory by default.  Because of this, when a new driver is installed in this directory in order to support a boot device, boot will fail.  It looks like this issue may have been missed up until now because when updating an existing driver(as opposed to installing a new one), the driver included in the initrd image is able to pick up the boot device, preventing the boot process from failing even though the updated driver isn't actually being used.

--Workaround 1:  Use dracut's add_drivers and omit_drivers directive to add the driver to initrd when it is installed and remove it when it is uninstalled.  This causes dracut to generate errors at the prompt whenever ANY driver update or module parameter update is performed that initiates an initrd rebuild.

"Failed to install module XXXXXXXX".

*Not really an acceptable solution*

--Workaround 2:  Write a module-setup.sh script with the check(), depends(), install(), and installkernel() routines.  Include this script in the driver rpm.  I tried this and unfortunately, the updated driver still does not get installed on the ramdisk image.

Version-Release number of selected component (if applicable):  033-359.el7

How reproducible: 100%

Steps to Reproduce:
1.  Install or update any driver
2.  Confirm that it is present in the main file system's "/lib/modules/<kernel>/extra/" directory
3.  Use lsinitrd to confirm that it is not present in the initial ramdisk image

Actual results:

Expected results:

Additional info:
Though our inbox lpfc drivers for RHEL will continue to support both OneConnect and LightPulse adapters but for business reasons our Broadcom management wants the ability to independently ship the two different divisional/productlines on their own schedules (kind of like getting a divorce at the product level). 

So our out-of-box driver has been split.  The OOB lpfc driver now supports LighPulse only adapters while the new brcmfcoe driver supports OneConnect only adapters.  

We tested a SAN boot configuration with RHEL7.2 where the inbox lpfc driver was being used to boot from a SAN disk via a CCX adapter.  We then installed the brcmfcoe driver rpm(which blacklists inbox lpfc).  We found that the brcmfcoe driver does not get installed in the initrd image.  Only on the main file system.  If the user reboots, the system will not come back up.  

With lpfc being blacklisted and with brcmfcoe not being present in initrd, there is no driver to attach to the OneConnect adapters.

Comment 2 Matthias Idemundia 2017-01-11 18:08:38 UTC
In "Workaround 1" we use add_drivers in /etc/dracut.conf to add the driver.  This is done in the kmodtool file.  We do this along with a "dracut --force" command to force a rebuild of the initrd image.

Comment 3 Matthias Idemundia 2017-01-11 18:11:34 UTC
To be clear, "Workaround 1" does update the initrd image however it generates the error messages associated with our driver that I mentioned.

Comment 4 Laurie Barry 2017-01-11 20:00:43 UTC
Andre/Dracut team,

We're are forced to move forward with deploying workaround 1 which will result in customer visible Dracut errors logged to the command line prompt and the console every single time any initrd change is made by the user whether it is related to BRCFCOE drive or not!

This is likely result in Customer Support Calls to both our company's unless some other solution can be identified by RH.  This is very unfortunate, please help to prevent these Customer Support Calls.

thank you

Comment 5 Harald Hoyer 2017-01-12 15:04:49 UTC
$ man depmod.d

       search subdirectory...
           This allows you to specify the order in which /lib/modules (or
           other configured module location) subdirectories will be
           processed by depmod. 

       override modulename kernelversion modulesubdirectory
           This command allows you to override which version of a
           specific module will be used when more than one module sharing
           the same name is processed by the depmod command. It is
           possible to specify one kernel or all kernels using the *
           wildcard.  modulesubdirectory is the name of the subdirectory
           under /lib/modules (or other module location) where the target
           module is installed.

Comment 6 Matthias Idemundia 2017-01-13 03:49:35 UTC
Greetings, Harold,

I apologize if I am missing something but, how does this help install a new driver to initrd?  I see how this can be helpful in that we can use depmod to make sure that lpfc loads before brcmfcoe or vice versa.  I.E. and alternative to blacklisting.  However, as far as I can tell, it does not address the issue of cleanly(without errors) installing the drivers onto initrd and rebuilding initrd during the driver's rpm installation.

-For example, is there a flag that can be set in kmodtool?
-Is there a directive that can be added to the spec file?

As I mentioned earlier, using dracut's "add_driver" does not work cleanly since it produces failure messages.

Comment 7 Matthias Idemundia 2017-01-18 19:01:21 UTC
Please provide any further feedback relating to cleanly installing the driver onto initrd.  As mentioned, using add_driver in kmodtool produces side effects when the driver is uninstalled or upgraded.

Comment 9 Harald Hoyer 2018-02-13 11:51:50 UTC
I don't understand the problem. Works for me without any error messages. Did you forget "depmnod <kernel-version>" in the rpm install script?

# rpm -qf /etc/depmod.d/dist.conf 

# cat /etc/depmod.d/dist.conf 
# depmod.conf

# override default search ordering for kmod packaging
search updates extra built-in weak-updates

# ls -l /lib/modules/$(uname -r)/extra/hello.ko
-rw-r--r--. 1 root root 104448 13. Feb 12:01 /lib/modules/3.10.0-850.el7.x86_64/extra/hello.ko

# depmod -a

# grep hello /lib/modules/$(uname -r)/modules.dep

# cat /etc/dracut.conf.d/hello.conf 
add_drivers+=" hello "

# dracut --force test.img

# lsinitrd test.img /usr/lib/modules/3.10.0-850.el7.x86_64/modules.dep | grep hello

# lsinitrd test.img | grep hello.ko
-rw-r--r--   1 root     root         4840 Feb 13 12:49 usr/lib/modules/3.10.0-850.el7.x86_64/extra/hello.ko

Comment 10 Harald Hoyer 2018-02-13 11:52:34 UTC
also note the spaces in add_drivers

Note You need to log in before you can comment on or make changes to this bug.