Bug 1373654

Any update on this?

Sarang,

Can you please provide a .srpm or a .spec file of your DUP at least please?

Please note that there is a manual workaround if the DUP driver is needed in the initrd:

Anaconda mounts the final system under /mnt/sysimage, so first we need to change root there:
# chroot /mnt/sysimage

Then we can just load the modules and add them to the inird manually:

# /sbin/modprobe bnx2x
# /sbin/dracut -f --add-driver bnx2x /boot/initramfs-2.6.32-531.el6.x86_64.img
# /sbin/lsinitrd /boot/initramfs-$(uname -r) | grep bnx2x

Hope this helps and regards,
-Stanislav

Hello Stanislav,

I am attaching the SRPM as requested. Please suggest if there is anything that can be done from the DUD.

I'll try out the workaround and let you know the findings.

Created attachment 1200919 [details]
SRPM for corresponding DUD

Hello Stanislav,

We tried out the steps that you mentioned for "bnx2x" driver. Those did not work for "fastlinq qed/qede" drivers.

Do note, the attached DUD is for fastlinq drivers and not Netxtreme2 (bnx2x) drivers.

The bnx2x drivers are inbox. And the modprobe command shows the inbox modules.
In case of qed/qede drivers - which are not inbox - the modprobe command fails to find the module.

However, the modules are correctly loaded from DUD at that instance.

Please suggest if there is anything else that we can try out here.

-Thanks,
Sarang

Sarang,
The problem of why the DUP modules are not present in the initrd after the installation is a problem in Anaconda, that's why I changed the component.
We need reply from Anaconda experts now. Thanks!

We have seen that it is also not part of /lib/modules/kernel in local installation.

Hello HPE,
Due to were we are in the RHEL 6 release cycle. This will most likely not be fixed in RHEL 6.
Please confirm if you would like the information in comment 3 including in a kbase.

Thank You
Joe Kachuck

Joe, HPE do want a kbase.  Please post the kbase in this bz for review once drafted.

Hello,
This is an example of the kbase:
#######
Subject:
Drivers from DUD not loading in initrd

Issue:
When installing with a Red Hat supplied DUD the driver does not get loaded for install.

Environment:
RHEL 6.x

Resolution:
If the DUP is needed during install. This is a manual work around for this issue:
* Begin the install with the DUP driver.
* Switch to a command terminal during the install.
* Change root the installing system:
# chroot /mnt/sysimage
* Then load the modules and add them to the inird manually:
# /sbin/modprobe YOURMODULE
# /sbin/dracut -f --add-driver bnx2x /boot/initramfs-KERNELRELEASE.el6.x86_64.img
# /sbin/lsinitrd /boot/initramfs-$(uname -r) | grep YOURMODULE

######

Stanislav would you be able to take a look. Please let me know if any additional information should be added.

Thank You
Joe Kachuck

(In reply to Joseph Kachuck from comment #24)
> Hello,
> This is an example of the kbase:
> #######
> Subject:
> Drivers from DUD not loading in initrd
> 
> Issue:
> When installing with a Red Hat supplied DUD the driver does not get loaded
> for install.
> 
> Environment:
> RHEL 6.x
> 
> Resolution:
> If the DUP is needed during install. This is a manual work around for this
> issue:
> * Begin the install with the DUP driver.
> * Switch to a command terminal during the install.
> * Change root the installing system:
> # chroot /mnt/sysimage
> * Then load the modules and add them to the inird manually:
> # /sbin/modprobe YOURMODULE
> # /sbin/dracut -f --add-driver bnx2x
> /boot/initramfs-KERNELRELEASE.el6.x86_64.img
> # /sbin/lsinitrd /boot/initramfs-$(uname -r) | grep YOURMODULE
> 
> ######
> 
> Stanislav would you be able to take a look. Please let me know if any
> additional information should be added.
> 
> Thank You
> Joe Kachuck
Hi Joe,
As mentioned the above workaround is not working for fastlinq qed/qede driver. The fastlinq qed/qede driver is not inbox.

> > Resolution:
> > If the DUP is needed during install. This is a manual work around for this
> > issue:
> > * Begin the install with the DUP driver.
> > * Switch to a command terminal during the install.
> > * Change root the installing system:
> > # chroot /mnt/sysimage
> > * Then load the modules and add them to the inird manually:
> > # /sbin/modprobe YOURMODULE
> > # /sbin/dracut -f --add-driver bnx2x
> > /boot/initramfs-KERNELRELEASE.el6.x86_64.img
> > # /sbin/lsinitrd /boot/initramfs-$(uname -r) | grep YOURMODULE

> Hi Joe,
> As mentioned the above workaround is not working for fastlinq qed/qede
> driver. The fastlinq qed/qede driver is not inbox.

Girisha, I don't see why the workaround described above would not work. All the RH built DUDs are always build against the GA kernel version which is also present on the installation disk. Also the installation of the DUD rpm file should call depmod to update the module database. Thus after the installation of the provided DUD .rpm package the modprobe command should load the qed/qede driver.

Can you please share more details of what part of the workaround doesn't work for you? Thanks!

(In reply to Stanislav Kozina from comment #26)
> > > Resolution:
> > > If the DUP is needed during install. This is a manual work around for this
> > > issue:
> > > * Begin the install with the DUP driver.
> > > * Switch to a command terminal during the install.
> > > * Change root the installing system:
> > > # chroot /mnt/sysimage
> > > * Then load the modules and add them to the inird manually:
> > > # /sbin/modprobe YOURMODULE
> > > # /sbin/dracut -f --add-driver bnx2x
> > > /boot/initramfs-KERNELRELEASE.el6.x86_64.img
> > > # /sbin/lsinitrd /boot/initramfs-$(uname -r) | grep YOURMODULE
> 
> > Hi Joe,
> > As mentioned the above workaround is not working for fastlinq qed/qede
> > driver. The fastlinq qed/qede driver is not inbox.
> 
> Girisha, I don't see why the workaround described above would not work. All
> the RH built DUDs are always build against the GA kernel version which is
> also present on the installation disk. Also the installation of the DUD rpm
> file should call depmod to update the module database. Thus after the
> installation of the provided DUD .rpm package the modprobe command should
> load the qed/qede driver.
> 
In case of qed/qede drivers - which are not inbox - the modprobe command fails to find the module.

However, the modules are correctly loaded from DUD at that instance.

> Can you please share more details of what part of the workaround doesn't
> work for you? Thanks!

> In case of qed/qede drivers - which are not inbox - the modprobe command
> fails to find the module.

This is caused by the missing entry for the module in the modprobe database.
Can you please run 'depmod' prior to the call of 'modprobe'?

> However, the modules are correctly loaded from DUD at that instance.

The modules themselves from the DUD are loaded manually using 'insmod' after they are unpacked from the DUD.

Thank you!
-Stanislav

Girisha, 
please answer comment 28.

(In reply to Stanislav Kozina from comment #28)
> > In case of qed/qede drivers - which are not inbox - the modprobe command
> > fails to find the module.
> 
> This is caused by the missing entry for the module in the modprobe database.
> Can you please run 'depmod' prior to the call of 'modprobe'?
> 
> > However, the modules are correctly loaded from DUD at that instance.
> 
> The modules themselves from the DUD are loaded manually using 'insmod' after
> they are unpacked from the DUD.
> 
> Thank you!
> -Stanislav
Hi Stanislav,
I did do the depmod before modprobe but still the modprobe fails to find the qede driver. Please refer the module.jpg

Created attachment 1248879 [details]
snapshot

Girisha,
My apologizes, I have been looking at the RHEL-7 anaconda sources, and I suppose you are hitting this issue with RHEL-6, is that right?

I'm not sure now where the driver modules are unpacked in RHEL-6, I'll have to check. It should be possible to load the required module at least manually using insmod if the qede module is unpacked outside of the standard modprobe path. I'll have more details for you soon after I check. Thank you!

Girisha,

I did build a testing kernel module which is not part of the standard RHEL-6.8 image (ersinmod.ko), packaged that as a DUP and tested if the module is correctly loaded in Anaconda. The module has not been loaded on its own, but 'modprobe' command has been able to load the module.

The modules from the driver disk should be unpacked and installed under /tmp/DD/lib/modules. The main directory /lib/modules/$(uname -r)/updates then contains a symbolic link to the /tmp/DD/lib/modules, and through this path modprobe can load these modules. Of course this works only if the driver modules have been correctly build for the same version of the kernel as is contained on the installation image (2.6.32-642.el6 for the RHEL-6.8 GA).

Can you please verify if your driver disk installs modules under the GA path?
Can you locate the qede.ko module under /tmp/DD/lib/modules and load it using insmod command directly? Thank you!

Hello,
Due to where we are in the RHEL 6.9 release. This will not make RHEL 6.9. This is now requested for RHEL 6.10.

Thank You
Joe Kachuck

Girisha, please see comment 33.

Hello,
RHEL 6 has entered Phase 3. In phase 3 only Critical impact Security Advisories and selected Urgent Priority Bug Fix Advisories will be accepted.
https://access.redhat.com/support/policy/updates/errata

At current this BZ does not meet these requirements. I am closing this BZ as WONTFIX.

Please reopen if this fix is required for RHEL 6. If so please also provide a justification for this fix.

Thank You
Joe Kachuck

Reopening as this is still an issue with 6.9 GA and we have several customers asking for this to be fixed.

Joe, any idea when this might be fixed.

Chad,

I'm not sure we have a clear understanding of the problem yet.
Please see Comment #33 -- has it been identified why the depmod/modprobe commands don't find the modules? That should work fine on RHEL-6.9 (per the comment).

My guess would be that the qed/qede drivers have not been built for the RHEL-6.9 GA kernel, and that's why the depmod doesn't look for them.

Thanks!

(In reply to Stanislav Kozina from comment #41)
> Chad,
> 
> 
> My guess would be that the qed/qede drivers have not been built for the
> RHEL-6.9 GA kernel, and that's why the depmod doesn't look for them.
> 

This is correct.  The drivers being injected are not part of the GA kernel.  Is there a workaround to make sure they're included in the initramfs being built for the newly installed operating system?

Chad,

> > My guess would be that the qed/qede drivers have not been built for the
> > RHEL-6.9 GA kernel, and that's why the depmod doesn't look for them.
> > 
> 
> This is correct.  The drivers being injected are not part of the GA kernel. 
> Is there a workaround to make sure they're included in the initramfs being
> built for the newly installed operating system?

The question is not whether the modules are part of the GA kernel release.
The question is what version of the kernel the external modules have been built for. In other words, what kernel-devel package has been installed for the build of the module?

The modules are installed into the 'extra' directory under the /lib/modules path for the kernel which the modules have been built for. By default depmod looks for modules under the path for the current kernel, that's why I believe that if the modules have been built for the RHEL-6.9 GA, depmod would find them and modprobe would just work. Then, if the modules are loaded, they would also be added to the initramfs environment.

(In reply to Stanislav Kozina from comment #43)
> 
> The question is not whether the modules are part of the GA kernel release.
> The question is what version of the kernel the external modules have been
> built for. In other words, what kernel-devel package has been installed for
> the build of the module?
> 
> The modules are installed into the 'extra' directory under the /lib/modules
> path for the kernel which the modules have been built for. By default depmod
> looks for modules under the path for the current kernel, that's why I
> believe that if the modules have been built for the RHEL-6.9 GA, depmod
> would find them and modprobe would just work. Then, if the modules are
> loaded, they would also be added to the initramfs environment.

Sarang, I presume the modules in question are build against RHEL 6.9 GA?

Chad,

That is correct. The modules are built against GA kernel 2.6.32-696.el6.

Do note, the issue is seen on older RHEL6.x distros as well.

Sarang,

Could you please provide us with the updated DUP image? Thanks!

Created attachment 1297134 [details]
Updated DUD - RHEL6.9

Sarang,

Thank you for kindly providing the testing DUD. I've tested the RHEL-6.9 GA in a VM (that means without the hardware attached) with your DUD and the results are:
 - modprobe command was working fine for the qed, qedi and qedr modules
 - the kmod-qlgc-fastling package has not been installed to the final system
 - because the package was not installed, the modules have not been installed too and thus the initramfs didn't contain the modules
 - when I installed the kmod package manually in the final system, the modules did appear in the initramfs

Please can you confirm that 'modprobe qed' command called from the Anaconda enviroment (called from the console accessible using the CTRL+ALT+F2 keys) loads the qed module correctly for you?

Also, does the kmod-qlgc-fastling package gets installed?

Thank you,
-Stanislav

Looking at the Anaconda rhle-6.9 sources.

The kmod-qlgc-fastlinq-8.22.0.0-99.rhel6u9.x86_64.rpm package (correctly) delivers the modules to the extra directory of the rhel-6.9 GA kernel:

$ rpm -ql -p kmod-qlgc-fastlinq-8.22.0.0-99.rhel6u9.x86_64.rpm  | grep ko$
warning: kmod-qlgc-fastlinq-8.22.0.0-99.rhel6u9.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 1c9c8ff1: NOKEY
/lib/modules/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qed.ko
/lib/modules/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qede.ko
/lib/modules/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qedf.ko
/lib/modules/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qedi.ko
/lib/modules/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qedr.ko

Anaconda unpacks these to the /tmp/DD directory:

loader/driverdisk.h:
#define DD_EXTRACTED "/tmp/DD"
loader/driverdisk.c:
    /* unpack packages from dest into location->path */ 
    if (dlabelUnpackRPMDir(dest, DD_EXTRACTED, kernelver)) {
        /* fatal error, log this and jump to exception handler */
        logMessage(ERROR, "Error unpacking RPMs from driver disc no.%d",
                disknum);
        goto loadDriverDiscException;
    }

As expected, the modules are unpacked under the /tmp/DD directory:

/tmp/DD/lib/modules/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qed.ko

When Anaconda decides which packages need to be installed it runs:

isys/isys.py:
def modulesWithPaths():                                                                                                
    mods = []
    for modline in open("/proc/modules", "r"):                                                                         
        modName = modline.split(" ", 1)[0]
        modInfo = iutil.execWithCapture("modinfo",
                ["-F", "filename", modName]).splitlines()
        modPaths = [ line.strip() for line in modInfo if line!="" ]                                                    
        mods.extend(modPaths)                                                                                          
    return mods 

yuminstall.py:
        #We need to install the packages which contain modules from DriverDiscs
        for modPath in isys.modulesWithPaths():
            log.debug("Checking for DUD module "+modPath)
            match = DD_EXTRACTED.match(modPath)
            if match:
                log.info("Requesting install of kmod-%s" % (match.group("modulename")))
                moduleProvides.append("kmod-"+match.group("modulename"))
            else:
                continue

constants.py:
DD_EXTRACTED = re.compile("/lib/modules/[^/]+/updates/DD/(?P<moduledir>.*/)?(?P<modulename>[^/.]+).ko.*")

So, at the end it compares the DD_EXTRACTED regexp with the path of the qed.ko module:

/tmp/DD/lib/modules/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qed.ko

This doesn't seem to match:

Python 2.6.6 (r266:84292, Aug  9 2016, 06:11:56) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> DD_EXTRACTED = re.compile("/lib/modules/[^/]+/updates/DD/(?P<moduledir>.*/)?(?P<modulename>[^/.]+).ko.*")
>>> match = DD_EXTRACTED.match("/tmp/DD/lib/modules/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qed.ko")
>>> print match
None

Samantha, can the Anaconda team please review this problem?

Hello,

I was wrong in Comment#50, the regexp actually does match because the path used for the matching comes from the modinfo(8) which goes through /lib/modules and the DD symlink:

$ modinfo -F filename qed
/lib/modules/2.6.32-696.el6.x86_64/updates/DD/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qed.ko

>>> import re
>>> DD_EXTRACTED = re.compile("/lib/modules/[^/]+/updates/DD/(?P<moduledir>.*/)?(?P<modulename>[^/.]+).ko.*")
>>> match = DD_EXTRACTED.match("/lib/modules/2.6.32-696.el6.x86_64/updates/DD/2.6.32-696.el6.x86_64/extra/qlgc-fastlinq/qed.ko")
>>> print match
<_sre.SRE_Match object at 0x7f49f53decf0>

However the code in yuminstall.py further unveils what the problem is:

            match = DD_EXTRACTED.match(modPath)
            if match:
                log.info("Requesting install of kmod-%s" % (match.group("modulename")))
                moduleProvides.append("kmod-"+match.group("modulename"))
            else:
                continue

After the match succeeds this tries to install kmod-qed package. This package does not exist.
In other words, Anaconda in RHEL-6 supports only DD with .rpm packages with the same name as the modules they contain.

Can you please rebuild your RPM package as kmod-qed and re-try? Thank you!

any new update on this bug?

Hello Trinh,

Please see Comment#51 -- the DUD loader in Anaconda is limited so it installs only .rpm modules using the same name as the .ko modules they contain.

Is there a chance you could re-package your driver into kmod-qed.rpm package?

Thanks!

Girisha, please answer comment 54?

Girisha said Cavium needs to answer it.

Sarang, can you please answer comment 54?

(In reply to Stanislav Kozina from comment #54)
> Hello Trinh,
> 
> Please see Comment#51 -- the DUD loader in Anaconda is limited so it
> installs only .rpm modules using the same name as the .ko modules they
> contain.
> 
> Is there a chance you could re-package your driver into kmod-qed.rpm package?
> 
> Thanks!

Hi Stan,
We have one more dud (netxtreme2) there also the rpm package name and the module names are different. That one installs successfully and we don't see this issue. The only difference is that the modules are already inbox in this case where as in fastlinq case they are not inbox.

Cavium please answer Stan questions and also correct if I am wrong.

Hi Stan,

Re-naming the package is not possible as fastlinq package includes four other drivers apart from qed.

We can however update our "Provides" section to give out kmod-qed, kmod-qede etc.

We will try that out and let you know how it goes.

Hi Girisha,

I now notice that apart from being inbox, netxtreme2 kmod RPM also provides kmod-bnx2, kmod-cnic, kmod-bnx2x, kmod-bnx2i, kmod-bnx2fc explicitly. That might be a reason it works for netxtreme2 and not for fastlinq.

I wonder why we don't see this problem on RHEL7, may be because RHEL6 and RHEL7 have different code base as Stan mentioned comment 32.

Hello Sarang,

> Re-naming the package is not possible as fastlinq package includes four
> other drivers apart from qed.

That's not an issue -- per Anaconda code, the RPM package is installed if it's name matches *any* of the modules currently loaded in the kernel.
So if the qed.ko module is loaded at the beginning of the installation and the name of the rpm package is kmod-qed, then the rpm package should be installed even if it contains more modules.
 
> We can however update our "Provides" section to give out kmod-qed, kmod-qede
> etc.
> 
> We will try that out and let you know how it goes.

Thank you, this looks like a good step to me.
But please also try to rename the .rpm package to kmod-qed. That should work, but I'm judging just by the reading of the Anaconda code.

> Hi Girisha,
> 
> I now notice that apart from being inbox, netxtreme2 kmod RPM also provides
> kmod-bnx2, kmod-cnic, kmod-bnx2x, kmod-bnx2i, kmod-bnx2fc explicitly. That
> might be a reason it works for netxtreme2 and not for fastlinq.

Agreed.

> I wonder why we don't see this problem on RHEL7, may be because RHEL6 and
> RHEL7 have different code base as Stan mentioned comment 32.

Correct. All the DUD loading code has been completely rewritten between RHEL-6 and RHEL-7 mostly to move as much code as possible from C to Python.

Hi Stan/Girisha,

Our test teams were able to see the drivers now make it to the initrd.

The RPM included within the DUD has been updated to have following "Provides"
kmod-qed
kmod-qede
kmod-qedr
kmod-qedi
kmod-qedf


I am attaching the DUD with this workaround if you like to see for yourself.

Thanks a lot for your help.

Created attachment 1308398 [details]
DUD with "Provides" workaround

I'm glad we've been able to find a workaround, thanks for sharing your updated DUD. I'm closing this bug now, please re-open if you hit any more issues. Thank you!