Bug 480136

Summary: mod: cannot find or load object file for dm_region_hash module
Product: Red Hat Enterprise Linux 5 Reporter: Qian Cai <qcai>
Component: crashAssignee: Dave Anderson <anderson>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.3CC: alex.boudhou
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The "mod -[sS]" command would fail with the error message: "mod: cannot find or load object file for <name> module" if the target module object filename contains both underscore and dash characters. Crash now parses these filenames correctly.
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 09:40:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Qian Cai 2009-01-15 11:11:53 UTC
Description of problem:
# crash vmlinux vmcore
...
      KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux
    DUMPFILE: /var/crash/127.0.0.1-2009-01-15-04:50:09/vmcore  [PARTIAL DUMP]
        CPUS: 2
        DATE: Thu Jan 15 04:50:00 2009
      UPTIME: 00:08:26
LOAD AVERAGE: 0.37, 0.14, 0.04
       TASKS: 98
    NODENAME: dell-pesc430-02.rhts.bos.redhat.com
     RELEASE: 2.6.18-128.el5
     VERSION: #1 SMP Wed Dec 17 11:41:38 EST 2008
     MACHINE: x86_64  (2793 Mhz)
      MEMORY: 1 GB
       PANIC: "SysRq : Trigger a crashdump"
         PID: 2744
     COMMAND: "bash"
        TASK: ffff810037fe07e0  [THREAD_INFO: ffff81002f9b6000]
         CPU: 0
       STATE: TASK_RUNNING (SYSRQ)

crash> mod -S
ffffffff88007e80  ehci_hcd          65741  /lib/modules/2.6.18-128.el5/kernel/drivers/usb/host/ehci-hcd.ko 
ffffffff88017800  ohci_hcd          55925  /lib/modules/2.6.18-128.el5/kernel/drivers/usb/host/ohci-hcd.ko 
ffffffff88026e00  uhci_hcd          57433  /lib/modules/2.6.18-128.el5/kernel/drivers/usb/host/uhci-hcd.ko 
ffffffff8803fd80  jbd               94257  /lib/modules/2.6.18-128.el5/kernel/fs/jbd/jbd.ko 
ffffffff8806ae00  ext3             168017  /lib/modules/2.6.18-128.el5/kernel/fs/ext3/ext3.ko 
ffffffff8809ca80  scsi_mod         196569  /lib/modules/2.6.18-128.el5/kernel/drivers/scsi/scsi_mod.ko 
ffffffff880aba00  sd_mod            56385  /lib/modules/2.6.18-128.el5/kernel/drivers/scsi/sd_mod.ko 
ffffffff880dfc80  libata           208721  /lib/modules/2.6.18-128.el5/kernel/drivers/ata/libata.ko 
ffffffff880eec00  ata_piix          56901  /lib/modules/2.6.18-128.el5/kernel/drivers/ata/ata_piix.ko 
ffffffff88107d80  dm_mod           100369  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-mod.ko 
ffffffff88114d00  dm_log            44865  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-log.ko 
ffffffff88122d00  dm_mirror         53065  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-mirror.ko 
ffffffff8812c780  dm_zero           35265  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-zero.ko 
ffffffff8813a680  dm_snapshot       51465  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-snapshot.ko 
ffffffff88145600  dm_mem_cache      38977  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-mem-cache.ko 
mod: cannot find or load object file for dm_region_hash module
ffffffff8815cb00  dm_message        36161  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-message.ko 
ffffffff88176080  dm_raid45         99025  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-raid45.ko 
ffffffff88180b80  pcspkr            36289  /lib/modules/2.6.18-128.el5/kernel/drivers/input/misc/pcspkr.ko 
...

I had not seen this before during 5.3 testing, because it looks like dmraid had not installed proper during kickstart at that time.

...
Installing dmraid-1.0.0.rc13-19.el5.x86_64
error: %post(dmraid-1.0.0.rc13-19.el5.x86_64) scriptlet failed, exit status 1
...

Then, there was no information from dmesg.
....
SCSI device sda: drive cache: write back
 sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
...

In the latest great 5.3 installation tree, dmraid has been installed properly, and dm-raid45 has been loaded by default.

# rpm -q dmraid
dmraid-1.0.0.rc13-33.el5

# dmesg
...
SCSI device sda: drive cache: write back
 sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel
device-mapper: dm-raid45: initialized v0.2429
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
...

# lsmod
...
dm_raid45              99025  0 
dm_message             36161  1 dm_raid45
dm_region_hash         46145  1 dm_raid45
dm_mem_cache           38977  1 dm_raid45
dm_snapshot            51465  0 
dm_zero                35265  0 
dm_mirror              53065  0 
dm_log                 44865  3 dm_raid45,dm_region_hash,dm_mirror
dm_mod                100369  11 dm_multipath,dm_raid45,dm_snapshot,dm_zero,dm_mirror,dm_log
...

Version-Release number of selected component (if applicable):
crash-4.0-7.2.3
kernel-2.6.18-128.el5
kexec-tools-1.102pre-56.el5
RHEL5.3-Server-20090106.0 installation tree

How reproducible:
always

Steps to Reproduce:
1. save a VMCore.
2. crash vmlinux vmcore
3. crash> mod -S

Comment 2 Dave Anderson 2009-01-15 16:20:40 UTC
The crash utility needs to take the "name" of the module as
seen by "lsmod", and then search the local /lib/modules/<release>
directory tree for the associated module.

Sometimes modules are named with strings containing underscores ('_'),
whereas their associated object file uses dashes ('-').

So in kernel version 2.6.18-92.el5, "lsmod" and the crash utility
"mod" command will show dm_crypt, dm_emc, dm_hp_sw, dm_mirror, dm_mod,
dm_multipath, dm_rdac, dm_round_robin, dm_stapshot, and dm_zero.
When searching the local filesystem, the crash utility will first
look for a module with the same name, and if can't find it, it will
try again substituting "_" for "-", and so it finds them here:  

  $ rpm -qpl kernel-2.6.18-92.el5.x86_64.rpm | grep dm-
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-crypt.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-emc.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-hp-sw.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-mirror.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-mod.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-multipath.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-rdac.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-round-robin.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-snapshot.ko
  /lib/modules/2.6.18-92.el5/kernel/drivers/md/dm-zero.ko
  $

So when crash looks for the module object file for "dm_crypt",
it first searches for "dm_crypt.ko", and when it fails, it then
tries for "dm-crypt.ko" and finds it OK.

This later set of modules does not follow convention, specifically
the "dm_region_hash" module.  The crash utility first looks for 
"dm_region_hash.ko", and doesn't find it; so then it replaces the
underscores with dashes, and looks for "dm-region-hash.ko" module.
But it doesn't find that either:

  mod: cannot find or load object file for dm_region_hash module

And that's because the driver writer did not follow convention
for that particular module:

  $ rpm -qpl kernel-2.6.18-128.el5.x86_64.rpm | grep dm-
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-crypt.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-emc.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-hp-sw.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-log.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-mem-cache.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-message.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-mirror.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-mod.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-multipath.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-raid45.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-rdac.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-region_hash.ko  <<<<<<
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-round-robin.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-snapshot.ko
  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-zero.ko
  $

Note the module name uses both underscores and dashes, i.e.,
"dm-region_hash.ko". 

So the crash utility cannot find it automatically using its current
search algorithm.

It's certainly easy enough to work around:

  crash> mod -s dm_region_hash /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-region_hash.ko
       MODULE       NAME               SIZE  OBJECT FILE
  ffffffff88152200  dm_region_hash    46145  /lib/modules/2.6.18-128.el5/kernel/drivers/md/dm-region_hash.ko 
  crash> 

But I note that the current set of 128.el5 modules for x86_64
has these modules containing both '_' and '-' characters:

  aes-x86_64.ko
  dm-region_hash.ko
  nls_euc-jp.ko
  nls_iso8859-1.ko
  nls_iso8859-13.ko
  nls_iso8859-14.ko
  nls_iso8859-15.ko
  nls_iso8859-2.ko
  nls_iso8859-3.ko
  nls_iso8859-4.ko
  nls_iso8859-5.ko
  nls_iso8859-6.ko
  nls_iso8859-7.ko
  nls_iso8859-9.ko
  nls_koi8-r.ko
  nls_koi8-ru.ko
  nls_koi8-u.ko

And so for example, if I insmod aes-x86_64.ko, the module is named
"aes_x86_64":

  $ insmod /lib/modules/2.6.18-92.el5/kernel/arch/x86_64/crypto/aes-x86_64.ko
  $ lsmod
  Module                  Size  Used by
  aes_x86_64             58601  0 
  ...

And not surprisingly, if try to load it on the live system with crash:

  crash> mod -S
  ...
  mod: cannot find or load object file for aes_x86_64 module
  ...

So let me revisit the crash utility's search scheme.

Good catch!

Thanks,
  Dave

Comment 6 Ruediger Landmann 2009-05-18 07:00:37 UTC
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
The "mod -[sS]" command would fail with the error message: "mod: cannot find or load object file for <name> module" if the target module object filename contains both underscore and dash characters. Crash now parses these filenames correctly.

Comment 7 Dave Anderson 2009-05-18 15:03:11 UTC
Release note looks fine.

Comment 9 errata-xmlrpc 2009-09-02 09:40:29 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1283.html