Bug 2401433

Summary: crash 8.0.6-4.fc42 is broken on kernel v6.14+ (Fixed upstream in v9.0.0)
Product: [Fedora] Fedora Reporter: Anderson Nascimento <anderson>
Component: crashAssignee: lijiang
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 42CC: anderson, lijiang, ltao
Target Milestone: ---Keywords: Upgrades
Target Release: ---Flags: fedora-admin-xmlrpc: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: crash-9.0.0-5.fc42 Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-10-31 01:59:43 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Anderson Nascimento 2025-10-04 08:55:44 UTC 
The crash package, version 8.0.6-4.fc42, available in Fedora 42, fails to execute commands like vtop and files on the default kernel v6.16.x due to an outdated structure member offset. This issue is fixed in upstream crash version 9.0.0. The Fedora package should be updated to a version that includes these fixes to maintain functionality with recent kernels.

Description of the issue

The currently packaged version of the crash utility in Fedora 42 is incompatible with the default kernel v6.16.7-200.fc42.x86_64. Commands that rely on internal kernel filesystem structure information, such as vtop and files, fail with an invalid structure member offset error.

The underlying cause is a kernel change (specifically, commit <58cf9c383c5c> in v6.14-rc1) that renamed the dentry structure member from d_iname to d_shortname. The older crash version is unaware of this change.

Environment

Fedora Release: 42

Kernel: 6.16.7-200.fc42.x86_64

Package: crash-8.0.6-4.fc42

$ uname -a
Linux fedoraserver42research 6.16.7-200.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 11 17:46:54 UTC 2025 x86_64 GNU/Linux
$ sudo crash vmlinux 
[sudo] password for user: 

crash 8.0.6-4.fc42
Copyright (C) 2002-2025  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

      KERNEL: vmlinux  [TAINTED]
    DUMPFILE: /proc/kcore
        CPUS: 4
        DATE: Wed Dec 31 21:00:00 -03 1969
      UPTIME: 4 days, 07:46:34
LOAD AVERAGE: 0.28, 0.23, 0.20
       TASKS: 351
    NODENAME: fedoraserver42research
     RELEASE: 6.16.7-200.fc42.x86_64
     VERSION: #1 SMP PREEMPT_DYNAMIC Thu Sep 11 17:46:54 UTC 2025
     MACHINE: x86_64  (1991 Mhz)
      MEMORY: 2 GB
         PID: 38187
     COMMAND: "crash"
        TASK: ffff888015dc8040  [THREAD_INFO: ffff888015dc8040]
         CPU: 2
       STATE: TASK_RUNNING (ACTIVE)

crash> vtop -c 12423 0x7fbe8716e000
VIRTUAL     PHYSICAL        
7fbe8716e000  (not mapped)

   PGD: 607d47f8 => 24669067
   PUD: 246697d0 => 5d20d067
   PMD: 5d20d1c0 => 38a09067
   PTE: 38a09b70 => 0

      VMA           START       END     FLAGS FILE

vtop: invalid structure member offset: dentry_d_iname
      FILE: filesys.c  LINE: 3243  FUNCTION: get_pathname_component()

BFD: /usr/sbin/crash: unknown type [0x13] section `.relr.dyn'
[/usr/sbin/crash] error trace: 556f6af6bd14 => 556f6af6ad62 => 556f6afb027f => 556f6b090a27
BFD: /usr/sbin/crash: unknown type [0x13] section `.relr.dyn'

vtop: invalid structure member offset: dentry_d_iname
      FILE: filesys.c  LINE: 3243  FUNCTION: get_pathname_component()

crash> files 12423
PID: 12423    TASK: ffff888012e92b00  CPU: 2    COMMAND: "map"

files: invalid structure member offset: dentry_d_iname
       FILE: filesys.c  LINE: 3243  FUNCTION: get_pathname_component()

BFD: /usr/sbin/crash: unknown type [0x13] section `.relr.dyn'
[/usr/sbin/crash] error trace: 556f6afb957f => 556f6afb74c2 => 556f6afb027f => 556f6b090a27
BFD: /usr/sbin/crash: unknown type [0x13] section `.relr.dyn'

files: invalid structure member offset: dentry_d_iname
       FILE: filesys.c  LINE: 3243  FUNCTION: get_pathname_component()

crash> 

This exact issue is acknowledged and fixed upstream in crash version 9.0.0.

commit 080b4baf5d5e998750f80289c847f162bb81a043
Author: Tao Liu <ltao>
Date:   Mon Feb 24 15:59:13 2025 +1300

    Fix the failing of cmd "runq -g" for v6.14-rc1 kernel
    
    Kernel commit <7b8a702d9438> ("sched/fair: Rename h_nr_running
    into h_nr_queued") renamed the member "h_nr_running" of struct
    cfs_rq into "h_nr_queued".
    
    Without the patch:
      crash> runq -g
      ...
      runq: invalid structure member offset: cfs_rq_nr_running
            FILE: task.c  LINE: 9480  FUNCTION: dump_tasks_in_lower_dequeued_cfs_rq()
    
      [./crash] error trace: 580934 => 580090 => 5f1f31 => 5f1eb3
    
        5f1eb3: OFFSET_verify.part.40+51
        5f1f31: OFFSET_verify+49
        580090: dump_tasks_in_lower_dequeued_cfs_rq+444
        580934: dump_tasks_in_task_group_cfs_rq+1229
    
      runq: invalid structure member offset: cfs_rq_nr_running
            FILE: task.c  LINE: 9480  FUNCTION: dump_tasks_in_lower_dequeued_cfs_rq()
    
    Signed-off-by: Tao Liu <ltao>
    Reported-by: Anderson Nascimento <andersonc0d3>

commit 2795136a515446b798ebbfa257c97f0ca6ecb8ec
Author: Tao Liu <ltao>
Date:   Mon Feb 24 15:59:12 2025 +1300

    Fix the failing of cmd "files" for v6.14-rc1 kernel
    
    Kernel commit <58cf9c383c5c> ("dcache: back inline names with
    a struct-wrapped array of unsigned long") renamed "unsigned char
    d_iname" into "union shortname_store d_shortname", the same change
    should be integrated into crash.
    
    Without the patch:
    
      crash> files
      files: invalid structure member offset: dentry_d_iname
             FILE: filesys.c  LINE: 3243  FUNCTION: get_pathname_component()
    
      [./crash] error trace: 55e532 => 55ea57 => 5f1f31 => 5f1eb3
      PID: 1490     TASK: ffff8dd00c4c2900  CPU: 0    COMMAND: "bash"
    
        5f1eb3: OFFSET_verify.part.40+51
        5f1f31: OFFSET_verify+49
        55ea57: get_pathname_component+69
        55e532: get_pathname+636
    
      files: invalid structure member offset: dentry_d_iname
             FILE: filesys.c  LINE: 3243  FUNCTION: get_pathname_component()
    
    Signed-off-by: Tao Liu <ltao>
    Reported-by: Anderson Nascimento <andersonc0d3>

URL: https://crash-utility.github.io/changelog/ChangeLog-9.0.0.txt

Please update the crash package for Fedora 42 to version 9.0.0 or backport the commits that incorporate the necessary structure member offset fixes to ensure functionality with modern Fedora kernels (v6.14 and later). Without this update, the crash utility is currently unusable for crucial debugging tasks on an updated Fedora 42 system.

Reproducible: Always

Steps to Reproduce:
1.Start crash, `crash vmlinux`
2.Execute the command `vtop -c PID ADDR` or `files -c PID`.
Actual Results:
crash> vtop -c 12423 0x7fbe8716e000
VIRTUAL     PHYSICAL        
7fbe8716e000  (not mapped)

   PGD: 607d47f8 => 24669067
   PUD: 246697d0 => 5d20d067
   PMD: 5d20d1c0 => 38a09067
   PTE: 38a09b70 => 0

      VMA           START       END     FLAGS FILE

vtop: invalid structure member offset: dentry_d_iname
      FILE: filesys.c  LINE: 3243  FUNCTION: get_pathname_component()

BFD: /usr/sbin/crash: unknown type [0x13] section `.relr.dyn'
[/usr/sbin/crash] error trace: 556f6af6bd14 => 556f6af6ad62 => 556f6afb027f => 556f6b090a27
BFD: /usr/sbin/crash: unknown type [0x13] section `.relr.dyn'

vtop: invalid structure member offset: dentry_d_iname
      FILE: filesys.c  LINE: 3243  FUNCTION: get_pathname_component()

crash>
Comment 1 lijiang 2025-10-09 11:36:51 UTC 
This is expected behavior, because the crash-utility is very old version, which can not work on the latest kernel.

Can you try the latest crash tool in Fedora with the following command?

sudo dnf upgrade --refresh --advisory=FEDORA-2025-29645419c2

In addition, I also filed a new Jira issue to track such cases, and to see if that can be updated automatically.
https://issues.redhat.com/browse/FC-2479

Thanks.
Comment 2 Anderson Nascimento 2025-10-11 13:38:25 UTC 
(In reply to lijiang from comment #1)
> This is expected behavior, because the crash-utility is very old version,
> which can not work on the latest kernel.

Understood. Thank you.

> 
> Can you try the latest crash tool in Fedora with the following command?
> 
> sudo dnf upgrade --refresh --advisory=FEDORA-2025-29645419c2

I receive the following when executing that command:

No advisory found matching the requested name: "FEDORA-2025-29645419c2"

> 
> In addition, I also filed a new Jira issue to track such cases, and to see
> if that can be updated automatically.
> https://issues.redhat.com/browse/FC-2479
> 
> Thanks.
Comment 3 Fedora Update System 2025-10-22 09:27:31 UTC 
FEDORA-2025-21b24d0a02 (crash-9.0.0-5.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-21b24d0a02
Comment 4 lijiang 2025-10-22 09:31:40 UTC 
> > Can you try the latest crash tool in Fedora with the following command?
> > 
> > sudo dnf upgrade --refresh --advisory=FEDORA-2025-29645419c2
> 
> I receive the following when executing that command:
> 
> No advisory found matching the requested name: "FEDORA-2025-29645419c2"
> 
Thank you for trying this, It's strange that the advisory is not found.

Anyway I have updated there, you can use the crash-9.0.0-5.fc42.

Thanks.
Comment 5 Fedora Update System 2025-10-23 02:31:08 UTC 
FEDORA-2025-21b24d0a02 has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-21b24d0a02`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-21b24d0a02

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 6 Fedora Update System 2025-10-31 01:59:43 UTC 
FEDORA-2025-21b24d0a02 (crash-9.0.0-5.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 7 Anderson Nascimento 2025-11-18 19:44:00 UTC 
(In reply to lijiang from comment #4)
> > > Can you try the latest crash tool in Fedora with the following command?
> > > 
> > > sudo dnf upgrade --refresh --advisory=FEDORA-2025-29645419c2
> > 
> > I receive the following when executing that command:
> > 
> > No advisory found matching the requested name: "FEDORA-2025-29645419c2"
> > 
> Thank you for trying this, It's strange that the advisory is not found.
> 
> Anyway I have updated there, you can use the crash-9.0.0-5.fc42.
> 
> Thanks.

Sorry for the delay. Thank you!