Bug 2264343 - rocm-smi: assertion failure with builtin AMD GPU
Summary: rocm-smi: assertion failure with builtin AMD GPU
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: rocm-smi
Version: 39
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jeremy Newton
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:083d36f92e0018a7e10ae7d335f...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-02-15 05:23 UTC by Jerry James
Modified: 2025-11-28 18:26 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-11-13 13:42:07 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: proc_pid_status (1.50 KB, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: maps (3.96 KB, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: limits (1.29 KB, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: environ (4.52 KB, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: open_fds (515 bytes, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: mountinfo (3.13 KB, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: os_info (734 bytes, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: cpuinfo (3.16 KB, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: core_backtrace (4.73 KB, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: dso_list (673 bytes, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details
File: backtrace (29.22 KB, text/plain)
2024-02-15 05:23 UTC, Jerry James
no flags Details

Description Jerry James 2024-02-15 05:23:29 UTC
Description of problem:
I had heard how cool btop is, so I just installed it.  The crash occurred on the very first attempt to use it (and every subsequent attempt).

Version-Release number of selected component:
btop-1.3.0-4.fc39

Additional info:
reporter:       libreport-2.17.11
type:           CCpp
reason:         btop killed by SIGABRT
journald_cursor: s=ebf36c8bf5b84d2793553a6245b7538f;i=a50bd;b=ef4aee87c54044cd8c0cc49ac8e91bf7;m=be705e9d4;t=61164c7e4d66a;x=7663e611266cecc1
executable:     /usr/bin/btop
cmdline:        btop
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/app-org.gnome.Terminal.slice/vte-spawn-995e6d7b-f67d-4d18-8953-6361f546445d.scope
rootdir:        /
uid:            1000
kernel:         6.7.4-200.fc39.x86_64
package:        btop-1.3.0-4.fc39
runlevel:       N 5
backtrace_rating: 4
crash_function: get_multiplier_from_str
comment:        I had heard how cool btop is, so I just installed it.  The crash occurred on the very first attempt to use it (and every subsequent attempt).

Truncated backtrace:
Thread no. 1 (5 frames)
 #6 get_multiplier_from_str at /usr/src/debug/rocm-smi-5.7.1-1.fc39.x86_64/src/rocm_smi.cc:121
 #8 freq_string_to_int at /usr/include/c++/13/bits/basic_string.h:222
 #9 get_frequencies at /usr/src/debug/rocm-smi-5.7.1-1.fc39.x86_64/src/rocm_smi.cc:1068
 #10 rsmi_dev_gpu_clk_freq_get at /usr/src/debug/rocm-smi-5.7.1-1.fc39.x86_64/src/rocm_smi.cc:1619
 #11 Gpu::Rsmi::collect<true> at src/linux/btop_collect.cpp:1380

Comment 1 Jerry James 2024-02-15 05:23:33 UTC
Created attachment 2016795 [details]
File: proc_pid_status

Comment 2 Jerry James 2024-02-15 05:23:34 UTC
Created attachment 2016796 [details]
File: maps

Comment 3 Jerry James 2024-02-15 05:23:36 UTC
Created attachment 2016797 [details]
File: limits

Comment 4 Jerry James 2024-02-15 05:23:37 UTC
Created attachment 2016798 [details]
File: environ

Comment 5 Jerry James 2024-02-15 05:23:38 UTC
Created attachment 2016799 [details]
File: open_fds

Comment 6 Jerry James 2024-02-15 05:23:40 UTC
Created attachment 2016800 [details]
File: mountinfo

Comment 7 Jerry James 2024-02-15 05:23:41 UTC
Created attachment 2016801 [details]
File: os_info

Comment 8 Jerry James 2024-02-15 05:23:42 UTC
Created attachment 2016802 [details]
File: cpuinfo

Comment 9 Jerry James 2024-02-15 05:23:43 UTC
Created attachment 2016803 [details]
File: core_backtrace

Comment 10 Jerry James 2024-02-15 05:23:45 UTC
Created attachment 2016804 [details]
File: dso_list

Comment 11 Jerry James 2024-02-15 05:23:47 UTC
Created attachment 2016805 [details]
File: backtrace

Comment 12 Jerry James 2024-02-15 05:24:49 UTC
The backtrace points to rocm-smi as the probable culprit.  I'll try to debug it tomorrow if nobody beats me to it.

Comment 13 Jerry James 2024-02-19 19:23:42 UTC
This is definitely a rocm-smi bug.  Reassigning the component.

In freq_string_to_int, we have:
fs = "   1066Mhz *"
ind = 1066
freq = 0
junk = "Mhz"
units_str = ""
star_str = ""
multiplier = <optimized out>

The empty units_str means that this line:

  long double multiplier = get_multiplier_from_str(units_str[0]);

passes the terminating null character to get_multiplier_from_str, which leads directly to the assertion failure.

Moving up one level to get_frequencies, we have:
val_vec = std::vector of length 3, capacity 4 = {"2: 400Mhz ", "3: 1067Mhz ", "   1066Mhz *"}

So something went awry with reading frequencies for my GPU, which is an Asus AMD Radeon RX 6500 XT.

GDB shows that it first reads from /sys/class/drm/card0/device/pp_dm_sclk, which has these contents:

0: 200Mhz 
1: 400Mhz *
2: 2000Mhz 

Then it reads from /sys/class/drm/card0/device/pp_dpm_mclk, which has these contents:

2: 400Mhz 
3: 1067Mhz 
   1066Mhz *

And that is where its string parsing algorithm breaks down, because the 3rd line does not start with 'n:' for some integer n.

Comment 14 Jerry James 2024-02-19 21:56:19 UTC
My mistake.  That Radeon 6500 XT is card1; card0 is the builtin GPU on an AMD Ryzen 7 5700G.

Comment 15 Tom Rix 2024-02-26 19:22:50 UTC
does 

> cat /sys/class/drm/card0/device/pp_dpm_mclk

Have the same offset index / missing index ?

If sysfs is misformatted, isn't this a kernel issue ?

Comment 16 Jerry James 2024-02-26 20:21:00 UTC
(In reply to Tom Rix from comment #15)
> does 
> 
> > cat /sys/class/drm/card0/device/pp_dpm_mclk
> 
> Have the same offset index / missing index ?

Yes:

$ cat /sys/class/drm/card0/device/pp_dpm_mclk
2: 400Mhz 
3: 1067Mhz 
   1066Mhz *

> If sysfs is misformatted, isn't this a kernel issue ?

Is it a sysfs formatting bug, or a failure of rocm-smi to parse the output correctly?  I took a look in the Linux 6.7.6 source tree and found places where this format is printed in:
- vangogh_print_legacy_clk_levels (drivers/gpu/drm/amd/pm/swsmu/smu11/vangogh_ppt.c): the !cur_value_match_level case at the bottom
- vangogh_print_clk_levels (drivers/gpu/drm/amd/pm/swsmu/smu11/vangogh_ppt.c): the !cur_value_match_level case near the bottom
- renoir_print_clk_levels (drivers/gpu/drm/amd/pm/swsmu/smu12/renoir_ppt.c): the !cur_value_match_level case at the bottom

Comment 17 Tom Rix 2024-03-16 12:22:13 UTC
Misformatted sysfs is a kernel issue.

Comment 18 Jerry James 2024-03-16 22:33:44 UTC
If you think that the sysfs output is misformatted, by all means reassign this bug to the kernel component.

To me it looks like the kernel developers decided to represent the case where the current frequency is not the same as any clock level, and rocm-smi does not correctly parse that case.  This format has been around for at least 3 years; e.g., see https://github.com/torvalds/linux/commit/ae7b32e7c30268196e1b3c61e0dcb50b4e842f39; search for "if (!cur_value_match_level)".  It shows up in several places under drivers/gpu/drm/amd/pm/swsmu in the Linux kernel source tree.

Comment 19 Aoife Moloney 2024-11-13 12:03:44 UTC
This message is a reminder that Fedora Linux 39 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 39 on 2024-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '39'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 39 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 20 Russell Odom 2025-11-28 18:26:58 UTC
This still occurs in F43 - bug 2417737 created


Note You need to log in before you can comment on or make changes to this bug.