RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1906482 - loading same extention twice with two different names segfaults crash
Summary: loading same extention twice with two different names segfaults crash
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: crash
Version: 8.2
Hardware: All
OS: Linux
high
medium
Target Milestone: rc
: 8.0
Assignee: ltao
QA Contact: xiaoying yan
Sujata Kurup
URL:
Whiteboard:
Depends On:
Blocks: 84KernelDebugPostPlanning
TreeView+ depends on / blocked
 
Reported: 2020-12-10 16:00 UTC by John Pittman
Modified: 2023-05-10 09:01 UTC (History)
7 users (show)

Fixed In Version: crash-7.3.0-2.el8
Doc Type: Known Issue
Doc Text:
.Reloading an identical crash extension may cause segmentation faults When you load a copy of an already loaded crash extension file, it might trigger a segmentation fault. Currently, the crash utility detects if an original file has been loaded. Consequently, due to two identical files co-existing in the crash utility, a namespace collision occurs, which triggers the crash utility to cause a segmentation fault. You can work around the problem by loading the crash extension file only once. As a result, segmentation faults no longer occur in the described scenario.
Clone Of:
Environment:
Last Closed: 2021-11-09 18:08:37 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:4199 0 None None None 2021-11-09 18:08:55 UTC

Description John Pittman 2020-12-10 16:00:59 UTC 
Description of problem:

loading same extension twice with two different names segfaults crash

Version-Release number of selected component (if applicable):

crash-7.2.7-3.el8

How reproducible:

Every time

Steps to Reproduce:

1. Load .so extension
2. cp .so extension giving new name
3. Load .so w/ new name

Actual results:

crash segfaults

Expected results:

crash should not segfault

Additional info:

crash> extend /cores/crashext/mpykdumpx86_64.so
Setting scroll off while initializing PyKdump
/cores/crashext/mpykdumpx86_64.so: shared object loaded

# cp /cores/crashext/mpykdumpx86_64.so /cores/crashext/mpykdumpx86_64_2.so

crash> extend /cores/crashext/mpykdumpx86_64_2.so
extend: /cores/crashext/mpykdumpx86_64_2.so: "epython" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "xportshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "crashinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "taskinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "nfsshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "hanginfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "fregs" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "tslog" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "scsi" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "scsishow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "dmshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "pstree" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "modinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "mdadm" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "nvme" is a duplicate of a currently-existing command
Segmentation fault (core dumped)
[user@host ~]$ 

==========================================

[user@host ~]$ gdb /usr/bin/crash
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-11.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
....

(gdb) run /cores/retrace/tasks/330549506/crash/vmcore /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-642.el6.x86_64/vmlinux
Starting program: /usr/bin/crash /cores/retrace/tasks/330549506/crash/vmcore /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-642.el6.x86_64/vmlinux
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments

crash 7.2.7-3.el8
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
....

crash> extend /cores/crashext/mpykdumpx86_64.so
warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Setting scroll off while initializing PyKdump
/cores/crashext/mpykdumpx86_64.so: shared object loaded
....

crash> extend /home/rdu/jpittman/mpykdumpx86_64_2.so
extend: /cores/crashext/mpykdumpx86_64_2.so: "epython" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "xportshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "crashinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "taskinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "nfsshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "hanginfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "fregs" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "tslog" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "scsi" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "scsishow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "dmshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "pstree" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "modinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "mdadm" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "nvme" is a duplicate of a currently-existing command

Program received signal SIGSEGV, Segmentation fault.
0x00007fffdef02185 in _fini () at epython.c:438
438	  for (ce = epython_curext->command_table, ce++; ce->name; ce++) {

(gdb) bt
#0  0x00007fffdef02185 in _fini () at epython.c:438
#1  0x00007ffff6b2946c in _dl_catch_exception () from /lib64/libc.so.6
#2  0x00007ffff7de8ea9 in _dl_close_worker () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7de9ab2 in _dl_close () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff6b29414 in _dl_catch_exception () from /lib64/libc.so.6
#5  0x00007ffff6b294d3 in _dl_catch_error () from /lib64/libc.so.6
#6  0x00007ffff7bd1939 in _dlerror_run () from /lib64/libdl.so.2
#7  0x00007ffff7bd12e8 in dlclose () from /lib64/libdl.so.2
#8  0x00000000005337b8 in load_extension ()
#9  0x0000000000533f8d in cmd_extend ()
#10 0x000000000046596d in exec_command ()
#11 0x0000000000465bba in main_loop ()
#12 0x00000000006dff87 in captured_command_loop ()
#13 0x00000000006de9fa in catch_errors ()
#14 0x00000000006e10d6 in captured_main ()
#15 0x00000000006de9fa in catch_errors ()
#16 0x00000000006e1454 in gdb_main_entry ()
#17 0x00000000004640b4 in main ()
Comment 1 John Pittman 2020-12-10 16:03:36 UTC 
Same issue in upstream crash 7.2.9:

Program received signal SIGSEGV, Segmentation fault.
0x00007fffdf492185 in _fini () at epython.c:438
438	  for (ce = epython_curext->command_table, ce++; ce->name; ce++) {
Missing separate debuginfos, use: yum debuginfo-install libxcrypt-4.1.1-4.el8.x86_64 lzo-2.08-14.el8.x86_64 ncurses-libs-6.1-7.20180224.el8.x86_64 xz-libs-5.2.4-3.el8.x86_64 zlib-1.2.11-16.el8_2.x86_64

(gdb) bt
#0  0x00007fffdf492185 in _fini () at epython.c:438
#1  0x00007ffff6b0c46c in _dl_catch_exception () from /lib64/libc.so.6
#2  0x00007ffff7de8ea9 in _dl_close_worker () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7de9ab2 in _dl_close () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff6b0c414 in _dl_catch_exception () from /lib64/libc.so.6
#5  0x00007ffff6b0c4d3 in _dl_catch_error () from /lib64/libc.so.6
#6  0x00007ffff7bd1939 in _dlerror_run () from /lib64/libdl.so.2
#7  0x00007ffff7bd12e8 in dlclose () from /lib64/libdl.so.2
#8  0x0000000000523578 in load_extension (lib=0xd53ce7 <program_context+231> "/home/rdu/jpittman/mpykdumpx86_64_2.so") at extensions.c:335
#9  0x0000000000523cad in cmd_extend () at extensions.c:110
#10 0x0000000000464249 in exec_command () at main.c:892
#11 0x000000000046447a in main_loop () at main.c:839
#12 0x00000000006b5e13 in captured_command_loop (data=data@entry=0x0) at main.c:258
#13 0x00000000006b464a in catch_errors (func=func@entry=0x6b5e00 <captured_command_loop>, func_args=func_args@entry=0x0, errstring=errstring@entry=0x909619 "", mask=mask@entry=6)
    at exceptions.c:557
#14 0x00000000006b6e06 in captured_main (data=data@entry=0x7fffffffe130) at main.c:1064
#15 0x00000000006b464a in catch_errors (func=func@entry=0x6b60e0 <captured_main>, func_args=func_args@entry=0x7fffffffe130, errstring=errstring@entry=0x909619 "", mask=mask@entry=6)
    at exceptions.c:557
#16 0x00000000006b7117 in gdb_main (args=0x7fffffffe130) at main.c:1079
#17 gdb_main_entry (argc=<optimized out>, argv=argv@entry=0x7fffffffe298) at main.c:1099
#18 0x00000000004f9374 in gdb_main_loop (argc=<optimized out>, argc@entry=3, argv=argv@entry=0x7fffffffe298) at gdb_interface.c:76
#19 0x0000000000462a47 in main (argc=3, argv=0x7fffffffe298) at main.c:720
Comment 5 ltao 2021-04-30 09:07:39 UTC 
Hello,

This bug cannot be fixed by code directly, instead fix by documentation.

The reason of crash segfault is, crash uses RTLD_NOW|RTLD_GLOBAL flags of dlopen to load an extension.
RTDL_GLOBAL will make symbols defined by this shared object available for symbol resolution of 
subsequently loaded shared objects. So symbols in subsequently loaded extensions are overwritten by the 
former loaded one with a same name.

When loading a specific extension twice with 2 different file names, when user want to execute function A
in extension2, it actually executes function A in extension1. Because extension1 loaded prior to extension2,
function A in extension1 has already been executed, and some global values has already been changed, such as
array index etc. Thus if function A in exetension1 is executed again, values are not in their initial status.
it can make array index out of range and some other unexpected behaviours, that is why segfaults happens.

To avoid the problem, function A in extension2 must not be overwritten by extension1. There are 2 ways to solve:

1) function A should have different names in extension1 and extension2. It is difficult because each extension
developers have no idea whether this name has been taken by others.
2) Make functions and global values static. It can make symbols only visible within the extension. Making it a rule
for extension developers to follow.

Since extension developers always take extensions/echo.c as reference, changes was made in the file[1].

[1] https://listman.redhat.com/archives/crash-utility/2021-April/msg00002.html
Comment 6 ltao 2021-05-24 06:02:11 UTC 
brew build:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600003
Comment 7 ltao 2021-05-24 09:03:35 UTC 
(In reply to ltao from comment #6)
> brew build:
> 
> https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600003

Use the following brew build link:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1603781
Comment 26 errata-xmlrpc 2021-11-09 18:08:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (crash bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4199
Note You need to log in before you can comment on or make changes to this bug.