Bug 1906482

Summary: loading same extention twice with two different names segfaults crash
Product: Red Hat Enterprise Linux 8 Reporter: John Pittman <jpittman>
Component: crashAssignee: ltao
Status: CLOSED ERRATA QA Contact: xiaoying yan <yiyan>
Severity: medium Docs Contact: Sujata Kurup <skurup>
Priority: high    
Version: 8.2CC: bhe, cye, kdump-bugs, ltao, ruyang, skurup, yiyan
Target Milestone: rcKeywords: Triaged
Target Release: 8.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: crash-7.3.0-2.el8 Doc Type: Known Issue
Doc Text:
.Reloading an identical crash extension may cause segmentation faults When you load a copy of an already loaded crash extension file, it might trigger a segmentation fault. Currently, the crash utility detects if an original file has been loaded. Consequently, due to two identical files co-existing in the crash utility, a namespace collision occurs, which triggers the crash utility to cause a segmentation fault. You can work around the problem by loading the crash extension file only once. As a result, segmentation faults no longer occur in the described scenario.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 18:08:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1888160    

Description John Pittman 2020-12-10 16:00:59 UTC
Description of problem:

loading same extension twice with two different names segfaults crash

Version-Release number of selected component (if applicable):

crash-7.2.7-3.el8

How reproducible:

Every time

Steps to Reproduce:

1. Load .so extension
2. cp .so extension giving new name
3. Load .so w/ new name

Actual results:

crash segfaults

Expected results:

crash should not segfault

Additional info:

crash> extend /cores/crashext/mpykdumpx86_64.so
Setting scroll off while initializing PyKdump
/cores/crashext/mpykdumpx86_64.so: shared object loaded

# cp /cores/crashext/mpykdumpx86_64.so /cores/crashext/mpykdumpx86_64_2.so

crash> extend /cores/crashext/mpykdumpx86_64_2.so
extend: /cores/crashext/mpykdumpx86_64_2.so: "epython" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "xportshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "crashinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "taskinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "nfsshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "hanginfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "fregs" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "tslog" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "scsi" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "scsishow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "dmshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "pstree" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "modinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "mdadm" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "nvme" is a duplicate of a currently-existing command
Segmentation fault (core dumped)
[user@host ~]$ 

==========================================

[user@host ~]$ gdb /usr/bin/crash
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-11.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
....

(gdb) run /cores/retrace/tasks/330549506/crash/vmcore /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-642.el6.x86_64/vmlinux
Starting program: /usr/bin/crash /cores/retrace/tasks/330549506/crash/vmcore /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-642.el6.x86_64/vmlinux
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments

crash 7.2.7-3.el8
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
....

crash> extend /cores/crashext/mpykdumpx86_64.so
warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Setting scroll off while initializing PyKdump
/cores/crashext/mpykdumpx86_64.so: shared object loaded
....

crash> extend /home/rdu/jpittman/mpykdumpx86_64_2.so
extend: /cores/crashext/mpykdumpx86_64_2.so: "epython" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "xportshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "crashinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "taskinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "nfsshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "hanginfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "fregs" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "tslog" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "scsi" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "scsishow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "dmshow" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "pstree" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "modinfo" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "mdadm" is a duplicate of a currently-existing command
extend: /cores/crashext/mpykdumpx86_64_2.so: "nvme" is a duplicate of a currently-existing command

Program received signal SIGSEGV, Segmentation fault.
0x00007fffdef02185 in _fini () at epython.c:438
438	  for (ce = epython_curext->command_table, ce++; ce->name; ce++) {

(gdb) bt
#0  0x00007fffdef02185 in _fini () at epython.c:438
#1  0x00007ffff6b2946c in _dl_catch_exception () from /lib64/libc.so.6
#2  0x00007ffff7de8ea9 in _dl_close_worker () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7de9ab2 in _dl_close () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff6b29414 in _dl_catch_exception () from /lib64/libc.so.6
#5  0x00007ffff6b294d3 in _dl_catch_error () from /lib64/libc.so.6
#6  0x00007ffff7bd1939 in _dlerror_run () from /lib64/libdl.so.2
#7  0x00007ffff7bd12e8 in dlclose () from /lib64/libdl.so.2
#8  0x00000000005337b8 in load_extension ()
#9  0x0000000000533f8d in cmd_extend ()
#10 0x000000000046596d in exec_command ()
#11 0x0000000000465bba in main_loop ()
#12 0x00000000006dff87 in captured_command_loop ()
#13 0x00000000006de9fa in catch_errors ()
#14 0x00000000006e10d6 in captured_main ()
#15 0x00000000006de9fa in catch_errors ()
#16 0x00000000006e1454 in gdb_main_entry ()
#17 0x00000000004640b4 in main ()

Comment 1 John Pittman 2020-12-10 16:03:36 UTC
Same issue in upstream crash 7.2.9:

Program received signal SIGSEGV, Segmentation fault.
0x00007fffdf492185 in _fini () at epython.c:438
438	  for (ce = epython_curext->command_table, ce++; ce->name; ce++) {
Missing separate debuginfos, use: yum debuginfo-install libxcrypt-4.1.1-4.el8.x86_64 lzo-2.08-14.el8.x86_64 ncurses-libs-6.1-7.20180224.el8.x86_64 xz-libs-5.2.4-3.el8.x86_64 zlib-1.2.11-16.el8_2.x86_64

(gdb) bt
#0  0x00007fffdf492185 in _fini () at epython.c:438
#1  0x00007ffff6b0c46c in _dl_catch_exception () from /lib64/libc.so.6
#2  0x00007ffff7de8ea9 in _dl_close_worker () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7de9ab2 in _dl_close () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff6b0c414 in _dl_catch_exception () from /lib64/libc.so.6
#5  0x00007ffff6b0c4d3 in _dl_catch_error () from /lib64/libc.so.6
#6  0x00007ffff7bd1939 in _dlerror_run () from /lib64/libdl.so.2
#7  0x00007ffff7bd12e8 in dlclose () from /lib64/libdl.so.2
#8  0x0000000000523578 in load_extension (lib=0xd53ce7 <program_context+231> "/home/rdu/jpittman/mpykdumpx86_64_2.so") at extensions.c:335
#9  0x0000000000523cad in cmd_extend () at extensions.c:110
#10 0x0000000000464249 in exec_command () at main.c:892
#11 0x000000000046447a in main_loop () at main.c:839
#12 0x00000000006b5e13 in captured_command_loop (data=data@entry=0x0) at main.c:258
#13 0x00000000006b464a in catch_errors (func=func@entry=0x6b5e00 <captured_command_loop>, func_args=func_args@entry=0x0, errstring=errstring@entry=0x909619 "", mask=mask@entry=6)
    at exceptions.c:557
#14 0x00000000006b6e06 in captured_main (data=data@entry=0x7fffffffe130) at main.c:1064
#15 0x00000000006b464a in catch_errors (func=func@entry=0x6b60e0 <captured_main>, func_args=func_args@entry=0x7fffffffe130, errstring=errstring@entry=0x909619 "", mask=mask@entry=6)
    at exceptions.c:557
#16 0x00000000006b7117 in gdb_main (args=0x7fffffffe130) at main.c:1079
#17 gdb_main_entry (argc=<optimized out>, argv=argv@entry=0x7fffffffe298) at main.c:1099
#18 0x00000000004f9374 in gdb_main_loop (argc=<optimized out>, argc@entry=3, argv=argv@entry=0x7fffffffe298) at gdb_interface.c:76
#19 0x0000000000462a47 in main (argc=3, argv=0x7fffffffe298) at main.c:720

Comment 5 ltao 2021-04-30 09:07:39 UTC
Hello,

This bug cannot be fixed by code directly, instead fix by documentation.

The reason of crash segfault is, crash uses RTLD_NOW|RTLD_GLOBAL flags of dlopen to load an extension.
RTDL_GLOBAL will make symbols defined by this shared object available for symbol resolution of 
subsequently loaded shared objects. So symbols in subsequently loaded extensions are overwritten by the 
former loaded one with a same name.

When loading a specific extension twice with 2 different file names, when user want to execute function A
in extension2, it actually executes function A in extension1. Because extension1 loaded prior to extension2,
function A in extension1 has already been executed, and some global values has already been changed, such as
array index etc. Thus if function A in exetension1 is executed again, values are not in their initial status.
it can make array index out of range and some other unexpected behaviours, that is why segfaults happens.

To avoid the problem, function A in extension2 must not be overwritten by extension1. There are 2 ways to solve:

1) function A should have different names in extension1 and extension2. It is difficult because each extension
developers have no idea whether this name has been taken by others.
2) Make functions and global values static. It can make symbols only visible within the extension. Making it a rule
for extension developers to follow.

Since extension developers always take extensions/echo.c as reference, changes was made in the file[1].

[1] https://listman.redhat.com/archives/crash-utility/2021-April/msg00002.html

Comment 7 ltao 2021-05-24 09:03:35 UTC
(In reply to ltao from comment #6)
> brew build:
> 
> https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600003

Use the following brew build link:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1603781

Comment 26 errata-xmlrpc 2021-11-09 18:08:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (crash bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4199