Bug 1827734
| Summary: | crash cannot be extended with trace.so: no commands registered | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Eric Sandeen <esandeen> |
| Component: | crash-trace-command | Assignee: | kdump-bugs <kdump-bugs> |
| Status: | CLOSED ERRATA | QA Contact: | Emma Wu <xiawu> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.3 | CC: | bhsharma, lijiang, ruyang, xiawu |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | 8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | crash-trace-command-2.0-16.el8 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-11-04 02:47:48 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1814488 | ||
|
Description
Eric Sandeen
2020-04-24 16:29:57 UTC
(In reply to Eric Sandeen from comment #0) > [root@fs-i40c-03 ~]# crash vmcore > /usr/lib/debug/lib/modules/3.10.0-1062.el7.x86_64/vmlinux > ... > > crash> extend trace.so > extend: /usr/lib64/crash/extensions/trace.so: no commands registered: shared > object unloaded > crash> quit > > [root@fs-i40c-03 ~]# rpm -q crash crash-trace-command > crash-7.2.7-3.el8.x86_64 > crash-trace-command-2.0-15.el8.x86_64 > > [root@fs-i40c-03 ~]# uname -r > 4.18.0-193.el8.x86_64 > > Thanks, > -Eric Which kernel are you reporting against, 3.10.0-1062.el7 or 4.18.0-193.el8? The host is RHEL8, running kernel 4.18.0-193.el8 The vmcore is from a RHEL7 host running kernel version 3.10.0-1062.el7 If that's not expected to work and this is NOTABUG I apologize. No, it should still work. Do you happen to have the vmcore around? It's hosted on galvatron (though I was examining it on my own lab box): galvatron.cee.redhat.com:/cores/retrace/tasks/982832507/crash/vmcore (galvatron doesn't have the trace extension installed) (If you have trouble getting access on that host I can provide it to you) No, thanks, I've got it. I'm now creating a rhel8 vmcore without user space stripped just on the outside chance that the kernel version has something to do with it. The module loads OK with a RHEL8 vmcore: # crash vmlinux vmcore crash 7.2.7-3.el8 Copyright (C) 2002-2020 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [370MB]: patching 94975 gdb minimal_symbol values KERNEL: vmlinux DUMPFILE: vmcore CPUS: 2 DATE: Fri Apr 24 13:10:02 2020 UPTIME: 00:25:22 LOAD AVERAGE: 0.44, 0.86, 0.66 TASKS: 148 NODENAME: hpe-dl380pgen8-02-vm-15.hpe2.lab.eng.bos.redhat.com RELEASE: 4.18.0-193.el8.x86_64 VERSION: #1 SMP Fri Mar 27 14:35:58 UTC 2020 MACHINE: x86_64 (2892 Mhz) MEMORY: 4 GB PANIC: "sysrq: SysRq : Trigger a crash" PID: 51706 COMMAND: "bash" TASK: ffff993095dc0000 [THREAD_INFO: ffff993095dc0000] CPU: 0 STATE: TASK_RUNNING (SYSRQ) crash> extend trace.so /usr/lib64/crash/extensions/trace.so: shared object loaded crash> So, I guess it can't really be considered a RHEL8 bug. I'm provisioning a RHEL7 machine to see what happens there. (In reply to Dave Anderson from comment #7) > > ... > So, I guess it can't really be considered a RHEL8 bug. > > I'm provisioning a RHEL7 machine to see what happens there. And it comes up OK on a RHEL7 host: # crash vmlinux vmcore crash 7.2.3-11.el7 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [672MB]: patching 86789 gdb minimal_symbol values KERNEL: vmlinux DUMPFILE: vmcore [PARTIAL DUMP] CPUS: 32 DATE: Thu Apr 23 12:44:08 2020 UPTIME: 23:15:36 LOAD AVERAGE: 176.49, 175.54, 164.79 TASKS: 2909 NODENAME: worker2.caas.ford.com RELEASE: 3.10.0-1062.el7.x86_64 VERSION: #1 SMP Thu Jul 18 20:25:13 UTC 2019 MACHINE: x86_64 (3196 Mhz) MEMORY: 767.9 GB PANIC: "SysRq : Trigger a crash" PID: 103070 COMMAND: "bash" TASK: ffff9cf49d30e2a0 [THREAD_INFO: ffff9cfaf2bd0000] CPU: 23 STATE: TASK_RUNNING (SYSRQ) crash> extend trace.so /usr/lib64/crash/extensions/trace.so: shared object loaded crash> So, it also can't be considered a RHEL7 bug. Eric, can you take a look at your vmcore on a RHEL7 host? (In reply to Dave Anderson from comment #8) ... > And it comes up OK on a RHEL7 host: > ... > So, it also can't be considered a RHEL7 bug. > > Eric, can you take a look at your vmcore on a RHEL7 host? BTW, I tested it on this RHEL7 host: [root@dell-per820-02 ~]# uname -r 3.10.0-1136.el7.x86_64 [root@dell-per820-02 ~]# I'm sorry, I should have mentioned that it does work for me when running crash on a RHEL7 host. So the only problem seems to be a RHEL7 vmcore on a RHEL8 host / RHEL8 crash utility? > So the only problem seems to be a RHEL7 vmcore on a RHEL8 host / RHEL8 crash utility?
That's what it looks like.
While the crash utility itself is designed to always be backwards-compatible, Fujitsu's
extension modules apparently are not.
I should have looked upstream sooner; this seems to fix it, at least for a RHEL7
vmcore on a RHEL8 crash host:
commit f6d451d64fc91c8f0010d5b1fe96330e37845d31
Author: Dave Anderson <anderson>
Date: Mon Aug 20 10:43:05 2018 -0400
Fix for the "trace.c" extension module for RHEL7.6, which moved the
ftrace_event_call.data member into a new structure contained within
an anonymous union. Without the patch, the module fails to load,
indicating "no commands registered: shared object unloaded".
(xuhuan.fnst.com)
Yep, I thought this was vaguely familiar -- it was a RHEL-7.6.0 errata: Bug 1596549 - [RHEL-7.6][crash-trace-cmd] Failed to extend trace.so: "no commands registered: shared object unloaded" https://bugzilla.redhat.com/show_bug.cgi?id=1596549 RHBA-2018:3330-03 crash-trace-command bug fix update https://errata.devel.redhat.com/advisory/35950 So, I'm going to close this if you have no objection. Dave, this is a RHEL8 bug so I'm not sure the RHEL7 errata helps here. ;) Right, but the testing of this package against earlier RHEL versions is not done by QE. AFAIK all support/customer vmcores are now going to galvatron.cee.redhat.com, which is running RHEL8. If RHEL8's crash packages have problems working with RHEL7 vmcores that may become an issue. If support for non-rhel8 vmcores is outside the scope of rhel8's crash packages then support may need a different solution... (In reply to Eric Sandeen from comment #17) > AFAIK all support/customer vmcores are now going to > galvatron.cee.redhat.com, which is running RHEL8. If RHEL8's crash packages > have problems working with RHEL7 vmcores that may become an issue. > > If support for non-rhel8 vmcores is outside the scope of rhel8's crash > packages then support may need a different solution... Just to clarify: s/crash packages/crash extension module packages/g (In reply to Eric Sandeen from comment #17) > AFAIK all support/customer vmcores are now going to > galvatron.cee.redhat.com, which is running RHEL8. If RHEL8's crash packages > have problems working with RHEL7 vmcores that may become an issue. > > If support for non-rhel8 vmcores is outside the scope of rhel8's crash > packages then support may need a different solution... Also, it's been typical for SBR to update their own crash utility on optimus (and now galvatron) to use upstream crash utility sources so as to advance beyond whatever the default RHEL version is. So they could simply build and install the current upstream version of the trace.c extension module. In any case, I'm not arguing against updating the RHEL8 version, but just looking for some guidance here. > s/crash packages/crash extension module packages/g Understood - I just meant in general if anything crash-related from RHEL8 wasn't expected to be backwards compatible.... > Also, it's been typical for SBR to update their own crash utility Ok - that's fine, if rhel7 / backward compatibility is out of scope for rhel8, feel free to close this WONTFIX and I'll find the right folks to just do a custom update on galvatron. Thanks for looking into all this! -Eric I've got the package/patch ready-to-go if Emma/Dave buy into it. Thanks all. Just for the record, I'm also fine with support maintaining their own version of crash tools if that is the best / most efficient path. I had wrongly assumed that the backwards compatibility was expected. -Eric Hi Emma, I built the package successfully: Information for build crash-trace-command-2.0-16.el8 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1180064 But the CI Gating failed: Brew build: https://brewweb.engineering.redhat.com/brew/search?match=glob&type=build&terms=+crash-trace-command-2.0-16.el8 CI Dashboard: https://dashboard.osci.redhat.com/#/artifact/brew-build/aid/28230116 There was an issue running required tests enabled for gating. Please investigate the following issues and fix the package, tests or waive the results. Some of the results are still missing and did not yet finish, the gating is waiting for them. 1 of 5 required tests failed, 1 result missing: >> Failed tests kernel-qe.kernel-ci.general-crash.tier0.functional - test logs: https://dashboard.osci.redhat.com/#/artifact/brew-build/aid/28230116?focus=tc:kernel-qe.kernel-ci.general-crash.tier0.functional - waive: Use CI dashboard or CLI command "$ waiverdb-cli -p rhel-8 -r 8778388" with "-c option" - retest: https://platform-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/kernel-general-rhel-crash-plugins-t0/9/rebuild >> Missing tests (still waiting for results): ... In my mind-boggling attempt to follow the CI URLs to figure out why the "rebuild" failed, I eventually found this: https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2020/04/42269/4226999/8189941/109603705/taskout.log chmod a+x ./runtest.sh ./pkginfo.pl ./runtest.sh cp: cannot stat '/mnt/testarea/boot.messages': No such file or directory ***** Start of crash-trace-command-2.0-16.el8 install test ***** pkginfo: crash-trace-command:crash-trace-command::2.0:16.el8: pkg: crash-trace-command pkgname: crash-trace-command pkgvariant: pkgver: 2.0 pkgrel: 16.el8 pkgarch: package crash-trace-command-2.0-16.el8-2.0-16.el8 is not installed package crash-trace-command-2.0-16.el8 is not installed pkg2Binstalled : crash-trace-command-2.0-16.el8 ***** Install package via yum -y install crash-trace-command-2.0-16.el8 ***** Updating Subscription Management repositories. Unable to read consumer identity This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Last metadata expiration check: 0:08:19 ago on Tue 28 Apr 2020 11:18:53 AM EDT. Error: No matching Packages to list *** crash-trace-command-2.0-16.el8 doesn't seem to exist in any yum repos .. *** package crash-trace-command-2.0-16.el8 is not installed ***** Could not install crash-trace-command from yum repo ***** package crash-trace-command is not installed crash-trace-command doesn't exist on the system.. installing error: Failed dependencies: trace-cmd is needed by crash-trace-command-2.0-16.el8.x86_64 Retrieving http://porkchop.redhat.com/brewroot/packages/crash-trace-command/2.0/16.el8/x86_64/crash-trace-command-2.0-16.el8.x86_64.rpm error: Failed dependencies: trace-cmd is needed by crash-trace-command-2.0-16.el8.x86_64 Retrieving http://porkchop.redhat.com/brewroot/packages/crash-trace-command/2.0/16.el8/x86_64/crash-trace-command-2.0-16.el8.x86_64.rpm ***** Could not install crash-trace-command from brew either ** /distribution/pkginstall/BrewInstallPackage FAIL Score:0 Uploading resultoutputfile.log .done Uploading pkginstall.g1FYHM .done I don't understand what the problem is. It seems to say that: (1) crash-trace-command-2.0-16.el8 doesn't exist in any yum repos (2) failed the trace-cmd package dependency I did do a retest of the failed rebuild test, and it still fails. So I don't what I can do. Please advise. Thanks, Dave Well, somebody kicked off a 3rd attempt last night, and the test passed. So thanks to whoever took care of it! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (crash-trace-command bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4666 |