(In reply to Eric Sandeen from comment #0)
> [root@fs-i40c-03 ~]# crash vmcore
> /usr/lib/debug/lib/modules/3.10.0-1062.el7.x86_64/vmlinux
> ...
> 
> crash> extend trace.so
> extend: /usr/lib64/crash/extensions/trace.so: no commands registered: shared
> object unloaded
> crash> quit
> 
> [root@fs-i40c-03 ~]# rpm -q crash crash-trace-command
> crash-7.2.7-3.el8.x86_64
> crash-trace-command-2.0-15.el8.x86_64
> 
> [root@fs-i40c-03 ~]# uname -r
> 4.18.0-193.el8.x86_64
> 
> Thanks,
> -Eric

Which kernel are you reporting against, 3.10.0-1062.el7 or 4.18.0-193.el8?

The host is RHEL8, running kernel 4.18.0-193.el8

The vmcore is from a RHEL7 host running kernel version 3.10.0-1062.el7 

If that's not expected to work and this is NOTABUG I apologize.

No, it should still work.  Do you happen to have the vmcore around?

It's hosted on galvatron (though I was examining it on my own lab box):

galvatron.cee.redhat.com:/cores/retrace/tasks/982832507/crash/vmcore

(galvatron doesn't have the trace extension installed)

(If you have trouble getting access on that host I can provide it to you)

No, thanks, I've got it.  I'm now creating a rhel8 vmcore without user space stripped
just on the outside chance that the kernel version has something to do with it.

The module loads OK with a RHEL8 vmcore:

  # crash vmlinux vmcore

  crash 7.2.7-3.el8
  Copyright (C) 2002-2020  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005, 2011  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
 
  GNU gdb (GDB) 7.6
  Copyright (C) 2013 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...

  WARNING: kernel relocated [370MB]: patching 94975 gdb minimal_symbol values

        KERNEL: vmlinux                                                  
      DUMPFILE: vmcore
          CPUS: 2
          DATE: Fri Apr 24 13:10:02 2020
        UPTIME: 00:25:22
  LOAD AVERAGE: 0.44, 0.86, 0.66
         TASKS: 148
      NODENAME: hpe-dl380pgen8-02-vm-15.hpe2.lab.eng.bos.redhat.com
       RELEASE: 4.18.0-193.el8.x86_64
       VERSION: #1 SMP Fri Mar 27 14:35:58 UTC 2020
       MACHINE: x86_64  (2892 Mhz)
        MEMORY: 4 GB
         PANIC: "sysrq: SysRq : Trigger a crash"
           PID: 51706
       COMMAND: "bash"
          TASK: ffff993095dc0000  [THREAD_INFO: ffff993095dc0000]
           CPU: 0
         STATE: TASK_RUNNING (SYSRQ)

  crash> extend trace.so
  /usr/lib64/crash/extensions/trace.so: shared object loaded
  crash> 

So, I guess it can't really be considered a RHEL8 bug.

I'm provisioning a RHEL7 machine to see what happens there.

(In reply to Dave Anderson from comment #7)
>
> ...
> So, I guess it can't really be considered a RHEL8 bug.
> 
> I'm provisioning a RHEL7 machine to see what happens there.

And it comes up OK on a RHEL7 host:
      
  # crash vmlinux vmcore
  
  crash 7.2.3-11.el7
  Copyright (C) 2002-2017  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005, 2011  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.6
  Copyright (C) 2013 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
  WARNING: kernel relocated [672MB]: patching 86789 gdb minimal_symbol values
  
        KERNEL: vmlinux                                                  
      DUMPFILE: vmcore  [PARTIAL DUMP]
          CPUS: 32
          DATE: Thu Apr 23 12:44:08 2020
        UPTIME: 23:15:36
  LOAD AVERAGE: 176.49, 175.54, 164.79
         TASKS: 2909
      NODENAME: worker2.caas.ford.com
       RELEASE: 3.10.0-1062.el7.x86_64
       VERSION: #1 SMP Thu Jul 18 20:25:13 UTC 2019
       MACHINE: x86_64  (3196 Mhz)
        MEMORY: 767.9 GB
         PANIC: "SysRq : Trigger a crash"
           PID: 103070
       COMMAND: "bash"
          TASK: ffff9cf49d30e2a0  [THREAD_INFO: ffff9cfaf2bd0000]
           CPU: 23
         STATE: TASK_RUNNING (SYSRQ)
  
  crash> extend trace.so
  /usr/lib64/crash/extensions/trace.so: shared object loaded
  crash> 

So, it also can't be considered a RHEL7 bug.

Eric, can you take a look at your vmcore on a RHEL7 host?

(In reply to Dave Anderson from comment #8)
...
> And it comes up OK on a RHEL7 host:
> ...
> So, it also can't be considered a RHEL7 bug.
>
> Eric, can you take a look at your vmcore on a RHEL7 host?
 
BTW, I tested it on this RHEL7 host:

  [root@dell-per820-02 ~]# uname -r
  3.10.0-1136.el7.x86_64
  [root@dell-per820-02 ~]#

I'm sorry, I should have mentioned that it does work for me when running crash on a RHEL7 host.

So the only problem seems to be a RHEL7 vmcore on a RHEL8 host / RHEL8 crash utility?

> So the only problem seems to be a RHEL7 vmcore on a RHEL8 host / RHEL8 crash utility?

That's what it looks like.

While the crash utility itself is designed to always be backwards-compatible, Fujitsu's
extension modules apparently are not.

I should have looked upstream sooner; this seems to fix it, at least for a RHEL7
vmcore on a RHEL8 crash host:

commit f6d451d64fc91c8f0010d5b1fe96330e37845d31
Author: Dave Anderson <anderson>
Date:   Mon Aug 20 10:43:05 2018 -0400

    Fix for the "trace.c" extension module for RHEL7.6, which moved the
    ftrace_event_call.data member into a new structure contained within
    an anonymous union.  Without the patch, the module fails to load,
    indicating "no commands registered: shared object unloaded".
    (xuhuan.fnst.com)

Yep, I thought this was vaguely familiar -- it was a RHEL-7.6.0 errata:

  Bug 1596549 - [RHEL-7.6][crash-trace-cmd] Failed to extend trace.so: "no commands registered: shared object unloaded" 
  https://bugzilla.redhat.com/show_bug.cgi?id=1596549

  RHBA-2018:3330-03 crash-trace-command bug fix update
  https://errata.devel.redhat.com/advisory/35950

So, I'm going to close this if you have no objection.

Dave, this is a RHEL8 bug so I'm not sure the RHEL7 errata helps here. ;)

Right, but the testing of this package against earlier RHEL versions
is not done by QE.

AFAIK all support/customer vmcores are now going to galvatron.cee.redhat.com, which is running RHEL8.  If RHEL8's crash packages have problems working with RHEL7 vmcores that may become an issue.

If support for non-rhel8 vmcores is outside the scope of rhel8's crash packages then support may need a different solution...

(In reply to Eric Sandeen from comment #17)
> AFAIK all support/customer vmcores are now going to
> galvatron.cee.redhat.com, which is running RHEL8.  If RHEL8's crash packages
> have problems working with RHEL7 vmcores that may become an issue.
> 
> If support for non-rhel8 vmcores is outside the scope of rhel8's crash
> packages then support may need a different solution...

Just to clarify:

  s/crash packages/crash extension module packages/g

(In reply to Eric Sandeen from comment #17)
> AFAIK all support/customer vmcores are now going to
> galvatron.cee.redhat.com, which is running RHEL8.  If RHEL8's crash packages
> have problems working with RHEL7 vmcores that may become an issue.
> 
> If support for non-rhel8 vmcores is outside the scope of rhel8's crash
> packages then support may need a different solution...

Also, it's been typical for SBR to update their own crash utility on optimus
(and now galvatron) to use upstream crash utility sources so as to advance 
beyond whatever the default RHEL version is.  So they could simply build and
install the current upstream version of the trace.c extension module.

In any case, I'm not arguing against updating the RHEL8 version, but just
looking for some guidance here.

> s/crash packages/crash extension module packages/g

Understood - I just meant in general if anything crash-related from RHEL8 wasn't expected to be backwards compatible....

> Also, it's been typical for SBR to update their own crash utility

Ok - that's fine, if rhel7 / backward compatibility is out of scope for rhel8, feel free to close this WONTFIX and I'll find the right folks to just do a custom update on galvatron.

Thanks for looking into all this!

-Eric

I've got the package/patch ready-to-go if Emma/Dave buy into it.

Thanks all.  Just for the record, I'm also fine with support maintaining their own version of crash tools if that is the best / most efficient path.  I had wrongly assumed that the backwards compatibility was expected.

-Eric

Hi Emma,

I built the package successfully:

  Information for build crash-trace-command-2.0-16.el8
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1180064

But the CI Gating failed:

  Brew build: https://brewweb.engineering.redhat.com/brew/search?match=glob&type=build&terms=+crash-trace-command-2.0-16.el8
  CI Dashboard: https://dashboard.osci.redhat.com/#/artifact/brew-build/aid/28230116
  
  There was an issue running required tests enabled for gating. Please investigate
  the following issues and fix the package, tests or waive the results.
  Some of the results are still missing and did not yet finish, the gating is waiting for them.
  
  1 of 5 required tests failed, 1 result missing:
    
    >> Failed tests
      
      kernel-qe.kernel-ci.general-crash.tier0.functional
        - test logs: https://dashboard.osci.redhat.com/#/artifact/brew-build/aid/28230116?focus=tc:kernel-qe.kernel-ci.general-crash.tier0.functional
        - waive: Use CI dashboard or CLI command "$ waiverdb-cli -p rhel-8 -r 8778388" with "-c option"
        - retest: https://platform-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/kernel-general-rhel-crash-plugins-t0/9/rebuild
      
    >> Missing tests (still waiting for results):
  
  ...
  
In my mind-boggling attempt to follow the CI URLs to figure out why the 
"rebuild" failed, I eventually found this:
  
  https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2020/04/42269/4226999/8189941/109603705/taskout.log
  
  chmod a+x ./runtest.sh ./pkginfo.pl
  ./runtest.sh
  cp: cannot stat '/mnt/testarea/boot.messages': No such file or directory
  ***** Start of crash-trace-command-2.0-16.el8 install test *****
  pkginfo: crash-trace-command:crash-trace-command::2.0:16.el8:  pkg: crash-trace-command pkgname: crash-trace-command pkgvariant:  pkgver: 2.0 pkgrel: 16.el8 pkgarch: 
  package crash-trace-command-2.0-16.el8-2.0-16.el8 is not installed
  package crash-trace-command-2.0-16.el8 is not installed
  pkg2Binstalled : crash-trace-command-2.0-16.el8 
  ***** Install package via yum -y install crash-trace-command-2.0-16.el8 *****
  Updating Subscription Management repositories.
  Unable to read consumer identity
  This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
  Last metadata expiration check: 0:08:19 ago on Tue 28 Apr 2020 11:18:53 AM EDT.
  Error: No matching Packages to list
  *** crash-trace-command-2.0-16.el8 doesn't seem to exist in any yum repos .. ***
  package crash-trace-command-2.0-16.el8 is not installed
  ***** Could not install crash-trace-command from yum repo *****
  package crash-trace-command is not installed
  crash-trace-command doesn't exist on the system..  installing
  error: Failed dependencies:
  	trace-cmd is needed by crash-trace-command-2.0-16.el8.x86_64
  Retrieving http://porkchop.redhat.com/brewroot/packages/crash-trace-command/2.0/16.el8/x86_64/crash-trace-command-2.0-16.el8.x86_64.rpm
  error: Failed dependencies:
  	trace-cmd is needed by crash-trace-command-2.0-16.el8.x86_64
  Retrieving http://porkchop.redhat.com/brewroot/packages/crash-trace-command/2.0/16.el8/x86_64/crash-trace-command-2.0-16.el8.x86_64.rpm
  ***** Could not install crash-trace-command from brew either
  ** /distribution/pkginstall/BrewInstallPackage FAIL Score:0
  Uploading resultoutputfile.log .done
  Uploading pkginstall.g1FYHM .done

I don't understand what the problem is.  It seems to say that:

 (1) crash-trace-command-2.0-16.el8 doesn't exist in any yum repos
 (2) failed the trace-cmd package dependency

I did do a retest of the failed rebuild test, and it still fails.

So I don't what I can do.  Please advise.

Thanks,
  Dave

Well, somebody kicked off a 3rd attempt last night, and the test passed.
So thanks to whoever took care of it!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (crash-trace-command bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4666