Bug 695413

Hi Denys,

Thanks for your report.  The man page and "crash -h" output has
remained pretty much static since its introduction in RHEL3, and
is long overdue for an update.

But let me try to answer your questions first:

> crash manpage:
> 
> SYNOPSIS
>        crash [ -h [ opt ] ] [ -v ] [ -s ] [ -i file ] [ -d num ] [ -S ] [ map-
>        file ] [ namelist ] [ dumpfile ]
> 
> OPTIONS
>        namelist
>               This  is  a  pathname to an uncompressed kernel image (a vmlinux
>               file) that has been compiled with the "-g" option, or  that  has
>               an  accessible,  associated,  debuginfo  file.
> 
> What "that" refers to? "uncompressed kernel image"? I am not a 
> native speaker, but I think "which" would be more appropriate.

Yes, both instances of "that" in the sentence above 
refer to the uncompressed kernel image.  What else 
could it refer to?

Anyway, I think that either "that" or "which" are
interchangable.  According to:

   http://dictionary.reference.com/browse/that

the usage in the man page is an example of using "that"
as a pronoun, where:

  (used as the subject or object of a relative clause, 
  especially one defining or restricting the antecedent, 
  sometimes replaceable by who, whom,  or which ): the 
  horse that he bought. 

But, if you feel that "which" works better, I have no 
problem changing it.  

And on an unrelated note, in RHEL6.2 I hope to rebase to
upstream, and the upstream version of crash now accepts 
compressed .gz or .bz2 vmlinux files.   So the "uncompressed"
part will become obsolete.

> What does "accessible,  associated" debuginfo file mean? 
> (IOW: how reader of > the manpage can find out whether he
> has it or not?)

Well, that all goes back to the initial introduction of the 
crash utility in RHEL3 -- and where RHEL3 also introduced the
kernel-debuginfo packages for the the first time.  

For example, in RHEL3, there were two x86_64 kernels released
in RHEL3, UP and MP, and where their .rpm files contained their 
respective "stripped" vmlinux files:

  $ rpm -qpl kernel-2.4.21-66.EL.x86_64.rpm | grep vmlinux
  /boot/vmlinux-2.4.21-66.EL
  $
  
  $ rpm -qpl kernel-smp-2.4.21-66.EL.x86_64.rpm | grep vmlinux
  /boot/vmlinux-2.4.21-66.ELsmp
  $
  
But those two vmlinux files alone were useless without their 
"associated" .debug files that came with the consolidated RHEL3
kernel-debuginfo package:
  
  $ rpm -qpl kernel-debuginfo-2.4.21-66.EL.x86_64.rpm | grep vmlinux
  /usr/lib/debug/boot/vmlinux-2.4.21-66.EL.debug
  /usr/lib/debug/boot/vmlinux-2.4.21-66.ELsmp.debug
  $

So if you just had the kernel package installed -- but didn't
have the kernel-debuginfo package installed -- then you'd have 
a /boot/vmlinux-<version> file that had no practical use.
(It didn't even get booted -- the "vmlinuz" file was the object
file that was actually loaded into memory and run...)
  
Fortunately, from RHEL4 and beyond, each kernel type has a single
"vmlinux" file with debuginfo data, which is contained in the 
per-type kernel-debuginfo package, and which gets installed in:

  /usr/lib/debug/lib/modules/<version>/vmlinux

Anyway, to answer your question, the "accessible, associated"
debuginfo file is relevant to RHEL3 only.

It made sense at the time...  ;-)

>               If the dumpfile
>               argument is entered, then this argument must also be  used.
> 
> So the format is actually [ namelist dumpfile ], not [ namelist ] 
> [ dumpfile ] as shown above?
 
Not always.  If a dumpfile argument is entered, then it is
mandatory that a namelist argument also be entered.  However,
if crash is being run on a live system, then the namelist
argument does not have to be entered, because it can typically
be found automatically.  But if crash is being run on a live
system, and the vmlinux namelist is *not* located in a "known" 
location, then it would have to put on the command line.  

How would you better express that?

>               If the namelist argument is not entered and no dumpfile argument is
>               entered, crash will search in several typical directories for  a
>               kernel namelist that matches the live system.
> 
>        mapfile
>               If the live system kernel, or the kernel from which the dumpfile
>               was derived, was not compiled with the -g switch, then the addi-
>               tional mapfile argument is required.
> 
> Aha, so the correct format is actually [ [ mapfile ] namelist dumpfile ]?
> If yes, please correct the help text.

Right -- but I still don't know how to better express the fact
that a namelist would still be required on a live system if the
namelist is not found in a "known" location?  

A mapfile argument should almost *never* be required.  It is only
necessary it the vmlinux namelist is not exactly the one that was
running when the system crashed, or the live system kernel.  Or
better put, if the kernel symbol values of the vmlinux namelist
do not match those of the running or crash kernel.  And the 
"mapfile" description above is alluding to the fact that if the 
original kernel was not built with -g, then the user *had* to build
one after the fact, and the kernel symbol values of the new
debuginfo vmlinux file would not match the original.  And because
of that, a mapfile argument would have to be used.

So with respect to the mapfile argument statement above, I agree
that it is definitely confusing.  In fact, that description goes
back to the RHEL2.1 timeframe.  Back then, we had the "netdump" 
kernel crash dumping facility, but there was no kernel-debuginfo
package, and the vmlinux file was *not* built with -g.  So to use
the netdump core file in RHEL2.1, you had to:

 (1) get the src.rpm of the crash kernel.
 (2) modify the kernel Makefile's CFLAGS to add -g
 (3) rebuild the kernel to get a vmlinux file with debuginfo
     data. 

Unfortunately, the symbols in the original stripped vmlinux file
and the newly-rebuilt "-g" vmlinux did not match, and so you had
to also use the System.map file of the original kernel, as in:

  $ crash vmlinux-rebuilt-with-g System.map-of-original-kernel

Clear as mud?
 
So I'm not sure how best to rewrite it given all of the changes
in RHEL kernel history?

Do you have any suggestions?

>               It may be either the associated System.map file,
> 
> ok...
> 
>               or the non-debug kernel namelist.   How-
>               ever,
> 
> I suggest "However" word to be dropped here.
 
OK fine.

>               if  the mapfile argument is used, then the namelist argu-
>               ment must be a kernel namelist of a similar kernel version  that
>               was built with the -g switch.
> 
> "of a similar kernel version" - similar to which version?
> What word "that" refers to above?

Similar to the version of the running (live), or crashed (dumpfile) kernel.
The "that" refers to the kernel namelist.  What else can it refer to?

>        dumpfile
>               This  is  a  pathname to a kernel memory core dump file.  If the
>               dumpfile argument is not entered, the session will be invoked on
>               the  live  system  using  /dev/mem,  which usually requires root
>               privileges.
> 
> Did I get it right that if the dumpfile argument is not entered, all other
> arguments cannot be entered too (because the format of the arguments are [ [
> mapfile ] namelist dumpfile ] and thus entering non-option arguments w/o
> entering dumpfile argument is not possible?

I'm sorry -- but I'm not sure I understand your question? 

If no dumpfile argument is entered, then by definition you are
running on the live system.  And if you are running on the live
system, you can still enter any, or all, of the other arguments.

Is that what you're asking?
 
> If yes: I suggest separating the second sentence about from dumpfile
> description, like this:
> 
>        dumpfile
>               This  is  a  pathname to a kernel memory core dump file.
 
Well, actually that's not true either.  To confuse the issue a
bit more, you are permitted to enter "/dev/mem", "/dev/crash" or
"/proc/kcore" as a memory source argument, which would override
the default live memory source.

>        If no arguments are entered, the session will be invoked on
>        the  live  system  using  /dev/mem,  which usually requires root
>        privileges.

Another reason that the man page *should* be reworked -- the 
replacement of /dev/mem with the RHEL "/dev/crash" driver in
RHEL4 and later kernels is not even mentioned at all.

> If no: then the manpage as it is written now is too confusing. At least for me.
> I'm am confused about this: is it syntactically possible to run crash without
> dumpfile argument but with some other argument(s) present? If yes, what does
> that mean?
 
Sure -- you'll just run on the "live" system.  Like this:

 # crash

 crash 5.1.1-2.el6
 Copyright (C) 2002-2011  Red Hat, Inc.
 Copyright (C) 2004, 2005, 2006  IBM Corporation
 Copyright (C) 1999-2006  Hewlett-Packard Co
 Copyright (C) 2005, 2006  Fujitsu Limited
 Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
 Copyright (C) 2005  NEC Corporation
 Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
 Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
 This program is free software, covered by the GNU General Public License,
 and you are welcome to change it and/or distribute copies of it under
 certain conditions.  Enter "help copying" to see the conditions.
 This program has absolutely no warranty.  Enter "help warranty" for details.
 
 GNU gdb (GDB) 7.0
 Copyright (C) 2009 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "x86_64-unknown-linux-gnu"...

       KERNEL: /usr/lib/debug/lib/modules/2.6.32-70.el6.x86_64/vmlinux
     DUMPFILE: /dev/crash
         CPUS: 4
         DATE: Mon Apr 11 14:55:59 2011
       UPTIME: 42 days, 05:32:25
 LOAD AVERAGE: 0.36, 0.08, 0.03
        TASKS: 326
     NODENAME: waste.usersys.redhat.com
      RELEASE: 2.6.32-70.el6.x86_64
      VERSION: #1 SMP Wed Aug 25 10:17:53 EDT 2010
      MACHINE: x86_64  (2792 Mhz)
       MEMORY: 1 GB
          PID: 8851
      COMMAND: "crash"
         TASK: ffff8800234ff4a0  [THREAD_INFO: ffff88000ef5e000]
          CPU: 2
        STATE: TASK_RUNNING (ACTIVE)

 crash> 

It found the vmlinux file in a known location, so it is 
normally not necessary to put it on the command line.

However, suppose that the kernel-debuginfo package 
containing the vmlinux file was not installed, but you
still wanted to run "live"?  Well, you could get a copy
of the vmlinux file, put it in /tmp, and then invoke
it as:

 $ crash /tmp/vmlinux

The point is that the crash utility just wants a 
"memory source" of *some* type, be it either:

 - one of about 17 different supported dumpfile formats
   of crashed kernels, or
 - live system memory -- via /dev/mem, /dev/crash, and most
   recently /proc/kcore.  

Most RHEL kernels are now built with CONFIG_STRICT_DEVMEM,
which rendered /dev/mem useless, and /proc/kcore has been
stripped of its usefulness in RHEL5 and RHEL6.  Hence the
RHEL /dev/crash driver was developed and is part of the
RHEL kernel.

> 
> Regarding crash -h: it basically shows the same info as manpage
> (and therefore is equally confusing). Also, it lists all options and
> parameters in [] brackets:
> 
>   [-S]
>        Use "/boot/System.map" as the [mapfile].
> 
> Usually, []'s aren't used like that in help text. They are used
> only in command line syntax line. I suggest changing it to:
> 
>   -S
>        Use "/boot/System.map" as the mapfile.

OK fine, that makes sense...

Dave

Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

(In reply to comment #2)
> But let me try to answer your questions first:
> 
> > crash manpage:
> > 
> > SYNOPSIS
> >        crash [ -h [ opt ] ] [ -v ] [ -s ] [ -i file ] [ -d num ] [ -S ] [ map-
> >        file ] [ namelist ] [ dumpfile ]
> > 
> > OPTIONS
> >        namelist
> >               This  is  a  pathname to an uncompressed kernel image (a vmlinux
> >               file) that has been compiled with the "-g" option, or  that  has
> >               an  accessible,  associated,  debuginfo  file.
> > 
> > What does "accessible,  associated" debuginfo file mean? 
> > (IOW: how reader of > the manpage can find out whether he
> > has it or not?)
> 
> Well, that all goes back to the initial introduction of the 
> crash utility in RHEL3 -- and where RHEL3 also introduced the
> kernel-debuginfo packages for the the first time.  
> 
> For example, in RHEL3, there were two x86_64 kernels released
> in RHEL3, UP and MP, and where their .rpm files contained their 
> respective "stripped" vmlinux files:
> 
>   $ rpm -qpl kernel-2.4.21-66.EL.x86_64.rpm | grep vmlinux
>   /boot/vmlinux-2.4.21-66.EL
>   $
> 
>   $ rpm -qpl kernel-smp-2.4.21-66.EL.x86_64.rpm | grep vmlinux
>   /boot/vmlinux-2.4.21-66.ELsmp
>   $
> 
> But those two vmlinux files alone were useless without their 
> "associated" .debug files that came with the consolidated RHEL3
> kernel-debuginfo package:
> 
>   $ rpm -qpl kernel-debuginfo-2.4.21-66.EL.x86_64.rpm | grep vmlinux
>   /usr/lib/debug/boot/vmlinux-2.4.21-66.EL.debug
>   /usr/lib/debug/boot/vmlinux-2.4.21-66.ELsmp.debug
>   $
> 
> So if you just had the kernel package installed -- but didn't
> have the kernel-debuginfo package installed -- then you'd have 
> a /boot/vmlinux-<version> file that had no practical use.
> (It didn't even get booted -- the "vmlinuz" file was the object
> file that was actually loaded into memory and run...)
> 
> Fortunately, from RHEL4 and beyond, each kernel type has a single
> "vmlinux" file with debuginfo data, which is contained in the 
> per-type kernel-debuginfo package, and which gets installed in:
> 
>   /usr/lib/debug/lib/modules/<version>/vmlinux

Looks like the above paragraph about /usr/lib/debug/lib/modules/<version>/vmlinux would be useful in manpage. I didn't figure it out from manpage and had to google for the usage example.

> >               If the dumpfile
> >               argument is entered, then this argument must also be  used.
> > 
> > So the format is actually [ namelist dumpfile ], not [ namelist ] 
> > [ dumpfile ] as shown above?
> 
> Not always.  If a dumpfile argument is entered, then it is
> mandatory that a namelist argument also be entered. However,
> if crash is being run on a live system, then the namelist
> argument does not have to be entered, because it can typically
> be found automatically.
>
> But if crash is being run on a live
> system, and the vmlinux namelist is *not* located in a "known" 
> location, then it would have to put on the command line.  
> 
> How would you better express that?

Ah, you are saying that crash can be called as
    crash NAMELIST DUMPFILE
or as
    crash [NAMELIST]
which cannot be fully expressed with []'s?

In cases like this, when you have widely different usage case, people usually resort to documenting them separately instead of trying to cram it into all-encompassing one-line syntax. Example:

# ln --help
Usage: ln [OPTION]... [-T] TARGET LINK_NAME   (1st form)
  or:  ln [OPTION]... TARGET                  (2nd form)
  or:  ln [OPTION]... TARGET... DIRECTORY     (3rd form)
  or:  ln [OPTION]... -t DIRECTORY TARGET...  (4th form)
In the 1st form, create a link to TARGET with the name LINK_NAME.
In the 2nd form, create a link to TARGET in the current directory.
In the 3rd and 4th forms, create links to each TARGET in DIRECTORY.
Create hard links by default, symbolic links with --symbolic.
When creating hard links, each TARGET must exist.  Symbolic links
can hold arbitrary text; if later resolved, a relative link is
interpreted in relation to its parent directory.

Can you use this approach?


> > Aha, so the correct format is actually [ [ mapfile ] namelist dumpfile ]?
> > If yes, please correct the help text.
> 
> Right -- but I still don't know how to better express the fact
> that a namelist would still be required on a live system if the
> namelist is not found in a "known" location? 

SYNOPSIS
    crash [-h [OPT]] [-i FILE] [-d NUM] [-vsS] [MAPFILE] NAMELIST DUMPFILE
    crash [-h [OPT]] [-i FILE] [-d NUM] [-vsS] [[MAPFILE] NAMELIST]
    ...<other usages if they are very different>....


> A mapfile argument should almost *never* be required.  It is only
> necessary it the vmlinux namelist is not exactly the one that was
> running when the system crashed, or the live system kernel.  Or
> better put, if the kernel symbol values of the vmlinux namelist
> do not match those of the running or crash kernel.  And the 
> "mapfile" description above is alluding to the fact that if the 
> original kernel was not built with -g, then the user *had* to build
> one after the fact, and the kernel symbol values of the new
> debuginfo vmlinux file would not match the original.  And because
> of that, a mapfile argument would have to be used.
> 
> So with respect to the mapfile argument statement above, I agree
> that it is definitely confusing.  In fact, that description goes
> back to the RHEL2.1 timeframe.  Back then, we had the "netdump" 
> kernel crash dumping facility, but there was no kernel-debuginfo
> package, and the vmlinux file was *not* built with -g.  So to use
> the netdump core file in RHEL2.1, you had to:
> 
>  (1) get the src.rpm of the crash kernel.
>  (2) modify the kernel Makefile's CFLAGS to add -g
>  (3) rebuild the kernel to get a vmlinux file with debuginfo
>      data. 
> 
> Unfortunately, the symbols in the original stripped vmlinux file
> and the newly-rebuilt "-g" vmlinux did not match, and so you had
> to also use the System.map file of the original kernel, as in:
> 
>   $ crash vmlinux-rebuilt-with-g System.map-of-original-kernel
> 
> Clear as mud?
> 
> So I'm not sure how best to rewrite it given all of the changes
> in RHEL kernel history?

In RHELx manpage, you probably need to focus on how to use the tool on RHELx, not on RHELx-2...



> Do you have any suggestions?

Examples tend to help understand typical operations better than long-winded explanations.



> >               if  the mapfile argument is used, then the namelist argu-
> >               ment must be a kernel namelist of a similar kernel version  that
> >               was built with the -g switch.
> > 
> > "of a similar kernel version" - similar to which version?
> > What word "that" refers to above?
> 
> Similar to the version of the running (live), or crashed (dumpfile) kernel.
> The "that" refers to the kernel namelist.  What else can it refer to?

Reading this again:

"If the mapfile argument is used, then the namelist argument must be a kernel namelist of a similar kernel version that was built with the -g switch."

I understand absolutely not a thing. It needs rephrasing.

"If the mapfile argument is used, then the namelist argument must be a kernel namelist of a similar kernel version that was built with the -g switch."


> >        dumpfile
> >               This  is  a  pathname to a kernel memory core dump file.  If the
> >               dumpfile argument is not entered, the session will be invoked on
> >               the  live  system  using  /dev/mem,  which usually requires root
> >               privileges.
> > 
> > Did I get it right that if the dumpfile argument is not entered, all other
> > arguments cannot be entered too (because the format of the arguments are [ [
> > mapfile ] namelist dumpfile ] and thus entering non-option arguments w/o
> > entering dumpfile argument is not possible?
> 
> I'm sorry -- but I'm not sure I understand your question? 
> 
> If no dumpfile argument is entered, then by definition you are
> running on the live system.  And if you are running on the live
> system, you can still enter any, or all, of the other arguments.
> 
> Is that what you're asking?

Yes.

Can you be more clear in the manpage that there are these two different modes of invocation - on a dump file, and on a live system?


> > If yes: I suggest separating the second sentence about from dumpfile
> > description, like this:
> > 
> >        dumpfile
> >               This  is  a  pathname to a kernel memory core dump file.
> 
> Well, actually that's not true either.  To confuse the issue a
> bit more, you are permitted to enter "/dev/mem", "/dev/crash" or
> "/proc/kcore" as a memory source argument, which would override
> the default live memory source.

Then how about this?
'This  is  a  pathname to a kernel memory core dump file, or "/dev/mem", "/dev/crash" or "/proc/kcore" if you want to operate on <what (in each of those cases, if they are different)>'

> 
> Ah, you are saying that crash can be called as
>     crash NAMELIST DUMPFILE
> or as
>     crash [NAMELIST]
> which cannot be fully expressed with []'s?
 
Exactly.

> In cases like this, when you have widely different usage case, 
> people usually resort to documenting them separately instead 
> of trying to cram it into all-encompassing one-line syntax. 
> Example:
> 
> # ln --help
> Usage: ln [OPTION]... [-T] TARGET LINK_NAME   (1st form)
>   or:  ln [OPTION]... TARGET                  (2nd form)
>   or:  ln [OPTION]... TARGET... DIRECTORY     (3rd form)
>   or:  ln [OPTION]... -t DIRECTORY TARGET...  (4th form)
> In the 1st form, create a link to TARGET with the name LINK_NAME.
> In the 2nd form, create a link to TARGET in the current directory.
> In the 3rd and 4th forms, create links to each TARGET in DIRECTORY.
> Create hard links by default, symbolic links with --symbolic.
> When creating hard links, each TARGET must exist.  Symbolic links
> can hold arbitrary text; if later resolved, a relative link is
> interpreted in relation to its parent directory.
> 
> Can you use this approach?

Yes -- that's a very helpful example!

> SYNOPSIS
>     crash [-h [OPT]] [-i FILE] [-d NUM] [-vsS] [MAPFILE] NAMELIST DUMPFILE
>     crash [-h [OPT]] [-i FILE] [-d NUM] [-vsS] [[MAPFILE] NAMELIST]
>     ...<other usages if they are very different>....
 
I like the CAPITAL-LETTER usage with respect to the primary
arguments.  But since I plan to add some additional options,
I also like the way "ln" does it, i.e.:

SYNOPSIS
       ln [OPTION]... [-T] TARGET LINK_NAME   (1st form)
       ln [OPTION]... TARGET                  (2nd form)
       ln [OPTION]... TARGET... DIRECTORY     (3rd form)
       ln [OPTION]... -t DIRECTORY TARGET...  (4th form)

where the list of OPTION's follow in a list.
  
> In RHELx manpage, you probably need to focus on how to use the tool on RHELx,
> not on RHELx-2...

Well, I'm not sure I agree with that...

The crash utility is kernel/RHEL version independent, i.e.,
you can look at a RHEL4 dumpfile on a RHEL6 host, or a RHEL6
dumpfile on a RHEL5 host, etc.  That's precisely what GSS does,
and so it makes sense to try to cover all possibilities, even
for upstream kernels, Fedora kernels, etc.

> Examples tend to help understand typical operations better
>  than long-winded explanations.

Right -- and that could be the best way to handle the coverage
of multiple RHEL versions.

> Reading this again:
> 
> "If the mapfile argument is used, then the namelist argument must be a kernel
> namelist of a similar kernel version that was built with the -g switch."
> 
> I understand absolutely not a thing. It needs rephrasing.
> 
> "If the mapfile argument is used, then the namelist argument must be a kernel
> namelist of a similar kernel version that was built with the -g switch."
 
Maybe I should come at it from a different direction, i.e., something like:

  If the kernel namelist argument is not exactly the same kernel
  as the live/crashed kernel, then the System.map of the original
 live/crashed kernel should be used.

In other words, the -g issue is almost orthogonal to the System.map
discussion...

> Can you be more clear in the manpage that there are these two different modes
> of invocation - on a dump file, and on a live system?

Fair enough.
 
> > Well, actually that's not true either.  To confuse the issue a
> > bit more, you are permitted to enter "/dev/mem", "/dev/crash" or
> > "/proc/kcore" as a memory source argument, which would override
> > the default live memory source.
> 
> Then how about this?
> 'This  is  a  pathname to a kernel memory core dump file, or 
> "/dev/mem", "/dev/crash" or "/proc/kcore" if you want to operate
> on <what (in each of those cases), if they are different)>'

Perhaps I should change it from "DUMPFILE" to "MEMORY-SOURCE", 
and then explain that if it's a live system, the MEMORY-SOURCE
defaults to the most correct device, but that the default device
can be overridden?  It sounds silly to refer to /dev/mem et al
as a "dumpfile". 

Dave

(In reply to comment #5)
> Maybe I should come at it from a different direction, i.e., something like:
> 
>   If the kernel namelist argument is not exactly the same kernel
>   as the live/crashed kernel, then the System.map of the original
>  live/crashed kernel should be used.

Yes, better. Let me try to build on this...

"Usually MAPFILE is not needed. However, sometimes recompiling vmlinux with -g introduces slight offsets to symbols in it. In this case, you can still pass debug vmlinux (one built with -g) as NAMEFILE, but you also need to pass System.map of the *non-debug* vmlinux as MAPFILE. crash will use information from System.map to correct offsets found in NAMEFILE."



> > > Well, actually that's not true either.  To confuse the issue a
> > > bit more, you are permitted to enter "/dev/mem", "/dev/crash" or
> > > "/proc/kcore" as a memory source argument, which would override
> > > the default live memory source.
> > 
> > Then how about this?
> > 'This  is  a  pathname to a kernel memory core dump file, or 
> > "/dev/mem", "/dev/crash" or "/proc/kcore" if you want to operate
> > on <what (in each of those cases), if they are different)>'
> 
> Perhaps I should change it from "DUMPFILE" to "MEMORY-SOURCE", 
> and then explain that if it's a live system, the MEMORY-SOURCE
> defaults to the most correct device, but that the default device
> can be overridden?  It sounds silly to refer to /dev/mem et al
> as a "dumpfile".

Yes. Other possible terms: KERNEL_IMAGE, KERNEL_DUMP.

(In reply to comment #6)
>
> Yes, better. Let me try to build on this...
> 
> "Usually MAPFILE is not needed. However, sometimes recompiling vmlinux with -g
> introduces slight offsets to symbols in it. In this case, you can still pass
> debug vmlinux (one built with -g) as NAMEFILE, but you also need to pass
> System.map of the *non-debug* vmlinux as MAPFILE. crash will use information
> from System.map to correct offsets found in NAMEFILE."
 
I like it!
 
> > Perhaps I should change it from "DUMPFILE" to "MEMORY-SOURCE", 
> > and then explain that if it's a live system, the MEMORY-SOURCE
> > defaults to the most correct device, but that the default device
> > can be overridden?  It sounds silly to refer to /dev/mem et al
> > as a "dumpfile".
> 
> Yes. Other possible terms: KERNEL_IMAGE, KERNEL_DUMP.

Perhaps I should maintain some consistency with the crash whitepaper,
the top listing if you google "crash utility":  

 http://people.redhat.com/anderson/crash_whitepaper#PREREQUISITES

There I use the term "memory image", so maybe I should use MEMORY_IMAGE?

(In reply to comment #7)
> > > Perhaps I should change it from "DUMPFILE" to "MEMORY-SOURCE", 
> > > and then explain that if it's a live system, the MEMORY-SOURCE
> > > defaults to the most correct device, but that the default device
> > > can be overridden?  It sounds silly to refer to /dev/mem et al
> > > as a "dumpfile".
> > 
> > Yes. Other possible terms: KERNEL_IMAGE, KERNEL_DUMP.
> 
> Perhaps I should maintain some consistency with the crash whitepaper,
> the top listing if you google "crash utility":  
> 
>  http://people.redhat.com/anderson/crash_whitepaper#PREREQUISITES
> 
> There I use the term "memory image", so maybe I should use MEMORY_IMAGE?

It's ok

Created attachment 496633 [details]
crash.8 man page


Proposed crash.8 man page for next crash utility upstream release.

View with: groff -mandoc -T ascii crash.8

The new, built-in, "crash -h" output is largely taken from the
new crash.8 man page text.  There's no need to make them drastically
different.  This is what the output would look like:

$ crash -h

USAGE:

  crash [OPTION]... NAMELIST MEMORY-IMAGE  (dumpfile form)
  crash [OPTION]... [NAMELIST]             (live system form)

OPTIONS:

  NAMELIST
    This is a pathname to an uncompressed kernel image (a vmlinux
    file), or a Xen hypervisor image (a xen-syms file) which has
    been compiled with the "-g" option.  If using the dumpfile form,
    a vmlinux file may be compressed in either gzip or bzip2 formats.

  MEMORY-IMAGE
    A kernel core dump file created by the netdump, diskdump, LKCD
    kdump, xendump or kvmdump facilities.

    If a MEMORY-IMAGE argument is not entered, the session will be
    invoked on the live system, which typically requires root privileges
    because of the device file used to access system RAM.  By default, 
    /dev/crash will be used if it exists.  If it does not exist, then 
    /dev/mem will be used; but if the kernel has been configured with 
    CONFIG_STRICT_DEVMEM, then /proc/kcore will be used.  It is permissible
    to explicitly enter /dev/crash, /dev/mem or /proc/kcore.

  mapfile
    If the NAMELIST file is not the same kernel that is running
    (live system form), or the kernel that was running when the system
    crashed (dumpfile form), then the System.map file of the original 
    kernel should be entered on the command line.

  -h [option]
    Without an option argument, display a crash usage help message.
    If the option argument is a crash command name, the help page
    for that command is displayed.  If it is the string "input", a
    page describing the various crash command line input options is
    displayed.  If it is the string "output", a page describing command
    line output options is displayed.  After the help message is 
    displayed, crash exits.

  -s     
    Proceed directly to the "crash>" prompt without displaying any
    version, GPL, or crash initialization data during startup.

  -i file
    Execute the command(s) contained in "file" prior to displaying 
    the "crash>" prompt for interactive user input.

  -d num 
    Set the internal debug level.  The higher the number, the more
    debugging data will be printed when crash initializes and runs.

  -S     
    Use /boot/System.map as the mapfile.

  -e  vi | emacs
    Set the readline(3) command  line editing mode to "vi" or "emacs".  
    The default editing mode is "vi".

  -f     
    Force the usage of a compressed vmlinux file if its original
    name does not start with "vmlinux".

  -k     
    Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo file.

  -t     
    Display the system-crash timestamp and exit.

  -L     
    Attempt to lock all of its virtual address space into memory by
    calling mlockall(MCL_CURRENT|MCL_FUTURE) during initialization.
    If the system call fails, an error message will be displayed,
    but the session continues.

  -c tty-device
    Open the tty-device as the console used for debug messages.

  -p page-size
    If a processor's page size cannot be determined by the dumpfile, 
    and the processor default cannot be used, use page-size.

  -m option=value
  --machdep option=value
    Pass an option and value pair to machine-dependent code.  These
    architecture-specific option/pairs should only be required in
    very rare circumstances:

    X86_64:
      physbase=<physical-address>
      irq_eframe_link=<value>
      max_physmem_bits=<value>
      vm=orig       (pre-2.6.11 virtual memory address ranges)
      vm=2.6.11     (2.6.11 and later virtual memory address ranges)
      vm=xen        (Xen kernel virtual memory address ranges)
      vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
    PPC64:
      vm=orig
      vm=2.6.14     (4-level page tables)
    IA64:
      phys_start=<physical-address>
      init_stack_size=<size>
      vm=4l         (4-level page tables)
    ARM:
      physbase=<physical-address>

  -x     
    Automatically load extension modules from a particular directory.
    The directory is determined by the following order of precedence:

    (1) the directory specified in the CRASH_EXTENSIONS shell 
        environment variable
    (2) /usr/lib64/crash/extensions  (64-bit  architectures)
    (3) /usr/lib/crash/extensions  (32-bit architectures)
    (4) the ./extensions subdirectory of the current directory

  --memory_module modname
    Use the modname as an alternative kernel module to the crash.ko
    module that creates the /dev/crash device.

  --memory_device device
    Use device as an alternative device to the /dev/crash, /dev/mem
    or /proc/kcore devices.

  --no_kallsyms
    Do not use kallsyms-generated symbol information contained within
    kernel module object files.

  --no_modules
    Do not access or display any kernel module related information.

  --no_ikconf
    Do not attempt to read configuration data that was built into
    kernels configured with CONFIG_IKCONFIG.

  --no_data_debug
    Do not verify the validity of all structure member offsets and
    structure sizes that it uses.

  --no_kmem_cache
    Do not initialize the kernel's slab cache infrastructure, and
    commands that use kmem_cache-related data will not work.

  --kmem_cache_delay
    Delay the initialization of the kernel's slab cache infrastructure
    until it is required by a run-time command.

  --readnow
    Pass this flag to the embedded gdb module, which will override
    its two-stage  strategy that it uses for reading symbol tables
    from the NAMELIST.

  --smp  
    Specify that the system being analyzed is an SMP kernel.

  -v
  --version
    Display the version of the crash utility, the version of the
    embedded gdb module, GPL information, and copyright notices.

  --cpus number
    Specify the number of cpus in the SMP system being analyzed.

  --hyper
    Force the session to be that of a Xen hypervisor.

  --p2m_mfn pfn
    When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile
    is typically analyzed with either the Xen hypervisor or the dom0
    kernel.  It is also possible to analyze any of the guest domU
    kernels if the pfn_to_mfn_list_list pfn value of the guest kernel
    is passed on the command line along with its NAMELIST and the 
    dumpfile.

  --xen_phys_start physical-address
    Supply the base physical address of the Xen hypervisor's text
    and static data for older xendump dumpfiles that did not pass
    that information in the dumpfile header.

  --zero_excluded
    If a kdump dumpfile has been filtered to exclude various types of
    non-essential  pages, any attempt to read them will fail.  With
    this flag, reads from any of those pages will return zero-filled
    memory.

  --no_panic
    Do not attempt to find the task that was running when the kernel
    crashed.  Set the initial context to that of the "swapper"  task
    on cpu 0.

  --more 
    Use /bin/more as the command output scroller, overriding the
    default of /usr/bin/less and any settings in either ./.crashrc
    or $HOME/.crashrc.

  --less 
    Use /usr/bin/less as the command output scroller, overriding any
    settings in either ./.crashrc or $HOME/.crashrc.

  --CRASHPAGER
    Use the output paging command defined in the CRASHPAGER shell
    environment variable, overriding any settings in either ./.crashrc 
    or $HOME/.crashrc.

  --no_scroll
    Do not pass run-time command output to any scrolling command.

  --no_crashrc
    Do not execute the commands in either $HOME/.crashrc or ./.crashrc.

  --mod directory
    When loading the debuginfo data of kernel modules with the "mod -S"
    command, search for their object files in directory instead of in 
    the standard location.

  --reloc size
    When analyzing live x86 kernels configured with a CONFIG_PHYSICAL_START 
    value that is larger than its CONFIG_PHYSICAL_ALIGN value, then it will
    be necessary to enter a relocation size equal to the difference between
    the two values.

  --minimal
    Bring up a session that is restricted to the log, dis, rd, sym,
    eval, set and exit commands.  This option may provide a way to
    extract some minimal/quick information from a corrupted or truncated
    dumpfile, or in situations where one of the several kernel subsystem 
    initialization routines would abort the crash session.

  --kvmhost [32|64]
    When examining an x86 KVM guest dumpfile, this option specifies
    that the KVM host that created the dumpfile was an x86 (32-bit)
    or an x86_64 (64-bit) machine, overriding the automatically
    determined value.

  --kvmio <size>
    override the automatically-calculated KVM guest I/O hole size.

FILES:

  .crashrc
    Initialization commands.  The file can be located in the user's
    HOME directory and/or the current directory.  Commands found in
    the .crashrc file in the HOME directory are executed before
    those in the current directory's .crashrc file.

ENVIRONMENT VARIABLES:

  EDITOR 
    Command input is read using readline(3).  If EDITOR is set to
    emacs or vi then suitable keybindings are used.  If EDITOR is
    not set, then vi is used.  This can be overridden by "set vi" or
    "set emacs" commands located in a .crashrc file, or by entering
    "-e emacs" on the crash command line.

  CRASHPAGER
    If CRASHPAGER is set, its value is used as the name of the program
    to which command output will be sent.  If not, then command output
    output is sent to "/usr/bin/less -E -X" by default.

  CRASH_MODULE_PATH
    Specifies an alternative directory tree to search for kernel
    module object files.

  CRASH_EXTENSIONS
    Specifies a directory containing extension modules that will be
    loaded automatically if the -x command line option is used.

$

Created attachment 496635 [details]
crash.8 man page

Fixed a few textual errors in the first attachment.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
The crash.8 man page and the associated built-in "crash -h" output have been re-written. The crash.8 man page now clarifies the required invocation options, adds all of the rarely-used command line options that have proliferated over the years, and updates the ENVIRONMENT variables section. The "crash -h" output now closely mimics the relevant parts of the crash.8 man page.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1648.html