Bug 834272 - bt command should print all hex values with 0x prefix
bt command should print all hex values with 0x prefix
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: crash (Show other bugs)
6.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Dave Anderson
Red Hat Kernel QE team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-21 07:57 EDT by Stanislav Kozina
Modified: 2013-03-03 20:35 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-08-16 11:32:25 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Stanislav Kozina 2012-06-21 07:57:34 EDT
Description of problem:

bt command prints many values in hex, however they all miss 0x prefix

Version-Release number of selected component (if applicable):

$ rpm -qf `which crash`
crash-5.1.8-1.el6.x86_64
megatron$ rpm -qf `which crash`
crash-6.0.5-0.x86_64

How reproducible:

Always.

Steps to Reproduce:
1. bt
2.
3.
  
Actual results:

Heavily shortened:
crash> bt
PID: 26899  TASK: ffff81020b54e100  CPU: 2   COMMAND: "java"
 #0 [ffff8101f95eb860] crash_kexec at ffffffff800b0938
    RIP: ffffffff8832fd52  RSP: ffff8101f95ebb08  RFLAGS: 00010206

Expected results:

Heavily shortened:
crash> bt
PID: 26899  TASK: 0xffff81020b54e100  CPU: 2   COMMAND: "java"
 #0 [0xffff8101f95eb860] crash_kexec at 0xffffffff800b0938
    RIP: 0xffffffff8832fd52  RSP: 0xffff8101f95ebb08  RFLAGS: 0x00010206

Additional info:

I expect 0x prefix in front of values of:
 - task pointer
 - stack addresses
 - return addresses
 - all register values
Comment 1 Dave Anderson 2012-06-21 09:20:34 EDT
> I expect 0x prefix in front of values of:
> - task pointer
> - stack addresses
> - return addresses
> - all register values

No chance.

As I mentioned in the other BZ's, a foundational tenet of the
crash utility is the avoidance of the 0x whenever possible, both for
user input and command output.
Comment 3 Dave Anderson 2012-06-21 16:51:37 EDT
> What is the reason behing removing 0x prefixes from hex numbers.
> I think it is very good "norm" set up for several decades, enabling
> people to quickly identify numeric basis. How do you help people to
> identify correct numerical base?
>
> e.g.
>
> Is 123456 Dec/Hex/Oct ?

As I indicated in another of these bugzillas, w/respect to the crash
utility's command output, it should be readily evident whether a field is
hexadecimal or decimal.  Virtual addresses are obviously hexadecimal, 
most other fields are decimal, etc, and whenever there is ambiguity in
any command's output the "0x" is in fact displayed:

  crash> list_head -o
  struct list_head {
     [0] struct list_head *next;
     [8] struct list_head *prev; 
  }
  SIZE: 16
  crash> hex
  output radix: 16 (hex)
  crash> list_head -o
  struct list_head {
     [0x0] struct list_head *next;
     [0x8] struct list_head *prev; 
  }
  SIZE: 0x10
  crash> 

Now, with respect to structure displays from the embedded gdb, it's
the gdb code that makes that decision, based upon its "output-radix"
environment variable.  It defaults to 10, but can be flipped back and
forth by use of the crash utility's "set radix 10" and "set radix 16"
commands, or the "hex" and "dec" built-in aliases.  In addition, all
commands that use gdb's structure output have their own -x and -d 
options that can be used to override the current default:

  crash> list_head -o
  struct list_head {
     [0] struct list_head *next;
     [8] struct list_head *prev;
  }
  SIZE: 16
  crash> list_head -ox
  struct list_head {
     [0x0] struct list_head *next;
     [0x8] struct list_head *prev;
  }
  SIZE: 0x10
  crash> 

Off the top of my head, the struct, vm, list, net, mach, dis and
task commands all have -x and -d options as of crash-6.0.7.
Comment 4 Stanislav Kozina 2012-06-22 07:12:16 EDT
> As I indicated in another of these bugzillas, w/respect to the crash
> utility's command output, it should be readily evident whether a field is
> hexadecimal or decimal.  Virtual addresses are obviously hexadecimal, 
> most other fields are decimal, etc,

The problem here - crash suppose that _every_ pointer type contains virtual address, and every non-pointer contains logicaly decimal value.
But this is not true - pretty often some "void *" fields contains some flags instead, and even vice versa. And then you have to know the exact type of such item in the structure to be able to understand how it's value is displayed.

For example timer_list structure (part of net_device):

      suspend_timer = {
<SNIP!>
        data = 18446612145505666128, 
        base = 0xffff880318328000, 

A bit unconvenient to read I believe.

and whenever there is ambiguity in
> any command's output the "0x" is in fact displayed:

Yes, thanks a lot for that!
But still you have to be prepared - by default you have to check base of every single value you see.

> Now, with respect to structure displays from the embedded gdb, it's
> the gdb code that makes that decision, based upon its "output-radix"
> environment variable.  It defaults to 10, but can be flipped back and
> forth by use of the crash utility's "set radix 10" and "set radix 16"
> commands, or the "hex" and "dec" built-in aliases.

I have 'set radix hex' in my crashrc, and I found crash very hardly usable without that (the other command in my .crashrc is 'set emacs' - the same applies).
I am telling about this all guys around - and till now all of them were graceful for both commands, because they were not aware of it.
In fact - it took me about 20 minutes to find this out in the man page.

In addition, all
> commands that use gdb's structure output have their own -x and -d 
> options that can be used to override the current default:

I am using -x anywhere possible by default.
Despite my 'set radix hex', I don't use 'struct' command without -x :-)

> Off the top of my head, the struct, vm, list, net, mach, dis and
> task commands all have -x and -d options as of crash-6.0.7.

None of vm, list, net, mach, dis (here I have no idea what -x actualy does, but not what you would expect) and task have '-x' option. Only struct has.
Comment 5 Dave Anderson 2012-06-22 08:42:46 EDT
> I have 'set radix hex' in my crashrc, and I found crash very hardly 
> usable without that (the other command in my .crashrc is 'set emacs'
> - the same applies).

"hardly usable?" with the gdb default?  Come on...

I can say just the opposite -- there are just as many structure members
that make far more sense when expressed in decimal.

Anyway, that's what the .crashrc file affords you, the capability of
having it your way.  Or you can just enter "hex" one time.  Although
apparently that must be a huge burden to you.  

Man, you just can't please all the people all of the time...

> None of vm, list, net, mach, dis (here I have no idea what -x actualy does,
> but not what you would expect) and task have '-x' option. Only struct has.

Right -- you're using an older version of crash.
Comment 6 Stanislav Kozina 2012-06-22 10:09:24 EDT
> "hardly usable?" with the gdb default?  Come on...

My apologizes - I tend to exxagerate my opinions.
But facts - I really don't like gdb, don't consider it ideal. It's too much focused on source code debugging - and I believe that for real CDA we need much more to focus on instructions, raw content of memory.
MDB/KMDB is my debugger.

But at least you have to admit - vi mode is definitely not default option anywhere, nor even in gdb;-)

> I can say just the opposite -- there are just as many structure members
> that make far more sense when expressed in decimal.

You can - but decimal values are readable even in hex format, and if you need the precise value, you can convert it. This is usualy needed only sometimes - such as timeout values for example.

On the other hand - pointers in dec would be absolutely unreadable, so that's why I believe keeping all values in hex is better option, better than having every value in the base which fits it the best.

> Anyway, that's what the .crashrc file affords you, the capability of
> having it your way.  Or you can just enter "hex" one time.  Although
> apparently that must be a huge burden to you.

Thanks a lot for crashrc, definitely good thing.
You can write 'hex' once, but if you do it every single time - you start searching how to do it automaticaly. That's how the programers are supposed to work, right? :-)

> Man, you just can't please all the people all of the time...

Definitely.
Please don't take me badly - I definitely appreciate your work on crash, it's a good tool, and without it our work would be _really_ hard. I am happy user of it since I started to work with Linux. There were just few things which I missed from MDB, that's why I suggested those in bugzilla.

I believe that's it's good that we actualy lead all the discussion here, because all arguments and decisions could be saved for future generations.

> Right -- you're using an older version of crash.

Correct:

# crash --version
crash 5.1.8-1.el6

I see my latest rhel-6.2 is not current enough. Are there any plans to update crash in rhel-6.2?
Comment 7 Dave Anderson 2012-06-22 10:50:11 EDT
> But facts - I really don't like gdb, don't consider it ideal.

Believe me -- I also have a love-hate relationship with gdb.

However, I come from a UNIX background, and used to work on
the AT&T/SVR4 version of the crash utility back in the early 80's,
and it was limited/tied to a particular kernel.  As such the code
used to kludge around gaining access/knowledge of kernel data 
structures by opening up the kernel header files to user-space 
by #define'ing _KERNEL_; but in doing so everything was hardwired 
to a particular kernel source tree.

Anyway, in the late 90's there was no kernel core dump or crash
utility in Linux, but there was "gdb /proc/kcore".  But to me,
that was pretty much useless because gdb is designed for user 
programs, and obviously has no smarts w/respect to the kernel.
On the other hand, gdb was obviously very powerful w/respect 
to data structure information, and also didn't have to be tied 
to a particular kernel version because it got all its information
dynamically from the vmlinux file's debuginfo data.  Not to mention
that you get other goodies like text disassembly, line numbers, 
being able to add the debuginfo data for kernel modules, etc.  

So I married the two concepts, i.e., a crash utility that is
kernel-centric, but with the power of gdb underneath the hood.

Anyway, in 1999, SGI came out with "lcrash" and Mission Critical
Linux (via me) came out with "crash" within a couple weeks of
one another.  They both competed for a few years, but eventually 
"lcrash" fell by the wayside.  One reason it did fail was because
they originally built the binary inside the kernel tree, and it 
was also tied to that particular kernel.  And it also tried to 
mimic gdb by putting in their own structure-handling stuff, 
but it just didn't measure up.

That being said, the gdb code is a rats-nest to understand,
and I hate having to deal with it...

> Please don't take me badly

And please excuse me if I come off as a curmudgeon (although I 
actually am one...)

> But at least you have to admit - vi mode is definitely not 
> default option anywhere, nor even in gdb;-)

I'll NEVER admit that!   ;-)

I'm old-school, man, I *hate* emacs...

> I see my latest rhel-6.2 is not current enough. Are there any 
> plans to update crash in rhel-6.2?

Unlikely.  There has been exactly one Z-stream release, back in
RHEL5.3.  For active RHEL releases, there is typically an errata
released with each RHEL X.Y update.  There is a RHEL6.3 errata, 
and I expect one with RHEL6.4 as well.  But for example, the RHEL6.3
errata version is crash-6.0.4-2.el6, and I already consider
that version obsolete.

So if you're a frequent user, I highly recommend that you
frequently download the latest:

  http://people.redhat.com/anderson

  $ wget http://people.redhat.com/anderson/crash-<release>.tar.gz
  ...
  $ cd crash-<release>
  $ make
  ...
  $ make install

I typically do a new release about once-a-month.  That is driven
by fixes, enhancements, and having to deal with the constantly
shifting sands of the underlying kernel.

You'll get notification of new releases on the crash-utility mailing
list.
Comment 8 Stanislav Kozina 2012-06-26 07:57:12 EDT
> Believe me -- I also have a love-hate relationship with gdb.

I do ;-)

> However, I come from a UNIX background, and used to work on
> the AT&T/SVR4 version of the crash utility back in the early 80's,
> and it was limited/tied to a particular kernel.  As such the code
> used to kludge around gaining access/knowledge of kernel data 
> structures by opening up the kernel header files to user-space 
> by #define'ing _KERNEL_; but in doing so everything was hardwired 
> to a particular kernel source tree.

Yes, this is too much hardwired. That's why I love mdb/kmdb with type format in CTF;-)

> Anyway, in the late 90's there was no kernel core dump or crash
> utility in Linux, but there was "gdb /proc/kcore".  But to me,
> that was pretty much useless because gdb is designed for user 
> programs, and obviously has no smarts w/respect to the kernel.
> On the other hand, gdb was obviously very powerful w/respect 
> to data structure information, and also didn't have to be tied 
> to a particular kernel version because it got all its information
> dynamically from the vmlinux file's debuginfo data.  Not to mention
> that you get other goodies like text disassembly, line numbers, 
> being able to add the debuginfo data for kernel modules, etc.  

I know that few years ago it was very hard to do any post-mortem CDA on Linux.
That's I know how great crash tool is, and I am very graceful for it, exactly as I wrote.
But still - because crash is tighted to gdb, is still has some burder with.
I don't say crash is not good - I just believe it can be better;-)

> So I married the two concepts, i.e., a crash utility that is
> kernel-centric, but with the power of gdb underneath the hood.

Great idea, quite a good result, compared to the cost of implementation;-)

> Anyway, in 1999, SGI came out with "lcrash" and Mission Critical
> Linux (via me) came out with "crash" within a couple weeks of
> one another.  They both competed for a few years, but eventually 
> "lcrash" fell by the wayside.  One reason it did fail was because
> they originally built the binary inside the kernel tree, and it 
> was also tied to that particular kernel.  And it also tried to 
> mimic gdb by putting in their own structure-handling stuff, 
> but it just didn't measure up.

Huh, I would probably not like specific debuger bound to given kernel version.
But despite thanks for mentioned that one.

> That being said, the gdb code is a rats-nest to understand,
> and I hate having to deal with it...

+1 ;-)

> And please excuse me if I come off as a curmudgeon (although I 
> actually am one...)

I am most likely the one who brought discussion to a bit more emotive level, so apologizes are on my side;-)

> > But at least you have to admit - vi mode is definitely not 
> > default option anywhere, nor even in gdb;-)
> 
> I'll NEVER admit that!   ;-)
> 
> I'm old-school, man, I *hate* emacs...

Actually, emacs is older than vi, sorry, mate... So you are not _so_ old school:-) I realized that when I met a guy who was using emacs since the beginning (1975?) and said "I have never get used to that modern vi(m) fancy stuff"...

But I completely agree with you - I am also vi fan, and I hate emacs as well.
What I said is that vi *mode* (in shell) is not default option, it was nothing against the great editor;-)

> > I see my latest rhel-6.2 is not current enough. Are there any 
> > plans to update crash in rhel-6.2?
> 
> Unlikely.  There has been exactly one Z-stream release, back in
> RHEL5.3.  For active RHEL releases, there is typically an errata
> released with each RHEL X.Y update.  There is a RHEL6.3 errata, 
> and I expect one with RHEL6.4 as well.  But for example, the RHEL6.3
> errata version is crash-6.0.4-2.el6, and I already consider
> that version obsolete.

I see. Was crash-6.0.4-2.el6 current at least during the rhel-6.3 development window? Or are we really integrating obsolete stuff?

> So if you're a frequent user, I highly recommend that you
> frequently download the latest:

Will do. Thanks a lot.

> I typically do a new release about once-a-month.  That is driven
> by fixes, enhancements, and having to deal with the constantly
> shifting sands of the underlying kernel.
> 
> You'll get notification of new releases on the crash-utility mailing
> list.

Great, thanks a lot!
Comment 9 Dave Anderson 2012-06-26 08:44:14 EDT
> > Unlikely.  There has been exactly one Z-stream release, back in
> > RHEL5.3.  For active RHEL releases, there is typically an errata
> > released with each RHEL X.Y update.  There is a RHEL6.3 errata, 
> > and I expect one with RHEL6.4 as well.  But for example, the RHEL6.3
> > errata version is crash-6.0.4-2.el6, and I already consider
> > that version obsolete.
>
> I see. Was crash-6.0.4-2.el6 current at least during the rhel-6.3 
> development > window? Or are we really integrating obsolete stuff?

Well, perhaps "obsolete" is a bit strong.  It's just that with every
~monthly release there is always new stuff that would be nice to have.
I would guess that there are few packages that undergo change at such
a rapid pace.

Speaking of change, I've got a new "bt -s [-xd]" option queued up
for crash-6.0.8 that shows symbol+offsets -- in the current output
radix or as overridden with -x|-d.  I went ahead and did all 
of the arches; I don't think anybody will be upset about it...
Comment 10 RHEL Product and Program Management 2012-07-10 04:52:11 EDT
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 11 Stanislav Kozina 2012-07-10 08:18:18 EDT
> Well, perhaps "obsolete" is a bit strong.  It's just that with every
> ~monthly release there is always new stuff that would be nice to have.
> I would guess that there are few packages that undergo change at such
> a rapid pace.

Yes, I see. And our release schedule for rhel line of products is simply a bit slow for crash:-)

> Speaking of change, I've got a new "bt -s [-xd]" option queued up
> for crash-6.0.8 that shows symbol+offsets -- in the current output
> radix or as overridden with -x|-d.  I went ahead and did all 
> of the arches; I don't think anybody will be upset about it...

Thanks a lot for this, I am looking forward to use it most of the time:-)
Comment 12 RHEL Product and Program Management 2012-07-10 19:10:43 EDT
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.

Note You need to log in before you can comment on or make changes to this bug.