Bug 489147

Summary: crash fail to read data above 0xf0000000 from XEN guest dump core
Product: Red Hat Enterprise Linux 5 Reporter: kerdosa
Component: crashAssignee: Dave Anderson <anderson>
Status: CLOSED NOTABUG QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: low    
Version: 5.2CC: kerdosa, qcai
Target Milestone: rc   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-04-14 15:25:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description kerdosa 2009-03-08 07:01:20 UTC
Description of problem: crash rd command fail to read data if address is above 0xf0000000.


Version-Release number of selected component (if applicable):
RHEL 5.2 with XEN, Guest OS: kernel 2.6.18-128.el5xen, crash-4.0-5.0.3.el5.

How reproducible:
In RHEL 5.2 XEN, get core dump of guest OS and run crash. In crash, try to read data (rd command) above address 0xf0000000.


Steps to Reproduce:
1. dump core of guest domain by xm dump-core command
2. run crash
3. crash> HYPERVISOR_shared_info
HYPERVISOR_shared_info = $16 = (shared_info_t *) 0xf545c000
crash> rd 0xf545c000
rd: read error: kernel virtual address: f545c000  type: "32-bit KVADDR"
crash>
  
Actual results:
rd: read error: kernel virtual address: f545c000  type: "32-bit KVADDR"
crash>

Expected results:
It should show the data of struct shared_info since 0xf545c000 is (shared_info_t *).

Additional info:

Comment 1 Dave Anderson 2009-03-09 13:31:59 UTC
> It should show the data of struct shared_info since 0xf545c000 is
> (shared_info_t *).

The capability of accessing "fixmap" hypervisor address space has never
been available, either running on the live system or against a vmcore.

The closest you can come to it is to read it as "machine" memory.

  crash> p *xen_start_info
  $10 = {
    magic = "xen-3.1-x86_64\000\000\000\000\000\000\000\000\000\000\000\000\000  
    \000\000\000\000", 
    nr_pages = 0x32a7a, 
    shared_info = 0xfe3000, 
    flags = 0x3,
    store_mfn = 0x3ca52, 
    store_evtchn = 0x7, 
    ...

  crash> rd -m 0xfe3000 <count>
  ...

I think it may be possible to access it if the pseudo-physical address
associated with machine address 0xfe3000 can be determined, but there's
no "m2p" functionality available in the crash utility, because with that
value, a "ptov" could be done on the pseudo-physical address to come up
with a unity-mapped kernel virtual address.  Then again, if the pseudo
physical address were above "highmem", that wouldn't work on a 32-bit x86.

Comment 2 Dave Anderson 2009-03-09 14:49:25 UTC
> I think it may be possible to access it if the pseudo-physical address
> associated with machine address 0xfe3000 can be determined, but there's
> no "m2p" functionality available in the crash utility ...

No, as it turns out, that won't work either, because there is no entry in
the kernel's p2m table for that machine address.  That's why the read fails.

Comment 3 Dave Anderson 2009-03-09 14:56:30 UTC
Note also that fixmap virtual addresses have never been readily accessible
from the crash utility -- in normal kernels (as well as in xen kernels).

Comment 4 kerdosa 2009-03-09 15:51:13 UTC
> The closest you can come to it is to read it as "machine" memory.
> 
>   crash> p *xen_start_info
>   $10 = {
>     magic = "xen-3.1-x86_64\000\000\000\000\000\000\000\000\000\000\000\000\000 
>     \000\000\000\000", 
>     nr_pages = 0x32a7a, 
>     shared_info = 0xfe3000, 
>     flags = 0x3,
>     store_mfn = 0x3ca52, 
>     store_evtchn = 0x7, 
>     ...
> 
>   crash> rd -m 0xfe3000 <count>
>   ...

First thank you very much for making crash available. it's excellent tool, I use it everyday.
I tried above command, but rd failed again.
crash> p *xen_start_info
$4 = {
  magic = "xen-3.0-x86_32p\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
  nr_pages = 0x60000,
  shared_info = 0xbd0000,
  flags = 0x0,
------------
crash> rd -m 0xbd0000 8
rd: read error: xen machine address: bd0000  type: "32-bit XENMACHADDR"

Comment 5 kerdosa 2009-03-09 15:55:13 UTC
(In reply to comment #3)
> Note also that fixmap virtual addresses have never been readily accessible
> from the crash utility -- in normal kernels (as well as in xen kernels).  

The xen kernel sets up page tables for fixmap address. Then I think crash utility can read it based on page tables. Isn't crash tool use page tables to decode a kernel virtual address(0xc000,0000 - 0xffffffff in install 32bit)?

Comment 6 Dave Anderson 2009-03-09 17:07:10 UTC
> crash> p *xen_start_info
> $4 = {
>   magic =
> "xen-3.0-x86_32p\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
>   nr_pages = 0x60000,
>   shared_info = 0xbd0000,
>   flags = 0x0,
> ------------
> crash> rd -m 0xbd0000 8
> rd: read error: xen machine address: bd0000  type: "32-bit XENMACHADDR" 

If that's the case, then the page associated with machine address bd0000
is not included in the vmcore file at all.  There's nothing that can be
done about that.  (I was able to access it with an x86_64 xen dump)

> The xen kernel sets up page tables for fixmap address. Then I think crash
> utility can read it based on page tables. Isn't crash tool use page tables to
> decode a kernel virtual address(0xc000,0000 - 0xffffffff in install 32bit)?  

For xen kernels, the virtual address is first translated to a pseudo-physical
address, and then the vmcore's m2p notes section is accessed to determine which
machine address that the pseudo-physical address corresponds to.  In your
case, the page was not included in the dumpfile.