Can I get the vmlinux and rhel6.core pair please?

Thanks Jeff, I'm copying them now. 

Also it would be helpful if you could run crash live on the guest 
(running the same kernel), and then post the output of this command:

 crash> help -m | grep phys_base

Also, this is of interest:

> crash: /tmp/rhel6.core: fwrite: No such file or directory

The fwrite() is actually writing to a temporary file created with
tmpfile(), and should never fail.  Is it possible that the filesystem
containing /tmp is full?

Ok, it fired up just fine on the live guest:

crash> help -m | grep phys_base  
                phys_base: 0


.../tmp is just a subdir of root on this guest and has tons of free space:

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      28423176   5958696  21020640  23% /

...permissions should be ok:
# stat /tmp
  File: `/tmp'
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: fd00h/64768d	Inode: 915713      Links: 3
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2010-05-18 09:36:48.925417562 -0400
Modify: 2010-05-18 09:37:10.724391881 -0400
Change: 2010-05-18 09:37:10.724391881 -0400

...I also put selinux in permissive mode temporarily to see if that was affecting it, but it didn't behave any differently.

Hmmm, works for me using my 5.0.3-plus-a-bunch-patches, but many of
the patches are KVM updates:

$ /var/CVS/crash/crash burn/2.6.32-26.el6.jtltest.005_virsh_dump/rhel6.core  burn/2.6.32-26.el6.jtltest.005_virsh_dump/vmlinux

crash 5.0.3p11
Copyright (C) 2002-2010  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.0                               
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

      KERNEL: burn/2.6.32-26.el6.jtltest.005_virsh_dump/vmlinux
    DUMPFILE: burn/2.6.32-26.el6.jtltest.005_virsh_dump/rhel6.core
        CPUS: 4
        DATE: Tue May 18 07:51:25 2010
      UPTIME: 00:21:54
LOAD AVERAGE: 0.00, 0.02, 0.00
       TASKS: 133
    NODENAME: dhcp231-227.rdu.redhat.com
     RELEASE: 2.6.32-26.el6.jtltest.005.x86_64.debug
     VERSION: #1 SMP Mon May 17 11:05:55 EDT 2010
     MACHINE: x86_64  (2992 Mhz)
      MEMORY: 1 GB
       PANIC: ""
         PID: 0
     COMMAND: "swapper"
        TASK: ffffffff817829e0  (1 of 4)  [THREAD_INFO: ffffffff8175e000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)
     WARNING: panic task not found

crash> 

Can you try this with your version of crash:

 # crash --machdep phys_base=0 vmlinux rhel6.core

> Can you try this with your version of crash:
>
> # crash --machdep phys_base=0 vmlinux rhel6.core 

I tried it with crash-5.0.3, and using the --machdep argument 
works for me:

$ crash burn/2.6.32-26.el6.jtltest.005_virsh_dump/rhel6.core  burn/2.6.32-26.el6.jtltest.005_virsh_dump/vmlinux --machdep phys_base=0

crash 5.0.3
Copyright (C) 2002-2010  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: setting phys_base to: 0x0                 

GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

      KERNEL: burn/2.6.32-26.el6.jtltest.005_virsh_dump/vmlinux
    DUMPFILE: burn/2.6.32-26.el6.jtltest.005_virsh_dump/rhel6.core
        CPUS: 4
        DATE: Tue May 18 07:51:25 2010
      UPTIME: 00:21:54
LOAD AVERAGE: 0.00, 0.02, 0.00
       TASKS: 133
    NODENAME: dhcp231-227.rdu.redhat.com
     RELEASE: 2.6.32-26.el6.jtltest.005.x86_64.debug
     VERSION: #1 SMP Mon May 17 11:05:55 EDT 2010
     MACHINE: x86_64  (2992 Mhz)
      MEMORY: 1 GB
       PANIC: ""
         PID: 0
     COMMAND: "swapper"
        TASK: ffffffff817829e0  (1 of 4)  [THREAD_INFO: ffffffff8175e000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)
     WARNING: panic task not found

crash> 

I *think* this workaround should also work with the RHEL6 version
of crash, but I don't have a RHEL6 x86_64 on-hand at the moment,
and I can't test it just yet.

> # crash --machdep phys_base=0 vmlinux rhel6.core 

Yep, works for me too. Thanks for the workaround!

OK, I know what the fix is that is addressed by the --machdep workaround.

The original failure to write to the tmpfile() issue is perplexing though.
It's creating a temporary file that maps kernel-memory-to-dumpfile-offset
because KVM saveVM files are not designed to be "random-access", so the
whole damn file needs to be "scanned" during initialization to find out
where things are.  Anyway, if the creation of that tmpfile fails, the crash
session probably ought to be killed immediately.  But I never figured it
would ever fail, and I currently just let it print an error message and
continue.

So I'm curious -- if you reproduce the original issue, i.e., where your
rhel6.core file is located in /tmp, do you still see the "fwrite" error
if you do *not* use the --machdep workaround?

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Actually, when I tested it, I used my original command and just added the option:

# crash --machdep phys_base=0 /usr/lib/debug/lib/modules/2.6.32-26.el6.jtltest.005.x86_64.debug/vmlinux /tmp/rhel6.core 

...that's what worked fine.

Right -- but can you try it again *without* the --machdep workaround to see
if you can reproduce this again: 

> crash: /tmp/rhel6.core: fwrite: No such file or directory

NEEDINFO re: comment #13

Oh yes:

# crash /usr/lib/debug/lib/modules/2.6.32-26.el6.jtltest.005.x86_64.debug/vmlinux /tmp/rhel6.core

...
crash: /tmp/rhel6.core: fwrite: No such file or directory
crash: read error: kernel virtual address: ffffff81905bc0ff  type: "possible"
WARNING: cannot read cpu_possible_map
crash: /tmp/rhel6.core: fwrite: No such file or directory
crash: read error: kernel virtual address: ffffff81905fc0ff  type: "present"
WARNING: cannot read cpu_present_map
crash: /tmp/rhel6.core: fwrite: No such file or directory
crash: read error: kernel virtual address: ffffff81905dc0ff  type: "online"
WARNING: cannot read cpu_online_map
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/lib/modules/2 and /tmp/rhel6.core do not match!
...

...but this is pretty much the same thing I did when I originally reported it. I'm not doing anything different, so maybe I'm missing the point of what you're asking?

> ...but this is pretty much the same thing I did when I originally
> reported it.  I'm not doing anything different, so maybe I'm missing
> the point of what you're asking?

Because I couldn't understand the "fwrite" errors on the tmpfile()-
generated file...

But now I see that it's a cut-and-paste error - that I have also 
subsequently fixed.

I have two functions that deal with the tmpfile() memory map, a "store"
function that's only used during that "scanning..." part early on, and
then a "load" function that's used during every read of the dumpfile.
And the "fwrite" error message that's in the "load" function was 
cut-and-pasted from the "store" function:

int
load_memfile_offset(uint64_t physaddr, off_t *entry_ptr)
{
        if (fseek(kvm->mem, MEMFILE_OFFSET(physaddr), SEEK_SET) < 0) {
                error(INFO, "%s: fseek: %s\n", pc->dumpfile, strerror(errno));
                return SEEK_ERROR;
        }

        if (fread((entry_ptr), sizeof(off_t), 1, kvm->mem) != 1) {
                error(INFO, "%s: fwrite: %s\n", pc->dumpfile, strerror(errno));
                return READ_ERROR;
        }

        return 0;
}

So it's actually failing a read, probably due to a miscalculation of
a physical address based upon the faulty "phys_base" that's being used.

So it's a false alarm on my part -- please forgive me...

Thanks Jeff

No worries. Glad you found the bug.

QA assist:

Using the supplied dumpfile, crash 5.0.0-14.el6 fails to initialize:

  # crash vmlinux rhel6.core
  
  crash 5.0.0-14.el6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0                               
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
  crash: rhel6.core: fwrite: No such file or directory
  crash: read error: kernel virtual address: ffffff81905bc0ff  type: "possible"
  WARNING: cannot read cpu_possible_map
  crash: rhel6.core: fwrite: No such file or directory
  crash: read error: kernel virtual address: ffffff81905fc0ff  type: "present"
  WARNING: cannot read cpu_present_map
  crash: rhel6.core: fwrite: No such file or directory
  crash: read error: kernel virtual address: ffffff81905dc0ff  type: "online"
  WARNING: cannot read cpu_online_map
  WARNING: cannot read linux_banner string
  crash: vmlinux and rhel6.core do not match!
  
  Usage:
    crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
  
  Enter "crash -h" for details.
  #
  
Crash version 5.0.0-15.el6 comes up as expected:
  
  # crash vmlinux rhel6.core 
  
  crash 5.0.0-15.el6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0                               
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
        KERNEL: vmlinux                           
      DUMPFILE: rhel6.core
          CPUS: 4
          DATE: Tue May 18 07:51:25 2010
        UPTIME: 00:21:54
  LOAD AVERAGE: 0.00, 0.02, 0.00
         TASKS: 133
      NODENAME: dhcp231-227.rdu.redhat.com
       RELEASE: 2.6.32-26.el6.jtltest.005.x86_64.debug
       VERSION: #1 SMP Mon May 17 11:05:55 EDT 2010
       MACHINE: x86_64  (2992 Mhz)
        MEMORY: 1 GB
         PANIC: ""
           PID: 0
       COMMAND: "swapper"
          TASK: ffffffff817829e0  (1 of 4)  [THREAD_INFO: ffffffff8175e000]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)
       WARNING: panic task not found
  
  crash>

I hit a 'initialization failed' on the virsh-dumped KVM guest core, which comes from kernel 2.6.32-30.el6.x86_64:
[root@rhel6 images]# crash -d 99 /usr/lib/debug/lib/modules/2.6.32-30.el6.x86_64/vmlinux rhel6.core 

crash 5.0.0-16.el6
Copyright (C) 2002-2010  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
rhel6.core: QEMU_VM_FILE_MAGIC
please wait... (scanning KVM dumpfile)
                                                
crash: rhel6.core: initialization failed

This is maybe a libvirt bug?

The core is here: http://lacrosse.corp.redhat.com/~phan/kvm_guest_core/rhel6.core

I cannot access the dumpfile:

# wget http://lacrosse.corp.redhat.com/~phan/kvm_guest_core/rhel6.core 
--09:06:55--  http://lacrosse.corp.redhat.com/~phan/kvm_guest_core/rhel6.core
Resolving lacrosse.corp.redhat.com... 10.11.255.147
Connecting to lacrosse.corp.redhat.com|10.11.255.147|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
09:06:55 ERROR 403: Forbidden.

(In reply to comment #21)
> I hit a 'initialization failed' on the virsh-dumped KVM guest core, which comes
> from kernel 2.6.32-30.el6.x86_64:
> [root@rhel6 images]# crash -d 99
> /usr/lib/debug/lib/modules/2.6.32-30.el6.x86_64/vmlinux rhel6.core 
> 
> crash 5.0.0-16.el6
> Copyright (C) 2002-2010  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for details.
> 
> rhel6.core: QEMU_VM_FILE_MAGIC
> please wait... (scanning KVM dumpfile)
> 
> crash: rhel6.core: initialization failed
> 
> This is maybe a libvirt bug?    

This bug is a new and different issue than the 5.0.0-15.el6 fix
that addressed this BZ.  This new issue is being tracked here:
 
 Bug 597187 - guest core by virsh dump cannot be analysed by crash:
              initialization failed 
 https://bugzilla.redhat.com/show_bug.cgi?id=597187

Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.