Bug 608779

Summary: Implement support for RHEL6 pvops/xen kernels as RHEL5 PV guests (live and xendump)
Product: Red Hat Enterprise Linux 6 Reporter: Dave Anderson <anderson>
Component: crashAssignee: Dave Anderson <anderson>
Status: CLOSED CURRENTRELEASE QA Contact: Dave Anderson <anderson>
Severity: urgent Docs Contact:
Priority: low    
Version: 6.0CC: clalance, ddutile, drjones, lwang, phan, qcai, yuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: crash-5.0.0-21.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-11-10 20:04:25 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
anaconda-tb-jldPrQ8bYeA8.xml none

Description Dave Anderson 2010-06-28 16:19:31 UTC 
Description of problem:

The crash utility fails with RHEL6 2.6.32-27 kernels when they are run 
as PV guests on a RHEL6 xen host.


Version-Release number of selected component (if applicable):
crash-5.0.0-19.el6
2.6.32-37.el6 kernel as RHEL5 PV guest

How reproducible:
Always.

Steps to Reproduce:
1. run crash on live system
2. run crash on a "xm -dump" dumpfile
3.
  
Actual results:

fails in both circumstances.


Expected results:


Additional info:
Comment 2 Andrew Jones 2010-06-28 16:33:45 UTC 
I'm not sure this will be a bisectable issue, although it's worth a shot to see
what results we get with earlier kernels. The thing is I only know of crash
being used once with Xen guests during RHEL6's development and it was on an HVM
guest about a month and half ago (-25 kernel?).
Comment 3 RHEL Program Management 2010-06-28 16:42:56 UTC 
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 4 Dave Anderson 2010-06-28 18:51:49 UTC 
Any prior RHEL6 xen guest work was done as an HVM guest.

After taking a relatively quick look at this by trying to find
out why we can't even run on a live system, it fails when the first 
non-unity-mapped kernel virtual-to-physical address translation
was attempted.  As it turns out, the whole MMU infrastructure
is using the pv_mmu_ops[] table that is filled with the functions
from "arch/x86/xen/mmu.c".

So apparently RHEL6 PV guests have never been tested to date, because
the crash utility has *never* supported Xen guest kernels that use the
upstream pv_ops[] infrastructure.  (It only works in paravirutal mode
with the RHEL5 "built-in" xen kernel)  

Anyway, if we can't run live, it's not even worth looking at the dumpfile
issue, which looks to be a different matter entirely.

So AFAICT, I'm back at square 1, trying to figure out how the
upstream/RHEL6 Xen PV MMU infrastructure actually works.  If we use
the time it took to get RHEL5 xen kernel support working as a guide,
doing just that took me several weeks (months?) to get working.

Perhaps I'll get lucky, but this could very well be a major, 
time-consuming project.

So for now, there's no chance of kdump/crash working with RHEL6 PV guests.
Comment 5 Dave Anderson 2010-06-29 15:01:09 UTC 
Note that this affects both x86_64 and x86 RHEL6 PV guests, so
there's 4 crash support issues to address:

  x86_64 live 
  x86_64 virsh dump
  x86 live
  x86 virsh dump

We (Don and I) had no luck installing a RHEL6 32-bit x86 guest.  
When anaconda was about to start up, the screen went blank forever;
we tried it multiple times.

In any case, I'll start work with "x86_64 live" system support
first.
Comment 6 Dave Anderson 2010-06-29 15:48:39 UTC 
> Dave,
>
> see if virt-manager gives you a place to add cmdline params to install.
>
> need to add:  spinlock-type=ticket
>
> when virt-manager gets to repo to pt to, see if there's
> a line to add cmdline params.

virt-manager does have that option -- so I added it, and saw it on the
final confirmation page, but it still hangs...
Comment 7 Andrew Jones 2010-06-29 16:13:07 UTC 
(In reply to comment #6)
> virt-manager does have that option -- so I added it, and saw it on the
> final confirmation page, but it still hangs...    

I was able to boot up to the "choose language" in anaconda with

virt-install \

     --paravirt \

     --name=rhel6-32pv-1 \

     --extra-args='console=hvc0 spinlock-type=ticket' \

     --location=http://download.devel.redhat.com/nightly/latest-RHEL6.0/6/Server/i386/os \

     --vcpus=2 \

     --ram=1024 \

     --file=/var/lib/xen/images/rhel6-32pv-1.img \

     --file-size=7 \

     --nographics



I'm afraid I don't know much about using virt-manager to see if there's anything missing from there.
Comment 8 Dave Anderson 2010-06-29 17:58:13 UTC 
Created attachment 427753 [details]
anaconda-tb-jldPrQ8bYeA8.xml

OK, tried virt-install, but anaconda crapped out at what presumably
was the same time.  This is the top of the attached exception
report:

The following was filed automatically by anaconda:
anaconda 13.21.54 exception report
Traceback (most recent call first):
  File "/usr/lib/anaconda/yuminstall.py", line 1946, in writeKS
    if repo.excludepkgs:
  File "/usr/lib/anaconda/instdata.py", line 276, in writeKS
    self.backend.writeKS(f)
  File "/usr/lib/anaconda/packages.py", line 66, in writeKSConfiguration
    anaconda.id.writeKS(fn)
  File "/usr/lib/anaconda/dispatch.py", line 208, in moveStep
    rc = stepFunc(self.anaconda)
  File "/usr/lib/anaconda/dispatch.py", line 126, in gotoNext
    self.moveStep()
  File "/usr/lib/anaconda/text.py", line 657, in run
    anaconda.dispatch.gotoNext()
  File "/usr/bin/anaconda", line 1106, in &lt;module&gt;
    anaconda.intf.run(anaconda)
AttributeError: 'AnacondaYumRepo' object has no attribute 'excludepkgs'
</binding><binding fileName="/tmp/anaconda-tb-jldPrQ" name="pythonUnhandledException" type="text">anaconda 13.21.54 exception report
Traceback (most recent call first):
  File "/usr/lib/anaconda/yuminstall.py", line 1946, in writeKS
    if repo.excludepkgs:
  File "/usr/lib/anaconda/instdata.py", line 276, in writeKS
    self.backend.writeKS(f)
  File "/usr/lib/anaconda/packages.py", line 66, in writeKSConfiguration
    anaconda.id.writeKS(fn)
  File "/usr/lib/anaconda/dispatch.py", line 208, in moveStep
    rc = stepFunc(self.anaconda)
  File "/usr/lib/anaconda/dispatch.py", line 126, in gotoNext
    self.moveStep()
  File "/usr/lib/anaconda/text.py", line 657, in run
    anaconda.dispatch.gotoNext()
  File "/usr/bin/anaconda", line 1106, in &lt;module&gt;
    anaconda.intf.run(anaconda)
AttributeError: 'AnacondaYumRepo' object has no attribute 'excludepkgs'

Local variables in innermost frame:
repo: anaconda-RedHatEnterpriseLinux-201006290857.i386
line: repo --name="Red Hat Enterprise Linux"  --baseurl=http://download.devel.redhat.com/nightly/latest-RHEL6.0/6/Server/i386/os/ --cost=100
self: &lt;yuminstall.YumBackend instance at 0xb5b258ac&gt;
f: &lt;open file '/mnt/sysimage/root/anaconda-ks.cfg', mode 'w' at 0xb5292ac0&gt;
Comment 9 Qian Cai 2010-07-05 09:19:47 UTC 
Dave, looks like you have encountered an anaconda bug. It was working fine for me for both x86_64 and i386 guest installation using a stable tree - http://download.devel.redhat.com/rel-eng/RHEL6.0-20100622.1/6/Server/i386/os/
Comment 10 Dave Anderson 2010-07-05 21:12:36 UTC 
(In reply to comment #9)
> Dave, looks like you have encountered an anaconda bug. It was working fine for
> me for both x86_64 and i386 guest installation using a stable tree -
> http://download.devel.redhat.com/rel-eng/RHEL6.0-20100622.1/6/Server/i386/os/    

Bug 607664  - AttributeError: 'AnacondaYumRepo' object has no attribute 'excludepkgs'
https://bugzilla.redhat.com/show_bug.cgi?id=607664
Comment 11 Dave Anderson 2010-07-08 16:08:19 UTC 
Status update:

This implementation consists of 4 components:

  (1) run crash live on an x86_64 RHEL6 pvops/xen guest.
  (2) run crash against an x86_64 RHEL6 povps/xen xendump dumpfile. 
  (3) run crash live on an x86 RHEL6 pvops/xen guest.
  (4) run crash against an x86 RHEL6 povps/xen xendump dumpfile.

I have just completed part (1):

  # ./crash 

  crash 5.0.5p7
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
 
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...

        KERNEL: /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/vmlinux
      DUMPFILE: /dev/crash
          CPUS: 2
          DATE: Thu Jul  8 11:59:46 2010
        UPTIME: 8 days, 20:28:36
  LOAD AVERAGE: 0.00, 0.01, 0.03
         TASKS: 79
      NODENAME: localhost.localdomain
       RELEASE: 2.6.32-37.el6.x86_64
       VERSION: #1 SMP Sun Jun 20 19:29:35 EDT 2010
       MACHINE: x86_64  (2666 Mhz)
        MEMORY: 511.6 MB
           PID: 24565
       COMMAND: "crash"
          TASK: ffff880003b79520  [THREAD_INFO: ffff88001cd9c000]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)

  crash>

Starting work on part (2)...
Comment 12 Dave Anderson 2010-07-13 14:01:57 UTC 
> This implementation consists of 4 components:
>
>   (1) run crash live on an x86_64 RHEL6 pvops/xen guest.
>   (2) run crash against an x86_64 RHEL6 povps/xen xendump dumpfile. 
>   (3) run crash live on an x86 RHEL6 pvops/xen guest.
>   (4) run crash against an x86 RHEL6 povps/xen xendump dumpfile.

Completed part (2): x86_64 virsh dump:
  
  # ./crash /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/vmlinux /root/2010-0628-1314.28-rhel6-64pv.8.core
  
  crash 5.0.5p7
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
        KERNEL: /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/vmlinux
      DUMPFILE: /root/2010-0628-1314.28-rhel6-64pv.8.core
          CPUS: 2
          DATE: Mon Jun 28 07:14:28 2010
        UPTIME: 00:03:25
  LOAD AVERAGE: 0.04, 0.05, 0.01
         TASKS: 149
      NODENAME: dhcp-26-248.brq.redhat.com
       RELEASE: 2.6.32-37.el6.x86_64
       VERSION: #1 SMP Sun Jun 20 19:29:35 EDT 2010
       MACHINE: x86_64  (2833 Mhz)
        MEMORY: 511.6 MB
         PANIC: ""
           PID: 0
       COMMAND: "swapper"
          TASK: ffffffff817480c0  (1 of 2)  [THREAD_INFO: ffffffff81724000]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)
       WARNING: panic task not found
  
  crash>
Comment 13 Dave Anderson 2010-07-13 18:19:32 UTC 
> This implementation consists of 4 components:
>
>   (1) run crash live on an x86_64 RHEL6 pvops/xen guest.
>   (2) run crash against an x86_64 RHEL6 povps/xen xendump dumpfile. 
>   (3) run crash live on an x86 RHEL6 pvops/xen guest.
>   (4) run crash against an x86 RHEL6 povps/xen xendump dumpfile.

Completed part (3): x86 live:
  
  # ./crash
  
  crash 5.0.5p8
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "i686-pc-linux-gnu"...
  
        KERNEL: /usr/lib/debug/lib/modules/2.6.32-44.el6.i686/vmlinux
      DUMPFILE: /dev/crash
          CPUS: 1
          DATE: Mon Jul 12 16:37:25 2010
        UPTIME: 3 days, 01:20:45
  LOAD AVERAGE: 0.00, 0.00, 0.00
         TASKS: 147
      NODENAME: localhost.localdomain
       RELEASE: 2.6.32-44.el6.i686
       VERSION: #1 SMP Wed Jul 7 15:39:10 EDT 2010
       MACHINE: i686  (2666 Mhz)
        MEMORY: 511.6 MB
           PID: 9807
       COMMAND: "crash"
          TASK: de8f2030  [THREAD_INFO: c271e000]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)
  
  crash>
Comment 14 Dave Anderson 2010-07-14 14:34:52 UTC 
> This implementation consists of 4 components:
>
>   (1) run crash live on an x86_64 RHEL6 pvops/xen guest.
>   (2) run crash against an x86_64 RHEL6 povps/xen xendump dumpfile. 
>   (3) run crash live on an x86 RHEL6 pvops/xen guest.
>   (4) run crash against an x86 RHEL6 povps/xen xendump dumpfile.

Completed part (4): x86 virsh dump:
  
  # ./crash /usr/lib/debug/lib/modules/2.6.32-44.el6.i686/vmlinux /root/2010-0713-0902.36-rhel6-32pv.4.core
  
  crash 5.0.5p10
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "i686-pc-linux-gnu"...
  
        KERNEL: /usr/lib/debug/lib/modules/2.6.32-44.el6.i686/vmlinux
      DUMPFILE: /root/2010-0713-0902.36-rhel6-32pv.4.core
          CPUS: 1
          DATE: Tue Jul 13 09:02:30 2010
        UPTIME: 3 days, 17:45:50
  LOAD AVERAGE: 17.12, 7.39, 2.81
         TASKS: 147
      NODENAME: localhost.localdomain
       RELEASE: 2.6.32-44.el6.i686
       VERSION: #1 SMP Wed Jul 7 15:39:10 EDT 2010
       MACHINE: i686  (2666 Mhz)
        MEMORY: 511.6 MB
         PANIC: "Oops: 0002 [#1] SMP " (check log for details)
           PID: 5246
       COMMAND: "bash"
          TASK: def1d030  [THREAD_INFO: dcfe4000]
           CPU: 0
         STATE: TASK_RUNNING (PANIC)
  
  crash>
Comment 16 Dave Anderson 2010-07-14 18:24:47 UTC 
QA assist:

 (1) run crash live on an x86 RHEL6 pvops/xen guest.
 (2) run crash against an x86 RHEL6 povps/xen xendump dumpfile. 
 (3) run crash live on an x86_64 RHEL6 pvops/xen guest.
 (4) run crash against an x86_64 RHEL6 povps/xen xendump dumpfile.
  
(1) x86 live on a RHEL6 pvops/xen guest:
  
  # crash
  
  crash 5.0.0-21.el6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "i686-pc-linux-gnu"...
  
        KERNEL: /usr/lib/debug/lib/modules/2.6.32-44.el6.i686/vmlinux
      DUMPFILE: /dev/crash
          CPUS: 1
          DATE: Wed Jul 14 13:50:13 2010
        UPTIME: 1 days, 04:45:54
  LOAD AVERAGE: 0.03, 0.02, 0.00
         TASKS: 148
      NODENAME: localhost.localdomain
       RELEASE: 2.6.32-44.el6.i686
       VERSION: #1 SMP Wed Jul 7 15:39:10 EDT 2010
       MACHINE: i686  (2666 Mhz)
        MEMORY: 511.6 MB
           PID: 1078
       COMMAND: "crash"
          TASK: c256a030  [THREAD_INFO: c2694000]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)
  
  crash> p pv_init_ops
  pv_init_ops = $3 = {
    patch = 0xc04040a0 <xen_patch>
  }
  crash>
  
(2) x86 virsh dump of a RHEL6 pvops/xen guest:
  
  # crash /usr/lib/debug/lib/modules/2.6.32-44.el6.i686/vmlinux 2010-0713-0902.36-rhel6-32pv.4.core
  
  crash 5.0.0-21.el6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "i686-pc-linux-gnu"...
  
        KERNEL: /usr/lib/debug/lib/modules/2.6.32-44.el6.i686/vmlinux
      DUMPFILE: 2010-0713-0902.36-rhel6-32pv.4.core
          CPUS: 1
          DATE: Tue Jul 13 09:02:30 2010
        UPTIME: 3 days, 17:45:50
  LOAD AVERAGE: 17.12, 7.39, 2.81
         TASKS: 147
      NODENAME: localhost.localdomain
       RELEASE: 2.6.32-44.el6.i686
       VERSION: #1 SMP Wed Jul 7 15:39:10 EDT 2010
       MACHINE: i686  (2666 Mhz)
        MEMORY: 511.6 MB
         PANIC: "Oops: 0002 [#1] SMP " (check log for details)
           PID: 5246
       COMMAND: "bash"
          TASK: def1d030  [THREAD_INFO: dcfe4000]
           CPU: 0
         STATE: TASK_RUNNING (PANIC)
  
  crash> p pv_init_ops
  pv_init_ops = $3 = {
    patch = 0xc04040a0 <xen_patch>
  }
  crash> 

(3) x86_64 live on a RHEL6 pvops/xen guest:

  # crash
  
  crash 5.0.0-21.el6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
        KERNEL: /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/vmlinux
      DUMPFILE: /dev/crash
          CPUS: 2
          DATE: Wed Jul 14 14:00:34 2010
        UPTIME: 14 days, 22:29:24
  LOAD AVERAGE: 0.00, 0.01, 0.00
         TASKS: 77
      NODENAME: localhost.localdomain
       RELEASE: 2.6.32-37.el6.x86_64
       VERSION: #1 SMP Sun Jun 20 19:29:35 EDT 2010
       MACHINE: x86_64  (2666 Mhz)
        MEMORY: 511.6 MB
           PID: 875
       COMMAND: "crash"
          TASK: ffff88000399f520  [THREAD_INFO: ffff88001ccc4000]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)
  
  crash> p pv_init_ops
  pv_init_ops = $3 = {
    patch = 0xffffffff8100bd10 <xen_patch>
  }
  crash> 

(4) x86_64 virsh dump of a RHEL6 pvops/xen guest:

  # crash /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/vmlinux 2010-0628-1314.28-rhel6-64pv.8.core
  
  crash 5.0.0-21.el6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
        KERNEL: /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/vmlinux
      DUMPFILE: 2010-0628-1314.28-rhel6-64pv.8.core
          CPUS: 2
          DATE: Mon Jun 28 07:14:28 2010
        UPTIME: 00:03:25
  LOAD AVERAGE: 0.04, 0.05, 0.01
         TASKS: 149
      NODENAME: dhcp-26-248.brq.redhat.com
       RELEASE: 2.6.32-37.el6.x86_64
       VERSION: #1 SMP Sun Jun 20 19:29:35 EDT 2010
       MACHINE: x86_64  (2833 Mhz)
        MEMORY: 511.6 MB
         PANIC: ""
           PID: 0
       COMMAND: "swapper"
          TASK: ffffffff817480c0  (1 of 2)  [THREAD_INFO: ffffffff81724000]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)
       WARNING: panic task not found
  
  crash> p pv_init_ops
  pv_init_ops = $3 = {
    patch = 0xffffffff8100bd10 <xen_patch>
  }
  crash>
Comment 18 releng-rhel@redhat.com 2010-11-10 20:04:25 UTC 
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.