Bug 457373 - SIGSEGV when Analyse Xen Hypervisor using Search Command
SIGSEGV when Analyse Xen Hypervisor using Search Command
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: crash (Show other bugs)
5.2
All Linux
medium Severity medium
: rc
: ---
Assigned To: Dave Anderson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-07-31 05:01 EDT by CAI Qian
Modified: 2009-01-20 17:13 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-20 17:13:41 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description CAI Qian 2008-07-31 05:01:20 EDT
Description of problem:
I have seen it on both i686 and x86_64.

# crash  /boot/xen-syms-2.6.18-92.el5 /var/crash/2008-07-31-01\:57/vmcore

   KERNEL: /boot/xen-syms-2.6.18-92.el5
DEBUGINFO: /usr/lib/debug/boot/xen-syms-2.6.18-92.el5.debug
 DUMPFILE: /var/crash/2008-07-31-01:57/vmcore
     CPUS: 2
  DOMAINS: 4
   UPTIME: 01:19:21
  MACHINE: Intel(R) Pentium(R) D CPU 3.20GHz  (3200 Mhz)
   MEMORY: 4 GB
  PCPU-ID: 1
     PCPU: ffbf3fb4
  VCPU-ID: 1
     VCPU: ffbed080  (VCPU_RUNNING)
DOMAIN-ID: 0
   DOMAIN: ff1c0080  (DOMAIN_RUNNING)
    STATE: CRASH

crash> search -u deadbeef
Segmentation fault (core dumped)

# gdb /usr/bin/crash core.8040
(gdb) bt
#0  0x080ba3c8 in vaddr_type ()
#1  0x080ba554 in cmd_search ()
#2  0x080859bc in exec_command ()
#3  0x080858af in main_loop ()
#4  0x0818e0fb in gdb_main_entry ()
#5  0x081c76d0 in throw_exception ()
#6  0x081c7728 in catch_errors ()
#7  0x0818e8a4 in gdb_main_entry ()
#8  0x081c76d0 in throw_exception ()
#9  0x081c7728 in catch_errors ()
#10 0x0818e0ab in gdb_main ()
#11 0x0818e0eb in gdb_main_entry ()
#12 0x080fc314 in gdb_main_loop ()
#13 0x080856d2 in main ()

Also, search command did not work properly when analysing Dom0 Kernel,
# crash /usr/lib/debug/lib/modules/2.6.18-92.el5xen/vmlinux
/var/crash/2008-07-31-01\:57/vmcore

      KERNEL: /usr/lib/debug/lib/modules/2.6.18-92.el5xen/vmlinux
    DUMPFILE: /var/crash/2008-07-31-01:57/vmcore
        CPUS: 2
        DATE: Thu Jul 31 01:56:28 2008
      UPTIME: 01:19:21
LOAD AVERAGE: 0.19, 0.57, 0.51
       TASKS: 97
    NODENAME: dell-pe830-02.rhts.bos.redhat.com
     RELEASE: 2.6.18-92.el5xen
     VERSION: #1 SMP Tue Apr 29 13:45:57 EDT 2008
     MACHINE: i686  (3200 Mhz)
      MEMORY: 3.7 GB
       PANIC: "kernel BUG at
/mnt/tests/kernel/kdump/setup-dom0/crash-lkdtm/lkdtm/lkdtm.c:264!"
         PID: 0
     COMMAND: "swapper"
        TASK: c1234550  (1 of 2)  [THREAD_INFO: c122f000]
         CPU: 1
       STATE: TASK_RUNNING (PANIC)

crash> search -u deadbeef
search: invalid kernel virtual address: 28  type: "mm_struct pgd"


Version-Release number of selected component (if applicable):
RHEL 5.2 GA tree
kernel-xen-2.6.18-92.el5
crash-4.0-5.0.3

How reproducible:
always
Comment 1 CAI Qian 2008-07-31 05:06:04 EDT
I have the machine (dell-pe830-02.rhts.bos.redhat.com) and vmcore reserved for
the next two days, in case if you might want to have a look.
Comment 2 Dave Anderson 2008-08-05 10:01:14 EDT
Can you please re-create the issue and make a vmcore/vmlinux pair
available for me to look at?
Comment 3 Dave Anderson 2008-08-05 10:03:01 EDT
In the future -- can you please *always* take a snapshot of the
suspect vmlinux/vmcore pairs, and save them somewhere.
Comment 4 CAI Qian 2008-08-06 02:34:04 EDT
You could find all files (xen-syms, vmcore, and vmlinux) at,
http://porkchop.devel.redhat.com/qa/qa/qcai/bz/457373
Comment 5 CAI Qian 2008-08-06 03:05:20 EDT
If the above line does not work, you could try it from SSH,

porkchop.devel.redhat.com:/mnt/redhat/qa/qa/qcai/bz/457371
Comment 6 CAI Qian 2008-08-06 03:06:13 EDT
In fact, it is,
porkchop.devel.redhat.com:/mnt/redhat/qa/qa/qcai/bz/457373
Comment 7 Dave Anderson 2008-08-06 10:50:03 EDT
It should be noted that the "search -u" is nonsensical in both cases
reported by this bugzilla, i.e.:

 1) when run on a hypervisor kernel
 2) when run on a vmlinux kernel thread

because there is no "user-space" memory to search in those cases.

Nonetheless, the hypervisor kernel should catch the attempt and
not cause a segmentation violation.  The vmlinux case is failing
"correctly", but also should probably not even make the attempt to
walk the (non-existent) user virtual address space.
Comment 8 Dave Anderson 2008-08-07 13:33:24 EDT
Fixed with this patch:

--- memory.c    4 Aug 2008 19:10:21 -0000       1.177
+++ memory.c    7 Aug 2008 17:26:51 -0000       1.179
@@ -10940,6 +10940,11 @@
                 switch(c)
                 {
                case 'u':
+                       if (XEN_HYPER_MODE())
+                               error(FATAL, 
+                                   "-u option is not applicable to the "
+                                   "Xen hypervisor\n");
+
                        if (!sflag) {
                                address_space_start(CURRENT_CONTEXT(),&start);
                                sflag++;
@@ -11048,6 +11053,8 @@
        switch (memtype)
        {
        case UVADDR:
+               if (is_kernel_thread(CURRENT_TASK()) || !task_mm(CURRENT_TASK(), TRUE))
+                       error(FATAL, "current context has no user address space\n");
                if (end > uvaddr_end) {
                        error(INFO, 
                  "address range starts in user space and ends kernel space\n");
Comment 9 CAI Qian 2008-08-07 21:52:50 EDT
Dave, is -k applicable to the Xen hypervisor? I also found out that by using the same files, search -k did not work,

crash xen-syms-2.6.18-92.el5 vmcore 

crash 4.0-5.0.3
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

   KERNEL: xen-syms-2.6.18-92.el5
DEBUGINFO: /usr/lib/debug/boot/xen-syms-2.6.18-92.el5.debug
 DUMPFILE: vmcore
     CPUS: 2
  DOMAINS: 4
   UPTIME: 00:35:15
  MACHINE: Intel(R) Pentium(R) D CPU 3.20GHz  (3200 Mhz)
   MEMORY: 4 GB
  PCPU-ID: 0
     PCPU: ff1d3fb4
  VCPU-ID: 0
     VCPU: ffbee080  (VCPU_RUNNING)
DOMAIN-ID: 0
   DOMAIN: ff1c0080  (DOMAIN_RUNNING)
    STATE: CRASH

crash> search -k deadbeef
search: cannot resolve: "vmlist"

crash> p vmlist
No symbol "vmlist" in current context.
p: gdb request failed: p vmlist
Comment 10 Dave Anderson 2008-08-08 08:33:52 EDT
> Dave, is -k applicable to the Xen hypervisor? I also found out that by using
> the same files, search -k did not work,

Actually I see a different error using the upstream version of crash.

But as I look at it more, I think that the "search" command is not applicable
to use with the Xen hypervisor binary, since the code is far too
Linux-specific.  I think the best idea is to remove the search command from
the Xen hypervisor crash command-set until the upstream Fujitsu/VAlinux 
guys who implemented it can make it work correctly with the crash utility.

I wouldn't worry about it.  Almost nobody uses crash with the xen-syms...
Comment 16 errata-xmlrpc 2009-01-20 17:13:41 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0240.html

Note You need to log in before you can comment on or make changes to this bug.