Bug 1402108

+++ This bug was initially created as a clone of Bug #1386909 +++

Description of problem:

latest crash crashes when running live analysis

Version-Release number of selected component (if applicable):

Latest crash (recompiled from https://people.redhat.com/anderson/ - nothing interesting in the git since the 7.1.6 release last week)


How reproducible:

It seems to depend on the kernel relocation offset maybe?
After rebooting it wouldn't crash (had offset 304MB), re-rebooted (496MB) crashed; next reboot (80MB) worked.. 944MB failed... etc.
tested on a VM with 2GB of ram.

Steps to Reproduce:
1. run crash

Actual results:

# crash

crash 7.1.6
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [736MB]: patching 82381 gdb minimal_symbol values

please wait... (patching 82381 gdb minimal_symbol values) Segmentation fault (core dumped)


---- 
gdb gives a huge backtrace that's obviously wrong;
#0  convert_time (count=18446744072363204608, buf=0x0) at tools.c:5606
#1  0x0000000000000001 in ?? ()
#2  0x0000000002bc8fa0 in ?? ()
#3  0x0000000000001000 in ?? ()
#4  0x0000000000000001 in ?? ()
#5  0x0000000000478c04 in please_wait (s=0xd00e20 <vm_table+320> "")
    at tools.c:5741
#6  0x0000000000001000 in ?? ()
#7  0x0000000000000001 in ?? ()
#8  0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741
#9  0x0000000000001000 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741
...
#16347 0x0000000000001000 in ?? ()
#16348 0x0000000000000001 in ?? ()
#16349 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741
#16350 0x0000000000001000 in ?? ()
#16351 0x0000000000000001 in ?? ()
#16352 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741
#16353 0x0000000000001000 in ?? ()
#16354 0x0000000000000001 in ?? ()
#16355 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741
#16356 0x0000000000001000 in ?? ()
#16357 0x0000000000000001 in ?? ()
#16358 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741
#16359 0x0000000000001000 in ?? ()
#16360 0x0000000000000001 in ?? ()
#16361 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741
#16362 0x0000000000001000 in ?? ()
#16363 0x0000000000000001 in ?? ()
#16364 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741
#16365 0x0000000000463318 in xen_hyper_dump_xen_hyper_offset_table (
    makestruct=<optimized out>, spec=<optimized out>)
    at xen_hyper_dump_tables.c:825
#16366 0x00007ffd0da3ee25 in ?? ()
#16367 0x00007ffd0da3ee2f in ?? ()
#16368 0x00007ffd0da3ee43 in ?? ()
#16369 0x00007ffd0da3ee4e in ?? ()
#16370 0x00007ffd0da3ee5e in ?? ()
#16371 0x00007ffd0da3eec3 in ?? ()
#16372 0x00007ffd0da3eed7 in ?? ()
#16373 0x00007ffd0da3eeee in ?? ()
#16374 0x00007ffd0da3eef6 in ?? ()
#16375 0x00007ffd0da3ef01 in ?? ()
#16376 0x00007ffd0da3ef0f in ?? ()
#16377 0x00007ffd0da3ef1c in ?? ()
#16378 0x00007ffd0da3ef4f in ?? ()
#16379 0x00007ffd0da3ef7f in ?? ()
#16380 0x00007ffd0da3efa2 in ?? ()
#16381 0x00007ffd0da3efbc in ?? ()
#16382 0x00007ffd0da3efd8 in ?? ()
#16383 0x0000000000000000 in ?? ()


Expected results:

get a crash prompt

Additional info:

Haven't checked actual kdump analysis as that looks broken as well, will look into that.

--- Additional comment from Dave Anderson on 2016-10-19 15:43:29 EDT ---

FWIW, I'm working on a 4.9.0-0.rc1.git0.2.fc26 live kernel which
was relocated 544MB.

--- Additional comment from Dominique Martinet on 2016-10-19 15:55:24 EDT ---

Okay so might be something else... Basically have a pristine fedora 24 except for the crash utility and extlinux boot.

Here's my /proc/cmdline:
BOOT_IMAGE=/boot/vmlinuz-4.7.7-200.fc24.x86_64 initrd=/boot/initramfs-4.7.7-200.fc24.x86_64.img root=/dev/vda selinux=0 nomodeset ro crashkernel=128M console=tty0 console=ttyS0



I upgraded kexec-tools to rawhide version and got kdump to work, I have the same problem with a 37MB vmcore file, that might help if you can't reproduce.
You can find it here:

http://asmadeus.notk.org/crash/vmcore

# rpm -qa|grep `uname -r`
kernel-core-4.7.7-200.fc24.x86_64
kernel-debuginfo-4.7.7-200.fc24.x86_64
kernel-4.7.7-200.fc24.x86_64
kernel-modules-4.7.7-200.fc24.x86_64
kernel-debuginfo-common-x86_64-4.7.7-200.fc24.x86_64
kernel-headers-4.7.7-200.fc24.x86_64

--- Additional comment from Dave Anderson on 2016-10-25 13:46:18 EDT ---

I've pushed a patch upstream that fixes this problem:

 https://github.com/crash-utility/crash/commit/774a92f48fabb0f98eb14875e54c6d5f573f074c
 
  Fix for a possible segmentation violation when analyzing Linux 4.7
  x86_64 kernels that are configured with CONFIG_RANDOMIZE_BASE.
  Depending upon the randomized starting address of the kernel text
  and static data, a segmentation violation may occur during session
  initialization, just after the patching of the gdb minimal_symbol
  values message.
  (anderson)


> I upgraded kexec-tools to rawhide version and got kdump to work, I have
> the same problem with a 37MB vmcore file, that might help if you can't
> reproduce.
> You can find it here:
>
> http://asmadeus.notk.org/crash/vmcore

Thanks for that -- here it is:

  # crash vmlinux-4.7.7-200.fc24 vmcore-4.7.7-200.fc24
  
  crash 7.1.6++
  Copyright (C) 2002-2016  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005, 2011  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.6
  Copyright (C) 2013 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
  WARNING: kernel relocated [784MB]: patching 82381 gdb minimal_symbol values
  
        KERNEL: vmlinux-4.7.7-200.fc24                                
      DUMPFILE: vmcore-4.7.7-200.fc24  [PARTIAL DUMP]
          CPUS: 4
          DATE: Wed Oct 19 21:49:33 2016
        UPTIME: 00:03:33
  LOAD AVERAGE: 0.36, 0.13, 0.04
         TASKS: 138
      NODENAME: f4
       RELEASE: 4.7.7-200.fc24.x86_64
       VERSION: #1 SMP Sat Oct 8 00:21:59 UTC 2016
       MACHINE: x86_64  (2591 Mhz)
        MEMORY: 2 GB
         PANIC: "sysrq: SysRq : Trigger a crash"
           PID: 1858
       COMMAND: "bash"
          TASK: ffff88007aa58000  [THREAD_INFO: ffff88007a5dc000]
           CPU: 2
         STATE: TASK_RUNNING (SYSRQ)
  
  crash> 

It's queued for crash-7.1.7, but I don't have a target release date at
this time.

In the meantime, you can clone git://github.com/crash-utility/crash.git,
and build it with "make lzo".

--- Additional comment from Dominique Martinet on 2016-10-26 07:47:36 EDT ---

I can confirm it works, thanks!

--- Additional comment from Dave Anderson on 2016-12-06 11:33:20 EST ---

Information for build crash-7.1.7-1.fc26
http://koji.fedoraproject.org/koji/buildinfo?buildID=823232

--- Additional comment from Dave Anderson on 2016-12-06 14:53:21 EST ---

Information for build crash-7.1.7-1.fc24
http://koji.fedoraproject.org/koji/buildinfo?buildID=823280