Hide Forgot
+++ This bug was initially created as a clone of Bug #1386909 +++ Description of problem: latest crash crashes when running live analysis Version-Release number of selected component (if applicable): Latest crash (recompiled from https://people.redhat.com/anderson/ - nothing interesting in the git since the 7.1.6 release last week) How reproducible: It seems to depend on the kernel relocation offset maybe? After rebooting it wouldn't crash (had offset 304MB), re-rebooted (496MB) crashed; next reboot (80MB) worked.. 944MB failed... etc. tested on a VM with 2GB of ram. Steps to Reproduce: 1. run crash Actual results: # crash crash 7.1.6 Copyright (C) 2002-2016 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [736MB]: patching 82381 gdb minimal_symbol values please wait... (patching 82381 gdb minimal_symbol values) Segmentation fault (core dumped) ---- gdb gives a huge backtrace that's obviously wrong; #0 convert_time (count=18446744072363204608, buf=0x0) at tools.c:5606 #1 0x0000000000000001 in ?? () #2 0x0000000002bc8fa0 in ?? () #3 0x0000000000001000 in ?? () #4 0x0000000000000001 in ?? () #5 0x0000000000478c04 in please_wait (s=0xd00e20 <vm_table+320> "") at tools.c:5741 #6 0x0000000000001000 in ?? () #7 0x0000000000000001 in ?? () #8 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741 #9 0x0000000000001000 in ?? () #10 0x0000000000000001 in ?? () #11 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741 ... #16347 0x0000000000001000 in ?? () #16348 0x0000000000000001 in ?? () #16349 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741 #16350 0x0000000000001000 in ?? () #16351 0x0000000000000001 in ?? () #16352 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741 #16353 0x0000000000001000 in ?? () #16354 0x0000000000000001 in ?? () #16355 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741 #16356 0x0000000000001000 in ?? () #16357 0x0000000000000001 in ?? () #16358 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741 #16359 0x0000000000001000 in ?? () #16360 0x0000000000000001 in ?? () #16361 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741 #16362 0x0000000000001000 in ?? () #16363 0x0000000000000001 in ?? () #16364 0x0000000000478c04 in please_wait (s=0x2bc8fa0 "") at tools.c:5741 #16365 0x0000000000463318 in xen_hyper_dump_xen_hyper_offset_table ( makestruct=<optimized out>, spec=<optimized out>) at xen_hyper_dump_tables.c:825 #16366 0x00007ffd0da3ee25 in ?? () #16367 0x00007ffd0da3ee2f in ?? () #16368 0x00007ffd0da3ee43 in ?? () #16369 0x00007ffd0da3ee4e in ?? () #16370 0x00007ffd0da3ee5e in ?? () #16371 0x00007ffd0da3eec3 in ?? () #16372 0x00007ffd0da3eed7 in ?? () #16373 0x00007ffd0da3eeee in ?? () #16374 0x00007ffd0da3eef6 in ?? () #16375 0x00007ffd0da3ef01 in ?? () #16376 0x00007ffd0da3ef0f in ?? () #16377 0x00007ffd0da3ef1c in ?? () #16378 0x00007ffd0da3ef4f in ?? () #16379 0x00007ffd0da3ef7f in ?? () #16380 0x00007ffd0da3efa2 in ?? () #16381 0x00007ffd0da3efbc in ?? () #16382 0x00007ffd0da3efd8 in ?? () #16383 0x0000000000000000 in ?? () Expected results: get a crash prompt Additional info: Haven't checked actual kdump analysis as that looks broken as well, will look into that. --- Additional comment from Dave Anderson on 2016-10-19 15:43:29 EDT --- FWIW, I'm working on a 4.9.0-0.rc1.git0.2.fc26 live kernel which was relocated 544MB. --- Additional comment from Dominique Martinet on 2016-10-19 15:55:24 EDT --- Okay so might be something else... Basically have a pristine fedora 24 except for the crash utility and extlinux boot. Here's my /proc/cmdline: BOOT_IMAGE=/boot/vmlinuz-4.7.7-200.fc24.x86_64 initrd=/boot/initramfs-4.7.7-200.fc24.x86_64.img root=/dev/vda selinux=0 nomodeset ro crashkernel=128M console=tty0 console=ttyS0 I upgraded kexec-tools to rawhide version and got kdump to work, I have the same problem with a 37MB vmcore file, that might help if you can't reproduce. You can find it here: http://asmadeus.notk.org/crash/vmcore # rpm -qa|grep `uname -r` kernel-core-4.7.7-200.fc24.x86_64 kernel-debuginfo-4.7.7-200.fc24.x86_64 kernel-4.7.7-200.fc24.x86_64 kernel-modules-4.7.7-200.fc24.x86_64 kernel-debuginfo-common-x86_64-4.7.7-200.fc24.x86_64 kernel-headers-4.7.7-200.fc24.x86_64 --- Additional comment from Dave Anderson on 2016-10-25 13:46:18 EDT --- I've pushed a patch upstream that fixes this problem: https://github.com/crash-utility/crash/commit/774a92f48fabb0f98eb14875e54c6d5f573f074c Fix for a possible segmentation violation when analyzing Linux 4.7 x86_64 kernels that are configured with CONFIG_RANDOMIZE_BASE. Depending upon the randomized starting address of the kernel text and static data, a segmentation violation may occur during session initialization, just after the patching of the gdb minimal_symbol values message. (anderson) > I upgraded kexec-tools to rawhide version and got kdump to work, I have > the same problem with a 37MB vmcore file, that might help if you can't > reproduce. > You can find it here: > > http://asmadeus.notk.org/crash/vmcore Thanks for that -- here it is: # crash vmlinux-4.7.7-200.fc24 vmcore-4.7.7-200.fc24 crash 7.1.6++ Copyright (C) 2002-2016 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [784MB]: patching 82381 gdb minimal_symbol values KERNEL: vmlinux-4.7.7-200.fc24 DUMPFILE: vmcore-4.7.7-200.fc24 [PARTIAL DUMP] CPUS: 4 DATE: Wed Oct 19 21:49:33 2016 UPTIME: 00:03:33 LOAD AVERAGE: 0.36, 0.13, 0.04 TASKS: 138 NODENAME: f4 RELEASE: 4.7.7-200.fc24.x86_64 VERSION: #1 SMP Sat Oct 8 00:21:59 UTC 2016 MACHINE: x86_64 (2591 Mhz) MEMORY: 2 GB PANIC: "sysrq: SysRq : Trigger a crash" PID: 1858 COMMAND: "bash" TASK: ffff88007aa58000 [THREAD_INFO: ffff88007a5dc000] CPU: 2 STATE: TASK_RUNNING (SYSRQ) crash> It's queued for crash-7.1.7, but I don't have a target release date at this time. In the meantime, you can clone git://github.com/crash-utility/crash.git, and build it with "make lzo". --- Additional comment from Dominique Martinet on 2016-10-26 07:47:36 EDT --- I can confirm it works, thanks! --- Additional comment from Dave Anderson on 2016-12-06 11:33:20 EST --- Information for build crash-7.1.7-1.fc26 http://koji.fedoraproject.org/koji/buildinfo?buildID=823232 --- Additional comment from Dave Anderson on 2016-12-06 14:53:21 EST --- Information for build crash-7.1.7-1.fc24 http://koji.fedoraproject.org/koji/buildinfo?buildID=823280
Information for build crash-7.1.7-1.fc25 http://koji.fedoraproject.org/koji/buildinfo?buildID=823252
crash-7.1.7-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b7c1bcbc18
crash-7.1.7-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b7c1bcbc18
crash-7.1.7-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.