I have a rawhide (f20) KVM guest running on a f18 host. When I take a "virsh dump" of the guest's image, I'm unable to open that with crash. Running crash on the same live kernel however seems to work fine. When I run crash on the dumpfile, I get this output: # crash /usr/lib/debug/lib/modules/3.9.0-0.rc7.git3.1.fc20.x86_64/vmlinux /mnt/salusa/rawhide.vmcore2 crash 6.1.6-1.fc20 Copyright (C) 2002-2013 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.3.1 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... crash: read error: kernel virtual address: ffffffff81807090 type: "cpu_possible_mask" ...relevant versions: host kernel: 3.8.8-202.fc18.x86_64 guest kernel: 3.9.0-0.rc7.git3.1.fc20.x86_64 crash: crash-6.1.6-1.fc20.x86_64 Let me know if you need any other info.
I can't surmise anything without the vmlinux/vmcore pair. If you can make them available, that would be a start. That being said, I haven't seen a "virsh dump" KVM dumpfile in quite some time given that kdump is the preferred method. The "virsh dump" method is a kludge from the start, since it basically takes a QEMU saved VM file and treats it like a crash dump. And my guess is that there are changes in the QEMU header that have obsoleted the crash utility "support" that Paolo Bonzini originally wrote and supplied to me.
I forgot another important point about this issue, which explains why continued support for the old saved-VM file format has been ignored. If you follow this thread: [Crash-utility] crash cannot work for the newest qemu's core https://www.redhat.com/archives/crash-utility/2012-June/msg00045.html which BTW, starts with an identical report to yours: > Hi, all > > When I use crash to deal with the vmcore generated by 'vrish dump', > I meet the following error: > > crash: seek error: kernel virtual address: ffffffff8180f1e0 type: "cpu_possible_mask". > > When I specify '-d 8' to get more information, I found the following message: > > NOTE: KVM/QEMU CPU_SAVE_VERSION 12 is greater than supported version 9 > > So I guess crash does not support it now. Does anyone have plan to fix it? > > Thanks > Wen Congyang Anyway, the discussion lead to the then-ongoing implementation of the "virsh dump --memory-only" QEMU monitor command, which creates an ELF vmcore dumpfile, and essentially obsoletes the use of "virsh dump" for taking KVM crash dumps. Support for it went into crash-6.1.0: http://people.redhat.com/anderson/crash.changelog.html#6_1_0 6.1.0 - Fix for 32-bit SADUMP dumpfiles to correctly check whether a requested physical address is within the 0-640K backup region. Without the patch, requested physical addresses that are larger than 32-bits are truncated to 32-bit values, leading to unexpected results. (d.hatayama.com) - Added support for the ELF dumpfile type that is generated by the new "virsh dump --memory-only" option. The "--memory-only" option uses a new "dump-guest-memory" QEMU monitor command that creates an ELF kdump vmcore clone. The "virsh dump" command continues to borrow the "migrate" QEMU monitor command to create a file that is designed for guest migration, and not well-suited for a vmcore because it is not designed for random-access of physical memory. A new "help -r" option has been added to dump the registers that are stored in per-cpu "QEMU" ELF notes; those notes are used to distinguish this dumpfile type from regular kdump ELF vmcores. The patch also combines common functionality between the new format and the SADUMP format. (qiaonuohan.com, anderson) ... Can you try using that "virsh dump --memory-only" option instead? Dave
"virsh dump --memory-only" seems to work just fine, thanks! I'll go ahead and close this as NOTABUG.
Nice! Thanks!
The change is in the kernel, actually. The kernel is now randomizing the IDT, so crash's method to guess the kernel base is not working anymore. Libvirt should probably not have introduced the "--memory-only" at all, and just switched to the memory-only dump unconditionally.
(In reply to comment #5) > The change is in the kernel, actually. The kernel is now randomizing the > IDT, so crash's method to guess the kernel base is not working anymore. > > Libvirt should probably not have introduced the "--memory-only" at all, and > just switched to the memory-only dump unconditionally. Totally agree -- I recall there being some discussion on making a wholesale replacement when Wen Conyang was developing/pushing the new scheme, but I can't recall why the argument was lost. But anyway, thanks for the randomized/IDT info, I wasn't aware of that. I was confused because I've got a few sample kvmdumps that work just fine with QEMU CPU_SAVE_VERSION 12.