Bug 1876589

Summary: Crash utility cannot analyse a core dump image from virsh
Product: [Fedora] Fedora Reporter: Kappa <commodorekappa+redhat>
Component: crashAssignee: lijiang
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: jieli, lijiang, ruqin, xiawu
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: crash-7.3.0-1.fc35 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-17 01:48:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kappa 2020-09-07 16:20:13 UTC
Description of problem:
Having some KVM guest OS utilizing 100% of processing power and is not responsive.

Used virsh dump to dump an image but it cannot be analyzed by crash utility.
The Host and Guest also run Fedora 32.

crash 7.2.8-2.fc32
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

vmcore_data:
                  flags: c0 (KDUMP_LOCAL|KDUMP_ELF64)
                   ndfd: 3
                    ofp: 7fab709d9500
            header_size: 2144
   num_pt_load_segments: 7
     pt_load_segment[0]:
            file_offset: 860
             phys_start: 0
               phys_end: a0000
              zero_fill: 0
     pt_load_segment[1]:
            file_offset: a0860
             phys_start: c0000
               phys_end: 80000000
              zero_fill: 0
     pt_load_segment[2]:
            file_offset: 7ffe0860
             phys_start: f0000000
               phys_end: f4000000
              zero_fill: 0
     pt_load_segment[3]:
            file_offset: 83fe0860
             phys_start: f4000000
               phys_end: f8000000
              zero_fill: 0
     pt_load_segment[4]:
            file_offset: 87fe0860
             phys_start: f9a10000
               phys_end: f9a12000
              zero_fill: 0
     pt_load_segment[5]:
            file_offset: 87fe2860
             phys_start: fffc0000
               phys_end: 100000000
              zero_fill: 0
     pt_load_segment[6]:
            file_offset: 88022860
             phys_start: 100000000
               phys_end: 148000000
              zero_fill: 0
             elf_header: 55d7fec6b400
                  elf32: 0
                notes32: 0
                 load32: 0
                  elf64: 55d7fec6b400
                notes64: 55d7fec6b440
                 load64: 55d7fec6b478
               sect0_64: 0
            nt_prstatus: 55d7fec6b600
            nt_prpsinfo: 0
          nt_taskstruct: 0
            task_struct: 0
             arch_data1: (unused)
             arch_data2: (unused)
           switch_stack: 0
              page_size: 0
         xen_kdump_data: (unused)
     num_prstatus_notes: 2
         num_qemu_notes: 2
             vmcoreinfo: 0
        size_vmcoreinfo: 0
     nt_prstatus_percpu:
        000055d7fec6b600 000055d7fec6b764
         nt_qemu_percpu:
        000055d7fec6b8dc 000055d7fec6baa8
       backup_src_start: 0
        backup_src_size: 0
          backup_offset: 0

Elf64_Ehdr:
                e_ident: \177ELF
      e_ident[EI_CLASS]: 2 (ELFCLASS64)
       e_ident[EI_DATA]: 1 (ELFDATA2LSB)
    e_ident[EI_VERSION]: 1 (EV_CURRENT)
      e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
 e_ident[EI_ABIVERSION]: 0
                 e_type: 4 (ET_CORE)
              e_machine: 62 (EM_X86_64)
              e_version: 1 (EV_CURRENT)
                e_entry: 0
                e_phoff: 40
                e_shoff: 0
                e_flags: 0
               e_ehsize: 40
            e_phentsize: 38
                e_phnum: 8
            e_shentsize: 0
                e_shnum: 0
             e_shstrndx: 0
Elf64_Phdr:
                 p_type: 4 (PT_NOTE)
               p_offset: 512 (200)
                p_vaddr: 0
                p_paddr: 0
               p_filesz: 1632 (660)
                p_memsz: 1632 (660)
                p_flags: 0 ()
                p_align: 0
Elf64_Phdr:
                 p_type: 1 (PT_LOAD)
               p_offset: 2144 (860)
                p_vaddr: 0
                p_paddr: 0
               p_filesz: 655360 (a0000)
                p_memsz: 655360 (a0000)
                p_flags: 0 ()
                p_align: 0
Elf64_Phdr:
                 p_type: 1 (PT_LOAD)
               p_offset: 657504 (a0860)
                p_vaddr: c0000
                p_paddr: c0000
               p_filesz: 2146697216 (7ff40000)
                p_memsz: 2146697216 (7ff40000)
                p_flags: 0 ()
                p_align: 0
Elf64_Phdr:
                 p_type: 1 (PT_LOAD)
               p_offset: 2147354720 (7ffe0860)
                p_vaddr: f0000000
                p_paddr: f0000000
               p_filesz: 67108864 (4000000)
                p_memsz: 67108864 (4000000)
                p_flags: 0 ()
                p_align: 0
Elf64_Phdr:
                 p_type: 1 (PT_LOAD)
               p_offset: 2214463584 (83fe0860)
                p_vaddr: f4000000
                p_paddr: f4000000
               p_filesz: 67108864 (4000000)
                p_memsz: 67108864 (4000000)
                p_flags: 0 ()
                p_align: 0
Elf64_Phdr:
                 p_type: 1 (PT_LOAD)
               p_offset: 2281572448 (87fe0860)
                p_vaddr: f9a10000
                p_paddr: f9a10000
               p_filesz: 8192 (2000)
                p_memsz: 8192 (2000)
                p_flags: 0 ()
                p_align: 0
Elf64_Phdr:
                 p_type: 1 (PT_LOAD)
               p_offset: 2281580640 (87fe2860)
                p_vaddr: fffc0000
                p_paddr: fffc0000
               p_filesz: 262144 (40000)
                p_memsz: 262144 (40000)
                p_flags: 0 ()
                p_align: 0
Elf64_Phdr:
                 p_type: 1 (PT_LOAD)
               p_offset: 2281842784 (88022860)
                p_vaddr: 100000000
                p_paddr: 100000000
               p_filesz: 1207959552 (48000000)
                p_memsz: 1207959552 (48000000)
                p_flags: 0 ()
                p_align: 0
Elf64_Nhdr:
               n_namesz: 5 ("CORE")
               n_descsz: 336
                 n_type: 1 (NT_PRSTATUS)
                         si.signo: 0  si.code: 0  si.errno: 0
                         cursig: 0  sigpend: 0  sighold: 0
                         pid: 1  ppid: 0  pgrp: 0  sid:0
                         utime: 0.000000  stime: 0.000000
                         cutime: 0.000000  cstime: 0.000000
                         ORIG_RAX: 0  fpvalid: 0
                              R15: ffffa522c0003e20  R14: 0000000000000000
                              R13: 0000000000000001  R12: 0000000000000033
                              RBP: 00000000000003ac  RBX: 0000000000000046
                              R11: 000000000002bf40  R10: 00000000000003ab
                               R9: ffffffff953f7c44   R8: 0000001d7358df44
                              RAX: 0000000000000001  RCX: ffff9116c1f88000
                              RDX: 0000000000000001  RSI: 0000000000000002
                              RDI: ffffffff953f7c30  RIP: ffffffff931497c4
                           RFLAGS: 0000000000000002  RSP: ffffa522c0003dd0
                          FS_BASE: 0000000000000000
                          GS_BASE: ffff9116c4a00000
                               CS: 0010  SS: 0018  DS: 0000
                               ES: 0000  FS: 0000  GS: 0000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000001 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         ffffa522c0003e20 0000000000000000
                         0000000000000001 0000000000000033
                         00000000000003ac 0000000000000046
                         000000000002bf40 00000000000003ab
                         ffffffff953f7c44 0000001d7358df44
                         0000000000000001 ffff9116c1f88000
                         0000000000000001 0000000000000002
                         ffffffff953f7c30 0000000000000000
                         ffffffff931497c4 0000000000000010
                         0000000000000002 ffffa522c0003dd0
                         0000000000000018 0000000000000000
                         ffff9116c4a00000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
Elf64_Nhdr:
               n_namesz: 5 ("CORE")
               n_descsz: 336
                 n_type: 1 (NT_PRSTATUS)
                         si.signo: 0  si.code: 0  si.errno: 0
                         cursig: 0  sigpend: 0  sighold: 0
                         pid: 2  ppid: 0  pgrp: 0  sid:0
                         utime: 0.000000  stime: 0.000000
                         cutime: 0.000000  cstime: 0.000000
                         ORIG_RAX: 0  fpvalid: 0
                              R15: 0000000000000001  R14: 000000000000000b
                              R13: 0000000000000000  R12: 0000000000000046
                              RBP: ffffa522c05dfe08  RBX: 0000000000000001
                              R11: 0000000000000000  R10: 0000000000000000
                               R9: 0000000000000000   R8: 0000000000000000
                              RAX: 0000000000000001  RCX: ffff9116c4001308
                              RDX: 0000000000000001  RSI: 0000000000000001
                              RDI: ffffffff9538506c  RIP: ffffffff9313ac20
                           RFLAGS: 0000000000000002  RSP: ffffa522c05dfd58
                          FS_BASE: 00007f205bfa1740
                          GS_BASE: ffff9116c4b00000
                               CS: 0010  SS: 0000  DS: 0000
                               ES: 0000  FS: 0000  GS: 0000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000002 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000001 000000000000000b
                         0000000000000000 0000000000000046
                         ffffa522c05dfe08 0000000000000001
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000001 ffff9116c4001308
                         0000000000000001 0000000000000001
                         ffffffff9538506c 0000000000000000
                         ffffffff9313ac20 0000000000000010
                         0000000000000002 ffffa522c05dfd58
                         0000000000000000 00007f205bfa1740
                         ffff9116c4b00000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
Elf64_Nhdr:
               n_namesz: 5 ("QEMU")
               n_descsz: 440
                 n_type: 0 (QEMUCPUState)
                         version: 1  size: 440
                         RAX: 0000000000000001     RBX: 0000000000000046
                         RCX: ffff9116c1f88000     RDX: 0000000000000001
                         RSI: 0000000000000002     RDI: ffffffff953f7c30
                         RSP: ffffa522c0003dd0     RBP: 00000000000003ac
                         RIP: ffffffff931497c4  RFLAGS: 0000000000000002
                          R8: 0000001d7358df44      R9: ffffffff953f7c44
                         R10: 00000000000003ab     R11: 000000000002bf40
                         R12: 0000000000000033     R13: 0000000000000001
                         R14: 0000000000000000     R15: ffffa522c0003e20
                          CS: selector: 0010  limit: ffffffff  flags: 00a09b00
                              pad: 00000000   base: 0000000000000000
                          DS: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: 0000000000000000
                          ES: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: 0000000000000000
                          FS: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: 0000000000000000
                          GS: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: ffff9116c4a00000
                          SS: selector: 0018  limit: ffffffff  flags: 00c09300
                              pad: 00000000   base: 0000000000000000
                         LDT: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: fffffe0000000000
                          TR: selector: 0040  limit: 00004087  flags: 00008b00
                              pad: 00000000   base: fffffe0000003000
                         GDT: selector: 0000  limit: 0000007f  flags: 00000000
                              pad: 00000000   base: fffffe0000001000
                         IDT: selector: 0000  limit: 00000fff  flags: 00000000
                              pad: 00000000   base: fffffe0000000000
                         CR0: 0000000080050033  CR1: 0000000000000000
                         CR2: 00000000055dde3f  CR3: 000000013ed6a000
                         CR4: 0000000000340ef0
                         000001b800000001 0000000000000001
                         0000000000000046 ffff9116c1f88000
                         0000000000000001 0000000000000002
                         ffffffff953f7c30 ffffa522c0003dd0
                         00000000000003ac 0000001d7358df44
                         ffffffff953f7c44 00000000000003ab
                         000000000002bf40 0000000000000033
                         0000000000000001 0000000000000000
                         ffffa522c0003e20 ffffffff931497c4
                         0000000000000002 ffffffff00000010
                         0000000000a09b00 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 ffff9116c4a00000
                         ffffffff00000018 0000000000c09300
                         0000000000000000 0000000000000000
                         0000000000000000 fffffe0000000000
                         0000408700000040 0000000000008b00
                         fffffe0000003000 0000007f00000000
                         0000000000000000 fffffe0000001000
                         00000fff00000000 0000000000000000
                         fffffe0000000000 0000000080050033
                         0000000000000000 00000000055dde3f
                         000000013ed6a000 0000000000340ef0
                         0000000000000000
Elf64_Nhdr:
               n_namesz: 5 ("QEMU")
               n_descsz: 440
                 n_type: 0 (QEMUCPUState)
                         version: 1  size: 440
                         RAX: 0000000000000001     RBX: 0000000000000001
                         RCX: ffff9116c4001308     RDX: 0000000000000001
                         RSI: 0000000000000001     RDI: ffffffff9538506c
                         RSP: ffffa522c05dfd58     RBP: ffffa522c05dfe08
                         RIP: ffffffff9313ac20  RFLAGS: 0000000000000002
                          R8: 0000000000000000      R9: 0000000000000000
                         R10: 0000000000000000     R11: 0000000000000000
                         R12: 0000000000000046     R13: 0000000000000000
                         R14: 000000000000000b     R15: 0000000000000001
                          CS: selector: 0010  limit: ffffffff  flags: 00a09b00
                              pad: 00000000   base: 0000000000000000
                          DS: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: 0000000000000000
                          ES: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: 0000000000000000
                          FS: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: 00007f205bfa1740
                          GS: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: ffff9116c4b00000
                          SS: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: 0000000000000000
                         LDT: selector: 0000  limit: 00000000  flags: 00000000
                              pad: 00000000   base: fffffe0000000000
                          TR: selector: 0040  limit: 00004087  flags: 00008b00
                              pad: 00000000   base: fffffe0000034000
                         GDT: selector: 0000  limit: 0000007f  flags: 00000000
                              pad: 00000000   base: fffffe0000032000
                         IDT: selector: 0000  limit: 00000fff  flags: 00000000
                              pad: 00000000   base: fffffe0000000000
                         CR0: 0000000080050033  CR1: 0000000000000000
                         CR2: ffff9116bc5247d8  CR3: 0000000141232000
                         CR4: 0000000000340ee0
                         000001b800000001 0000000000000001
                         0000000000000001 ffff9116c4001308
                         0000000000000001 0000000000000001
                         ffffffff9538506c ffffa522c05dfd58
                         ffffa522c05dfe08 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000046
                         0000000000000000 000000000000000b
                         0000000000000001 ffffffff9313ac20
                         0000000000000002 ffffffff00000010
                         0000000000a09b00 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         00007f205bfa1740 0000000000000000
                         0000000000000000 ffff9116c4b00000
                         0000000000000000 0000000000000000
                         0000000000000000 0000000000000000
                         0000000000000000 fffffe0000000000
                         0000408700000040 0000000000008b00
                         fffffe0000034000 0000007f00000000
                         0000000000000000 fffffe0000032000
                         00000fff00000000 0000000000000000
                         fffffe0000000000 0000000080050033
                         0000000000000000 ffff9116bc5247d8
                         0000000141232000 0000000000340ee0
                         0000000000000000
readmem: read_kdump()
<readmem: 13ed6a000, PHYSADDR, "pgd", 4096, (ROE), 55d7feca3270>
<read_kdump: addr: 13ed6a000 paddr: 13ed6a000 cnt: 4096>
read_netdump: addr: 0 paddr: 13ed6a000 cnt: 4096 offset: c6d8c860
PGD DIRECTORY: 13ed6a000
PAGE DIRECTORY: 147fc9067
<readmem: 147fc9000, PHYSADDR, "pud page", 4096, (FOE), 55d7feca4280>
<read_kdump: addr: 147fc9000 paddr: 147fc9000 cnt: 4096>
read_netdump: addr: 0 paddr: 147fc9000 cnt: 4096 offset: cffeb860
   PUD: 147fc9000 => 147fc7067
<readmem: 147fc7000, PHYSADDR, "pmd page", 4096, (FOE), 55d7feca5290>
<read_kdump: addr: 147fc7000 paddr: 147fc7000 cnt: 4096>
read_netdump: addr: 0 paddr: 147fc7000 cnt: 4096 offset: cffe9860
   PMD: 147fc7000 => 147fc6067
<readmem: 147fc6000, PHYSADDR, "page table", 4096, (FOE), 55d7feca62a0>
<read_kdump: addr: 147fc6000 paddr: 147fc6000 cnt: 4096>
read_netdump: addr: 0 paddr: 147fc6000 cnt: 4096 offset: cffe8860
   PTE: 147fc6000 => 800000005b37d161
  PAGE: 5b37d000

      PTE         PHYSICAL  FLAGS
800000005b37d161  5b37d000  (PRESENT|ACCESSED|DIRTY|GLOBAL|NX)
<readmem: 5b37d000, PHYSADDR, "idt_table", 16, (FOE), 7ffd756a2630>
<read_kdump: addr: 5b37d000 paddr: 5b37d000 cnt: 16>
read_netdump: addr: 0 paddr: 5b37d000 cnt: 16 offset: 5b35d860
calc_kaslr_offset: idtr=fffffe0000000000
calc_kaslr_offset: pgd=13ed6a000
calc_kaslr_offset: idtr(phys)=5b37d000
calc_kaslr_offset: divide_error(vmlinux): 0
calc_kaslr_offset: divide_error(vmcore): ffffffff93c00870
PGD DIRECTORY: 13ed6a000
PAGE DIRECTORY: 5aa0f067
<readmem: 5aa0f000, PHYSADDR, "pud page", 4096, (FOE), 55d7feca4280>
<read_kdump: addr: 5aa0f000 paddr: 5aa0f000 cnt: 4096>
read_netdump: addr: 0 paddr: 5aa0f000 cnt: 4096 offset: 5a9ef860
   PUD: 5aa0ffe0 => 0
kaslr_helper: failed to determine which kernel was running at crash,
kaslr_helper: asssuming the kdump 1st kernel.
calc_kaslr_offset: kaslr_offset=ffffffff93c00870
calc_kaslr_offset: phys_base=c43ff790
crash: pv_ops exists: ARCH_PVOPS
<readmem: 5a000200, PHYSADDR, "linux_banner verify", 13, (ROE|Q), 7ffd756a21f0>
<read_kdump: addr: 5a000200 paddr: 5a000200 cnt: 13>
read_netdump: addr: 0 paddr: 5a000200 cnt: 13 offset: 59fe0a60
gdb /data/vmlinux-5.8.4
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(328 -> 0)
  GETBUF(1500 -> 1)

WARNING: kernel relocated [1731MB]: patching 126940 gdb minimal_symbol values

  FREEBUF(1)
FREEBUF(0)
<readmem: ffffffff16083cc8, KVADDR, "__pgtable_l5_enabled", 4, (FOE|Q), 7ffd756a2b18>
<read_kdump: addr: ffffffff16083cc8 paddr: 77ff16083cc8 cnt: 4>
read_netdump: READ_ERROR: offset not found for paddr: 77ff16083cc8
crash: read error: kernel virtual address: ffffffff16083cc8  type: "__pgtable_l5_enabled"

Comment 2 Fedora Program Management 2021-04-29 16:37:05 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 3 Ben Cotton 2021-05-25 16:42:36 UTC
Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 6 lijiang 2021-06-15 06:05:21 UTC
Crash can not still work on the latest Fedora vmcore from "virsh dump". It reported a failure of initialization and eventually failed on the "qemu_load()". So reopen this issue and move it to rawhide. Thanks.

Comment 9 lijiang 2021-06-15 09:12:59 UTC
Hi, Emma, Ruowen and Kappa

It doesn't support to analyze a vmcore dumped by "virsh dump domain file", the vmcore must be dumped with option "--memory-only", otherwise it will fail to analyze vmcore.

Important
The crash utility no longer supports the default core dump file format of the virsh dump command. If you use crash to analyze a core dump file created by virsh dump, you must use the --memory-only option.


Here is a documentation link:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-domain_commands-creating_a_dump_file_of_a_domains_core

Thanks.

Comment 10 lijiang 2021-06-15 09:36:18 UTC
Hi, Kappa

Would you mind trying it again with the following command? I didn't see this issue on the latest crash-utility.

# virsh dump fedora34 /tmp/vmcore --memory-only --format=kdump-lzo

Domain 'fedora34' dumped to /tmp/vmcore

# crash /usr/lib/debug/usr/lib/modules/5.12.9-300.fc34.x86_64/vmlinux /tmp/vmcore 

crash 7.3.0-1.fc34
Copyright (C) 2002-2021  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6                              
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [192MB]: patching 134298 gdb minimal_symbol values

      KERNEL: /usr/lib/debug/usr/lib/modules/5.12.9-300.fc34.x86_64/vmlinux
    DUMPFILE: /tmp/vmcore  [PARTIAL DUMP]
        CPUS: 4
        DATE: Tue Jun 15 05:24:18 EDT 2021
      UPTIME: 00:31:45
LOAD AVERAGE: 0.00, 0.00, 0.00
       TASKS: 175
    NODENAME: fedora
     RELEASE: 5.12.9-300.fc34.x86_64
     VERSION: #1 SMP Thu Jun 3 13:51:40 UTC 2021
     MACHINE: x86_64  (2499 Mhz)
      MEMORY: 8 GB
       PANIC: ""
         PID: 0
     COMMAND: "swapper/0"
        TASK: ffffffff8ea1a940  (1 of 4)  [THREAD_INFO: ffffffff8ea1a940]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)
     WARNING: panic task not found

crash> bt
PID: 0      TASK: ffffffff8ea1a940  CPU: 0   COMMAND: "swapper/0"
    [exception RIP: native_safe_halt+14]
    RIP: ffffffff8dbf6e5e  RSP: ffffffff8ea03eb0  RFLAGS: 00000206
    RAX: ffffffff8dbf6d00  RBX: 0000000000000000  RCX: ffff99e7b7c2b180
    RDX: 0000000000000000  RSI: 0000000000000083  RDI: 0000000000000000
    RBP: ffffffff8ea1a940   R8: 000001beef6153db   R9: 0000000000000004
    R10: 0000000000000a08  R11: 0000000000000cb5  R12: 0000000000000000
    R13: 0000000000000000  R14: 000000000000008e  R15: 000000000026b508
    CS: 0010  SS: 0018
 #0 [ffffffff8ea03eb0] default_idle at ffffffff8dbf6d0a
 #1 [ffffffff8ea03eb8] default_idle_call at ffffffff8dbf6f78
 #2 [ffffffff8ea03ec0] do_idle at ffffffff8d114d00
 #3 [ffffffff8ea03ef8] cpu_startup_entry at ffffffff8d114f09
 #4 [ffffffff8ea03f08] start_kernel at ffffffff8f258631
 #5 [ffffffff8ea03f50] secondary_startup_64_no_verify at ffffffff8d000107
crash> kmem -s
CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
ffff99e646013a00      216          0         0      0     4k  nf_conntrack_expect
ffff99e646013000      256         48       125      5     8k  nf_conntrack
ffff99e64a5b5500      184          0         0      0     4k  nf-frags
ffff99e64a5b5b00      640         22        23      1    16k  rpc_inode_cache
ffff99e6462c3500     2048         15        16      1    32k  rpc_buffers
ffff99e6462c3b00      232         15        16      1     4k  rpc_tasks
ffff99e649d35e00      736         42        44      2    16k  fat_inode_cache
ffff99e649d35100       32          0         0      0     4k  fat_cache
ffff99e647b09000      136          0         0      0     4k  kvm_async_pf
ffff99e647b09f00     9728          0         0      0    32k  kvm_vcpu
ffff99e647b09d00      184          0         0      0     4k  kvm_mmu_page_header
ffff99e647b09300       32          0         0      0     4k  pte_list_desc
ffff99e647b09600     2672          0         0      0    32k  x86_emulator
ffff99e647b09200     4160          0         0      0    32k  x86_fpu
ffff99e646013f00       48         84        85      1     4k  zspage
ffff99e646013d00        8        511       512      1     4k  zs_handle
ffff99e647b09c00      152          0         0      0     4k  fuse_request
ffff99e647b09900      824          0         0      0    16k  fuse_inode
ffff99e6464d0600      528          0         0      0    16k  xfs_dqtrx
ffff99e6464d0200      496          0         0      0     8k  xfs_dquot
ffff99e6464d0c00      360       2159      2163    103     8k  xfs_buf
ffff99e649d35400      200          0         0      0     4k  xfs_bui_item
ffff99e649d35700      168          0         0      0     4k  xfs_bud_item
ffff99e649d35800      424          0         0      0     8k  xfs_cui_item
ffff99e649d35a00      168          0         0      0     4k  xfs_cud_item
ffff99e6409c9000      680          0         0      0    16k  xfs_rui_item
ffff99e6409c9f00      168          0         0      0     4k  xfs_rud_item
ffff99e6409c9d00      176          0         0      0     4k  xfs_icr
ffff99e6409c9300      192        113       126      6     4k  xfs_ili
ffff99e6409c9600      960       9490      9504    594    16k  xfs_inode
ffff99e6409c9200      424         50        57      3     8k  xfs_efi_item
crash>

Comment 14 Kappa 2021-06-16 06:32:07 UTC
It could run okay with Fedora 34 now.
But previously, at specific conditions, the virsh dump memory/core has problem.
It happened when the guest is at 100% processing power or hanged.

I tried to reproduce the problem (waiting for VM to hang) and try to run the virsh dump command.
Also, I used --memory-only option when I run the dump command in the past (https://bugzilla.redhat.com/show_bug.cgi?id=1876589#c0)

Usually the guest reboot by itself or hanged and used 100% processing power.
If it reboot my itself I can't run the virsh dump.
Usually the problem happened again within a few days.

Comment 15 Kappa 2021-06-16 18:43:18 UTC
When the guest VM has problem, I have test the virsh dump on it.
The crash program is okay now.

Version is crash-7.3.0-1.fc34.x86_64.

Comment 16 lijiang 2021-06-17 01:48:21 UTC
Thank you for the confirmation, Kappa.

This has been fixed on Fedora 34(crash-7.3.0-1.fc34) and rawhide(crash-7.3.0-1.fc35), so I will close it as current release.

https://src.fedoraproject.org/rpms/crash