Description of problem: The kdump utility in latest Rawhide image failed to work in QEMU installation. The image (Version Jul. 22nd) was fetched from https://mirrors.ustc.edu.cn/fedora/development/rawhide/Workstation/x86_64/iso/ The kdump.conf is kept as default, and crashkernel=160M The failure log is pasted below: kdump: dump target is /dev/mapper/fedora-root kdump: saving to /sysroot//var/crash/127.0.0.1-2017-07-25-16:31:53/ [ 3.688586] EXT4-fs (dm-0): re-mounted. Opts: (null) kdump: saving vmcore-dmesg.txt kdump: saving vmcore-dmesg.txt complete kdump: saving vmcore kdump: saving vmcore failed [FAILED] Failed to start Kdump Vmcore Save Service. See 'systemctl status kdump-capture.service' for details. [ SKIP ] Ordering cycle found, skipping System Initialization Stopping udev Kernel Device Manager... Stopping Journal Service... Version-Release number of selected component (if applicable): 2.0.15-4 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Can you try #default shell in kdump.conf and run the makedumpfile manually with --message-level=31
(In reply to Dave Young from comment #1) > Can you try #default shell in kdump.conf and run the makedumpfile manually > with --message-level=31 kdump:/# makedumpfile -l --message-level 31 -d 31 /proc/vmcore /var/crash/ sadump: does not have partition header sadump: read dump device as unknown format sadump: unknown format LOAD (0) phys_start : 44000000 phys_end : 46773000 virt_start : ffffffff8f000000 virt_end : ffffffff91773000 LOAD (1) phys_start : 1000 phys_end : 9fc00 virt_start : ffff8ef900001000 virt_end : ffff8ef90009fc00 LOAD (2) phys_start : 100000 phys_end : 2b000000 virt_start : ffff8ef900100000 virt_end : ffff8ef92b000000 LOAD (3) phys_start : 35000000 phys_end : 5dbe0000 virt_start : ffff8ef935000000 virt_end : ffff8ef95dbe0000 Linux kdump page_size : 4096 max_mapnr : 5dbe0 There is enough free memory to be done in one cycle. Buffer size for the cyclic mode: 95992 vtop4_x86_64: Can't get the symbol of init_level4_pgt. readmem: Can't convert a virtual address(ffffffff8fe18284) to physical address. readmem: type_addr: 0, addr:ffffffff8fe18284, size:390 check_release: Can't get the address of system_utsname. makedumpfile Failed.
Try boot 1st kernel with "nokaslr" kernel cmdline param see if it still fail..
(In reply to Dave Young from comment #3) > Try boot 1st kernel with "nokaslr" kernel cmdline param see if it still > fail.. kdump:/# makedumpfile -l --message-level 31 -d 31 /proc/vmcore /var/crash/ sadump: does not have partition header sadump: read dump device as unknown format sadump: unknown format LOAD (0) phys_start : 1000000 phys_end : 3773000 virt_start : ffffffff81000000 virt_end : ffffffff83773000 LOAD (1) phys_start : 1000 phys_end : 9fc00 virt_start : ffff880000001000 virt_end : ffff88000009fc00 LOAD (2) phys_start : 100000 phys_end : 2b000000 virt_start : ffff880000100000 virt_end : ffff88002b000000 LOAD (3) phys_start : 35000000 phys_end : 5dbe0000 virt_start : ffff880035000000 virt_end : ffff88005dbe0000 Linux kdump page_size : 4096 max_mapnr : 5dbe0 There is enough free memory to be done in one cycle. Buffer size for the cyclic mode: 95992 vtop4_x86_64: Can't get the symbol of init_level4_pgt. readmem: Can't convert a virtual address(ffffffff81e18284) to physical address. readmem: type_addr: 0, addr:ffffffff81e18284, size:390 check_release: Can't get the address of system_utsname. makedumpfile Failed.
Maybe Pratyush has some thoughts about this..
Bisected the first bad commit is below, replace all init_level4_pgt with new init_top_pgt, kdump works fine. But maybe in makedumpfile we should use the new variable first if it fails then switch to old name. Pratyush, can you post a fix to upstream? 65ade2f872b474fa8a04c2d397783350326634e6 is the first bad commit commit 65ade2f872b474fa8a04c2d397783350326634e6 Author: Kirill A. Shutemov <kirill.shutemov.com> Date: Tue Jun 6 14:31:27 2017 +0300 x86/boot/64: Rename init_level4_pgt and early_level4_pgt With CONFIG_X86_5LEVEL=y, level 4 is no longer top level of page tables. Let's give these variable more generic names: init_top_pgt and early_top_pgt.
Oh, sorry..Patch has been posted to upstream only and not for fedora. So, moving back to assigned.
How can I get this fix for the fedora 26? I use it in the kernel development...
(In reply to Daniel Bristot de Oliveira from comment #8) > How can I get this fix for the fedora 26? I use it in the kernel > development... Can you please try kexec-tools rpm from following. https://koji.fedoraproject.org/koji/taskinfo?taskID=21387027 If that works, will send the backport patch for fc26.
It returns... kdump:/# makedumpfile -l --message-level 31 -d 31 /proc/vmcore /var/crash/ sadump: does not have partition header sadump: read dump device as unknown format sadump: unknown format LOAD (0) phys_start : 1000000 phys_end : 24bf000 virt_start : ffffffff81000000 virt_end : ffffffff824bf000 LOAD (1) phys_start : 1000 phys_end : 9fc00 virt_start : ffff880000001000 virt_end : ffff88000009fc00 LOAD (2) phys_start : 100000 phys_end : f000000 virt_start : ffff880000100000 virt_end : ffff88000f000000 LOAD (3) phys_start : 2f000000 phys_end : bffe0000 virt_start : ffff88002f000000 virt_end : ffff8800bffe0000 LOAD (4) phys_start : 100000000 phys_end : 140000000 virt_start : ffff880100000000 virt_end : ffff880140000000 Linux kdump page_size : 4096 max_mapnr : 140000 There is enough free memory to be done in one cycle. Buffer size for the cyclic mode: 327680 The kernel version is not supported. The makedumpfile operation may be incomplete. num of NODEs : 1 Memory type : SPARSEMEM_EX mem_map (0) mem_map : ffffea0000000004 pfn_start : 0 pfn_end : 8000 mem_map (1) mem_map : ffffea0000200004 pfn_start : 8000 pfn_end : 10000 mem_map (2) mem_map : ffffea0000400004 pfn_start : 10000 pfn_end : 18000 mem_map (3) mem_map : ffffea0000600004 pfn_start : 18000 pfn_end : 20000 mem_map (4) mem_map : ffffea0000800004 pfn_start : 20000 pfn_end : 28000 mem_map (5) mem_map : ffffea0000a00004 pfn_start : 28000 pfn_end : 30000 mem_map (6) mem_map : ffffea0000c00004 pfn_start : 30000 pfn_end : 38000 mem_map (7) mem_map : ffffea0000e00004 pfn_start : 38000 pfn_end : 40000 mem_map (8) mem_map : ffffea0001000004 pfn_start : 40000 pfn_end : 48000 mem_map (9) mem_map : ffffea0001200004 pfn_start : 48000 pfn_end : 50000 mem_map (10) mem_map : ffffea0001400004 pfn_start : 50000 pfn_end : 58000 mem_map (11) mem_map : ffffea0001600004 pfn_start : 58000 pfn_end : 60000 mem_map (12) mem_map : ffffea0001800004 pfn_start : 60000 pfn_end : 68000 mem_map (13) mem_map : ffffea0001a00004 pfn_start : 68000 pfn_end : 70000 mem_map (14) mem_map : ffffea0001c00004 pfn_start : 70000 pfn_end : 78000 mem_map (15) mem_map : ffffea0001e00004 pfn_start : 78000 pfn_end : 80000 mem_map (16) mem_map : ffffea0002000004 pfn_start : 80000 pfn_end : 88000 mem_map (17) mem_map : ffffea0002200004 pfn_start : 88000 pfn_end : 90000 mem_map (18) mem_map : ffffea0002400004 pfn_start : 90000 pfn_end : 98000 mem_map (19) mem_map : ffffea0002600004 pfn_start : 98000 pfn_end : a0000 mem_map (20) mem_map : ffffea0002800004 pfn_start : a0000 pfn_end : a8000 mem_map (21) mem_map : ffffea0002a00004 pfn_start : a8000 pfn_end : b0000 mem_map (22) mem_map : ffffea0002c00004 pfn_start : b0000 pfn_end : b8000 mem_map (23) mem_map : ffffea0002e00004 pfn_start : b8000 pfn_end : c0000 mem_map (24) mem_map : 0 pfn_start : c0000 pfn_end : c8000 mem_map (25) mem_map : 0 pfn_start : c8000 pfn_end : d0000 mem_map (26) mem_map : 0 pfn_start : d0000 pfn_end : d8000 mem_map (27) mem_map : 0 pfn_start : d8000 pfn_end : e0000 mem_map (28) mem_map : 0 pfn_start : e0000 pfn_end : e8000 mem_map (29) mem_map : 0 pfn_start : e8000 pfn_end : f0000 mem_map (30) mem_map : 0 pfn_start : f0000 pfn_end : f8000 mem_map (31) mem_map : 0 pfn_start : f8000 pfn_end : 100000 mem_map (32) mem_map : ffffea0004000004 pfn_start : 100000 pfn_end : 108000 mem_map (33) mem_map : ffffea0004200004 pfn_start : 108000 pfn_end : 110000 mem_map (34) mem_map : ffffea0004400004 pfn_start : 110000 pfn_end : 118000 mem_map (35) mem_map : ffffea0004600004 pfn_start : 118000 pfn_end : 120000 mem_map (36) mem_map : ffffea0004800004 pfn_start : 120000 pfn_end : 128000 mem_map (37) mem_map : ffffea0004a00004 pfn_start : 128000 pfn_end : 130000 mem_map (38) mem_map : ffffea0004c00004 pfn_start : 130000 pfn_end : 138000 mem_map (39) mem_map : ffffea0004e00004 pfn_start : 138000 pfn_end : 140000 mmap() is available on the kernel. Checking for memory holes : [100.0 %] |STEP [Checking for memory holes ] : 0.006703 seconds vtop4_x86_64: Can't get a valid pmd_pte. readmem: Can't convert a virtual address(ffffea0003000000) to physical address. readmem: type_addr: 0, addr:ffffea0003000000, size:32768 __exclude_unnecessary_pages: Can't read the buffer of struct page. create_2nd_bitmap: Can't exclude unnecessary pages. makedumpfile Failed.
(In reply to Daniel Bristot de Oliveira from comment #10) > makedumpfile Failed. So, I created a fc26 vm and tested it at my end, and it did work. Did you made sure that initramfs has been rebuilt after new rpm installation? You won't have new makedumpfile command without rebuilt. [Between, what kernel version are you using for fc26? It seems fc26 has 4.11.8-300.fc26.x86_64,which should work even with old makedumpfile as well. I tested makedumpfile from my koji build with both default fc26 kernel and kernel 4.13.0-0.rc6.git0.1.fc27.x86_64 and it worked.] Since, still FC26 has kernel < v4.13,so I do not think we need any patch for makedumpfile from this BZ in fc26.
(In reply to Daniel Bristot de Oliveira from comment #10) > Checking for memory holes : [100.0 %] |STEP [Checking for memory > holes ] : 0.006703 seconds > vtop4_x86_64: Can't get a valid pmd_pte. > readmem: Can't convert a virtual address(ffffea0003000000) to physical > address. > readmem: type_addr: 0, addr:ffffea0003000000, size:32768 > __exclude_unnecessary_pages: Can't read the buffer of struct page. > create_2nd_bitmap: Can't exclude unnecessary pages. I see, This problem looks different than that of this BZ. Probably you can have a new bug for it. Specify kernel version and kexec-tools version where you observed it.
(In reply to Daniel Bristot de Oliveira from comment #10) > It returns... > > kdump:/# makedumpfile -l --message-level 31 -d 31 /proc/vmcore /var/crash/ > sadump: does not have partition header > sadump: read dump device as unknown format > sadump: unknown format > LOAD (0) > phys_start : 1000000 > phys_end : 24bf000 > virt_start : ffffffff81000000 > virt_end : ffffffff824bf000 > LOAD (1) > phys_start : 1000 > phys_end : 9fc00 > virt_start : ffff880000001000 > virt_end : ffff88000009fc00 > LOAD (2) > phys_start : 100000 > phys_end : f000000 > virt_start : ffff880000100000 > virt_end : ffff88000f000000 > LOAD (3) > phys_start : 2f000000 > phys_end : bffe0000 > virt_start : ffff88002f000000 > virt_end : ffff8800bffe0000 > LOAD (4) > phys_start : 100000000 > phys_end : 140000000 > virt_start : ffff880100000000 > virt_end : ffff880140000000 > Linux kdump > page_size : 4096 > > max_mapnr : 140000 > There is enough free memory to be done in one cycle. > > Buffer size for the cyclic mode: 327680 > The kernel version is not supported. > The makedumpfile operation may be incomplete. > > num of NODEs : 1 > > > Memory type : SPARSEMEM_EX > > mem_map (0) > mem_map : ffffea0000000004 > pfn_start : 0 > pfn_end : 8000 > mem_map (1) > mem_map : ffffea0000200004 > pfn_start : 8000 > pfn_end : 10000 > mem_map (2) > mem_map : ffffea0000400004 > pfn_start : 10000 > pfn_end : 18000 > mem_map (3) > mem_map : ffffea0000600004 > pfn_start : 18000 > pfn_end : 20000 > mem_map (4) > mem_map : ffffea0000800004 > pfn_start : 20000 > pfn_end : 28000 > mem_map (5) > mem_map : ffffea0000a00004 > pfn_start : 28000 > pfn_end : 30000 > mem_map (6) > mem_map : ffffea0000c00004 > pfn_start : 30000 > pfn_end : 38000 > mem_map (7) > mem_map : ffffea0000e00004 > pfn_start : 38000 > pfn_end : 40000 > mem_map (8) > mem_map : ffffea0001000004 > pfn_start : 40000 > pfn_end : 48000 > mem_map (9) > mem_map : ffffea0001200004 > pfn_start : 48000 > pfn_end : 50000 > mem_map (10) > mem_map : ffffea0001400004 > pfn_start : 50000 > pfn_end : 58000 > mem_map (11) > mem_map : ffffea0001600004 > pfn_start : 58000 > pfn_end : 60000 > mem_map (12) > mem_map : ffffea0001800004 > pfn_start : 60000 > pfn_end : 68000 > mem_map (13) > mem_map : ffffea0001a00004 > pfn_start : 68000 > pfn_end : 70000 > mem_map (14) > mem_map : ffffea0001c00004 > pfn_start : 70000 > pfn_end : 78000 > mem_map (15) > mem_map : ffffea0001e00004 > pfn_start : 78000 > pfn_end : 80000 > mem_map (16) > mem_map : ffffea0002000004 > pfn_start : 80000 > pfn_end : 88000 > mem_map (17) > mem_map : ffffea0002200004 > pfn_start : 88000 > pfn_end : 90000 > mem_map (18) > mem_map : ffffea0002400004 > pfn_start : 90000 > pfn_end : 98000 > mem_map (19) > mem_map : ffffea0002600004 > pfn_start : 98000 > pfn_end : a0000 > mem_map (20) > mem_map : ffffea0002800004 > pfn_start : a0000 > pfn_end : a8000 > mem_map (21) > mem_map : ffffea0002a00004 > pfn_start : a8000 > pfn_end : b0000 > mem_map (22) > mem_map : ffffea0002c00004 > pfn_start : b0000 > pfn_end : b8000 > mem_map (23) > mem_map : ffffea0002e00004 > pfn_start : b8000 > pfn_end : c0000 > mem_map (24) > mem_map : 0 > pfn_start : c0000 > pfn_end : c8000 > mem_map (25) > mem_map : 0 > pfn_start : c8000 > pfn_end : d0000 > mem_map (26) > mem_map : 0 > pfn_start : d0000 > pfn_end : d8000 > mem_map (27) > mem_map : 0 > pfn_start : d8000 > pfn_end : e0000 > mem_map (28) > mem_map : 0 > pfn_start : e0000 > pfn_end : e8000 > mem_map (29) > mem_map : 0 > pfn_start : e8000 > pfn_end : f0000 > mem_map (30) > mem_map : 0 > pfn_start : f0000 > pfn_end : f8000 > mem_map (31) > mem_map : 0 > pfn_start : f8000 > pfn_end : 100000 > mem_map (32) > mem_map : ffffea0004000004 > pfn_start : 100000 > pfn_end : 108000 > mem_map (33) > mem_map : ffffea0004200004 > pfn_start : 108000 > pfn_end : 110000 > mem_map (34) > mem_map : ffffea0004400004 > pfn_start : 110000 > pfn_end : 118000 > mem_map (35) > mem_map : ffffea0004600004 > pfn_start : 118000 > pfn_end : 120000 > mem_map (36) > mem_map : ffffea0004800004 > pfn_start : 120000 > pfn_end : 128000 > mem_map (37) > mem_map : ffffea0004a00004 > pfn_start : 128000 > pfn_end : 130000 > mem_map (38) > mem_map : ffffea0004c00004 > pfn_start : 130000 > pfn_end : 138000 > mem_map (39) > mem_map : ffffea0004e00004 > pfn_start : 138000 > pfn_end : 140000 > mmap() is available on the kernel. > Checking for memory holes : [100.0 %] |STEP [Checking for memory > holes ] : 0.006703 seconds > vtop4_x86_64: Can't get a valid pmd_pte. > readmem: Can't convert a virtual address(ffffea0003000000) to physical > address. > readmem: type_addr: 0, addr:ffffea0003000000, size:32768 > __exclude_unnecessary_pages: Can't read the buffer of struct page. > create_2nd_bitmap: Can't exclude unnecessary pages. > > makedumpfile Failed. Hi, Tested with 1. upstream kernel git hash 42ff72cf (Aug 30 2017) 2. kexec-tools-2.0.15-1.fc26.3.x86_64 3. makedumpfile utility from kexec-tools-2.0.15-12.fc28.x86_64 and it works. Maybe you can use the newer makedumpfile as a workaround.