Bug 1440587 - kexec: failed load kdump kernel - Can't find kernel text map area from kcore
Summary: kexec: failed load kdump kernel - Can't find kernel text map area from kcore
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kexec-tools
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Pratyush Anand
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1441452 1448115
TreeView+ depends on / blocked
 
Reported: 2017-04-10 03:26 UTC by Qiao Zhao
Modified: 2017-05-04 14:47 UTC (History)
5 users (show)

Fixed In Version: kexec-tools-2.0.13-7.fc25.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1441452 (view as bug list)
Environment:
Last Closed: 2017-04-12 20:23:42 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Qiao Zhao 2017-04-10 03:26:15 UTC
Description of problem:
    + is_secure_boot_enforced
    + local secure_boot_file setup_mode_file
    + local secure_boot_byte setup_mode_byte
    ++ find /sys/firmware/efi/efivars -name 'SecureBoot-*'
    + secure_boot_file=
    ++ find /sys/firmware/efi/efivars -name 'SetupMode-*'
    + setup_mode_file=
    + '[' -f '' ']'
    + return 1
    + /sbin/kexec -p '--command-line=BOOT_IMAGE=/vmlinuz-4.10.8-200.fc25.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap LANG=en_US.UTF-8 console=ttyS0 console=ttyS1 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never disable_cpu_apicid=0' --initrd=/boot/initramfs-4.10.8-200.fc25.x86_64kdump.img /boot/vmlinuz-4.10.8-200.fc25.x86_64
    Can't find kernel text map area from kcore
    Cannot load /boot/vmlinuz-4.10.8-200.fc25.x86_64
    + '[' 255 == 0 ']'
    + echo 'kexec: failed to load kdump kernel'
    kexec: failed to load kdump kernel
    + return 1
    + return 1
    + '[' 1 '!=' 0 ']'
    + echo 'Starting kdump: [FAILED]'
    Starting kdump: [FAILED]
    + return 1
    + exit 1


Version-Release number of selected component (if applicable):
kernel-4.10.8-200.fc25.x86_64
kexec-tools-2.0.13-7.fc25.1.x86_64

How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Pratyush Anand 2017-04-10 12:35:20 UTC
I have a patch (https://github.com/pratyushanand/kexec-tools/commit/2c8ce09f420fb0f1c3b4d700dc00f3a4ffbf94f5) which fixes the reported issue. However, crash still did not like the vmcore. 

# uname -r
4.10.8-200.fc25.x86_64
# rpm -qa | grep crash
crash-7.1.7-1.fc25.x86_64

kexec-tools from:
https://koji.fedoraproject.org/koji/taskinfo?taskID=18898972 

I get following
crash: invalid structure size: tnt
       FILE: kernel.c  LINE: 10458  FUNCTION: show_kernel_taints()

[/usr/bin/crash] error trace: 4ea605 => 4dbc45 => 4d801c => 536752

  536752: SIZE_verify+146
  4d801c: (undetermined)
  4dbc45: (undetermined)
  4ea605: display_sys_stats+2213


So, it seems that we will have to take following crash commit for fc25 as well.

commit 24a696228c56fd4354d29abe05b206373e0c8bfb
Author: Dave Anderson <anderson>
Date:   Thu Jan 5 14:55:18 2017 -0500

    Fix for Linux 4.10 commit 7fd8329ba502ef76dd91db561c7aed696b2c7720
    "taint/module: Clean up global and module taint flags handling".
    Without the patch, when running against Linux 4.10-rc1 and later
    kernels, the crash utility fails during session initialization with
    the message "crash: invalid structure size: tnt".
    (panand)


I tried to compile crash natively, but this is what I see.

-------------------------------------------------------------------------------
	   ../readline/libreadline.a ../opcodes/libopcodes.a ../bfd/libbfd.a  ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a    -ldl -lncurses -lz -lm  -llzma ../libiberty/libiberty.a  build-gnulib/import/libgnu.a -ldl -Wl,--dynamic-list=./proc-service.list -llzo2 -lz -ldl -rdynamic
c-exp.o: In function `main':
/root/crash/gdb-7.6/gdb/c-exp.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/crash/main.c:81: first defined here
cp-name-parser.o: In function `main':
/root/crash/gdb-7.6/gdb/cp-name-parser.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/crash/main.c:1948: first defined here
ada-exp.o: In function `main':
/root/crash/gdb-7.6/gdb/ada-exp.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/crash/main.c:1948: first defined here
f-exp.o: In function `main':
/root/crash/gdb-7.6/gdb/f-exp.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/crash/main.c:1948: first defined here
p-exp.o: In function `main':
/root/crash/gdb-7.6/gdb/p-exp.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/crash/main.c:1948: first defined here
go-exp.o: In function `parse_string_or_char':
/root/crash/gdb-7.6/gdb/go-exp.y:943: undefined reference to `c_parse_escape'
macroexp.o: In function `get_string_literal':
/root/crash/gdb-7.6/gdb/macroexp.c:418: undefined reference to `c_parse_escape'
macroexp.o: In function `get_character_constant':
/root/crash/gdb-7.6/gdb/macroexp.c:364: undefined reference to `c_parse_escape'
ada-lang.o: In function `parse':
/root/crash/gdb-7.6/gdb/ada-lang.c:12703: undefined reference to `ada_parse'
ada-lang.o:(.rodata+0x1b90): undefined reference to `ada_error'
c-lang.o:(.rodata+0x28): undefined reference to `c_parse'
c-lang.o:(.rodata+0x30): undefined reference to `c_error'
c-lang.o:(.rodata+0x148): undefined reference to `c_parse'
c-lang.o:(.rodata+0x150): undefined reference to `c_error'
c-lang.o:(.rodata+0x268): undefined reference to `c_parse'
c-lang.o:(.rodata+0x270): undefined reference to `c_error'
c-lang.o:(.rodata+0x388): undefined reference to `c_parse'
c-lang.o:(.rodata+0x390): undefined reference to `c_error'
d-lang.o:(.rodata+0x1e8): undefined reference to `c_parse'
d-lang.o:(.rodata+0x1f0): undefined reference to `c_error'
f-lang.o:(.rodata+0x28): undefined reference to `f_parse'
f-lang.o:(.rodata+0x30): undefined reference to `f_error'
----------------------------------------------------------------------------

Do not know what is the issue. I have following on the beaker machine. Dave, any  idea, what am I missing? See, if you can help with a fc25 crash utility which works with scp at least. You can use kexec-tools from https://koji.fedoraproject.org/koji/taskinfo?taskID=18898972.

[root@ibm-ls22-03 crash]# rpm -qa | grep -e glibc -e ncurses -e lzo -e zlib -e snappy
glibc-langpack-en-2.24-4.fc25.x86_64
ncurses-libs-6.0-6.20160709.fc25.x86_64
ncurses-c++-libs-6.0-6.20160709.fc25.x86_64
glibc-headers-2.24-4.fc25.x86_64
ncurses-devel-6.0-6.20160709.fc25.x86_64
lzo-devel-2.08-8.fc24.x86_64
ncurses-base-6.0-6.20160709.fc25.noarch
glibc-2.24-4.fc25.x86_64
ncurses-compat-libs-6.0-6.20160709.fc25.x86_64
glibc-devel-2.24-4.fc25.x86_64
lzo-2.08-8.fc24.x86_64
snappy-devel-1.1.3-2.fc24.x86_64
ncurses-6.0-6.20160709.fc25.x86_64
glibc-common-2.24-4.fc25.x86_64
snappy-1.1.3-2.fc24.x86_64
zlib-1.2.8-10.fc24.x86_64
lzo-minilzo-2.08-8.fc24.x86_64
zlib-static-1.2.8-10.fc24.x86_64
zlib-devel-1.2.8-10.fc24.x86_64

Comment 2 Dave Anderson 2017-04-10 13:31:44 UTC
(In reply to Pratyush Anand from comment #1)
> I have a patch
> (https://github.com/pratyushanand/kexec-tools/commit/
> 2c8ce09f420fb0f1c3b4d700dc00f3a4ffbf94f5) which fixes the reported issue.
> However, crash still did not like the vmcore. 
> 
> # uname -r
> 4.10.8-200.fc25.x86_64
> # rpm -qa | grep crash
> crash-7.1.7-1.fc25.x86_64
> 
> kexec-tools from:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=18898972 
> 
> I get following
> crash: invalid structure size: tnt
>        FILE: kernel.c  LINE: 10458  FUNCTION: show_kernel_taints()
> 
> [/usr/bin/crash] error trace: 4ea605 => 4dbc45 => 4d801c => 536752
> 
>   536752: SIZE_verify+146
>   4d801c: (undetermined)
>   4dbc45: (undetermined)
>   4ea605: display_sys_stats+2213
> 
> 
> So, it seems that we will have to take following crash commit for fc25 as
> well.
> 
> commit 24a696228c56fd4354d29abe05b206373e0c8bfb
> Author: Dave Anderson <anderson>
> Date:   Thu Jan 5 14:55:18 2017 -0500
> 
>     Fix for Linux 4.10 commit 7fd8329ba502ef76dd91db561c7aed696b2c7720
>     "taint/module: Clean up global and module taint flags handling".
>     Without the patch, when running against Linux 4.10-rc1 and later
>     kernels, the crash utility fails during session initialization with
>     the message "crash: invalid structure size: tnt".
>     (panand)

Since Fedora kernels are constantly updated to recent upstream versions,
it's only a matter of time before the crash utility breaks.  For the issue
above, crash-7.1.8-1.fc26 has the patch above, and has been available 
in Rawhide since Feb 23rd:

  Information for build crash-7.1.8-1.fc26
  https://koji.fedoraproject.org/koji/buildinfo?buildID=861562

Keep this link handy somewhere:

  Information for package crash
  https://koji.fedoraproject.org/koji/packageinfo?packageID=307

> 
> I tried to compile crash natively, but this is what I see.
> 
> -----------------------------------------------------------------------------
> --
> 	   ../readline/libreadline.a ../opcodes/libopcodes.a ../bfd/libbfd.a 
> ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a    -ldl -lncurses
> -lz -lm  -llzma ../libiberty/libiberty.a  build-gnulib/import/libgnu.a -ldl
> -Wl,--dynamic-list=./proc-service.list -llzo2 -lz -ldl -rdynamic
> c-exp.o: In function `main':
> /root/crash/gdb-7.6/gdb/c-exp.c:1: multiple definition of `main'
> ../../crashlib.a(main.o):/root/crash/main.c:81: first defined here
> cp-name-parser.o: In function `main':
> /root/crash/gdb-7.6/gdb/cp-name-parser.c:1: multiple definition of `main'
> ../../crashlib.a(main.o):/root/crash/main.c:1948: first defined here
> ada-exp.o: In function `main':
> /root/crash/gdb-7.6/gdb/ada-exp.c:1: multiple definition of `main'
> ../../crashlib.a(main.o):/root/crash/main.c:1948: first defined here
> f-exp.o: In function `main':
> /root/crash/gdb-7.6/gdb/f-exp.c:1: multiple definition of `main'
> ../../crashlib.a(main.o):/root/crash/main.c:1948: first defined here
> p-exp.o: In function `main':
> /root/crash/gdb-7.6/gdb/p-exp.c:1: multiple definition of `main'
> ../../crashlib.a(main.o):/root/crash/main.c:1948: first defined here
> go-exp.o: In function `parse_string_or_char':
> /root/crash/gdb-7.6/gdb/go-exp.y:943: undefined reference to `c_parse_escape'
> macroexp.o: In function `get_string_literal':
> /root/crash/gdb-7.6/gdb/macroexp.c:418: undefined reference to
> `c_parse_escape'
> macroexp.o: In function `get_character_constant':
> /root/crash/gdb-7.6/gdb/macroexp.c:364: undefined reference to
> `c_parse_escape'
> ada-lang.o: In function `parse':
> /root/crash/gdb-7.6/gdb/ada-lang.c:12703: undefined reference to `ada_parse'
> ada-lang.o:(.rodata+0x1b90): undefined reference to `ada_error'
> c-lang.o:(.rodata+0x28): undefined reference to `c_parse'
> c-lang.o:(.rodata+0x30): undefined reference to `c_error'
> c-lang.o:(.rodata+0x148): undefined reference to `c_parse'
> c-lang.o:(.rodata+0x150): undefined reference to `c_error'
> c-lang.o:(.rodata+0x268): undefined reference to `c_parse'
> c-lang.o:(.rodata+0x270): undefined reference to `c_error'
> c-lang.o:(.rodata+0x388): undefined reference to `c_parse'
> c-lang.o:(.rodata+0x390): undefined reference to `c_error'
> d-lang.o:(.rodata+0x1e8): undefined reference to `c_parse'
> d-lang.o:(.rodata+0x1f0): undefined reference to `c_error'
> f-lang.o:(.rodata+0x28): undefined reference to `f_parse'
> f-lang.o:(.rodata+0x30): undefined reference to `f_error'
> ----------------------------------------------------------------------------
> 
> Do not know what is the issue. I have following on the beaker machine. Dave,
> any  idea, what am I missing? See, if you can help with a fc25 crash utility
> which works with scp at least. You can use kexec-tools from
> https://koji.fedoraproject.org/koji/taskinfo?taskID=18898972.
> 
> [root@ibm-ls22-03 crash]# rpm -qa | grep -e glibc -e ncurses -e lzo -e zlib
> -e snappy
> glibc-langpack-en-2.24-4.fc25.x86_64
> ncurses-libs-6.0-6.20160709.fc25.x86_64
> ncurses-c++-libs-6.0-6.20160709.fc25.x86_64
> glibc-headers-2.24-4.fc25.x86_64
> ncurses-devel-6.0-6.20160709.fc25.x86_64
> lzo-devel-2.08-8.fc24.x86_64
> ncurses-base-6.0-6.20160709.fc25.noarch
> glibc-2.24-4.fc25.x86_64
> ncurses-compat-libs-6.0-6.20160709.fc25.x86_64
> glibc-devel-2.24-4.fc25.x86_64
> lzo-2.08-8.fc24.x86_64
> snappy-devel-1.1.3-2.fc24.x86_64
> ncurses-6.0-6.20160709.fc25.x86_64
> glibc-common-2.24-4.fc25.x86_64
> snappy-1.1.3-2.fc24.x86_64
> zlib-1.2.8-10.fc24.x86_64
> lzo-minilzo-2.08-8.fc24.x86_64
> zlib-static-1.2.8-10.fc24.x86_64
> zlib-devel-1.2.8-10.fc24.x86_64

The "c_parse" etc. errors that you are seeing are typically due to
the fact that your host machine does not have /usr/bin/bison or
/usr/bin/flex installed.

If you build with a src.rpm, rpmbuild will catch the missing package.
That was added to the crash.spec file a while ago:

7.0.2    - Added "bison" to the BuildRequires line of the crash.spec file.
           Without the patch, the build of the embedded gdb-7.6 module will fail
           unless either /usr/bin/bison or /usr/bin/yacc are available.  The
           failure will result in a stream of error messages from different
           files that indicate:

             multiple definition of 'main'
             undefined reference to 'c_parse_escape'
             undefined reference to 'ada_parse'
             undefined reference to 'ada_error'
             undefined reference to 'c_parse'
             undefined reference to 'c_error'
             undefined reference to 'cp_demangled_name_to_comp'
             undefined reference to 'cp_demangled_name_parse_free'
             undefined reference to 'cp_comp_to_string'
             undefined reference to 'cp_new_demangle_parse_info'

           and the build fails like so:

             collect2: ld returned 1 exit status
             make[4]: *** [gdb] Error 1
             crash build failed

Comment 3 Pratyush Anand 2017-04-10 15:14:38 UTC
Hi Dave,

(In reply to Dave Anderson from comment #2)
>   Information for build crash-7.1.8-1.fc26
>   https://koji.fedoraproject.org/koji/buildinfo?buildID=861562

I am not able to install it on fc25.

# rpm -Uvh crash-7.1.8-1.fc26.x86_64.rpm 
error: Failed dependencies:
	libreadline.so.7()(64bit) is needed by crash-7.1.8-1.fc26.x86_64

Probably, we will need a fc25 version as well.

> 

> The "c_parse" etc. errors that you are seeing are typically due to
> the fact that your host machine does not have /usr/bin/bison or
> /usr/bin/flex installed.

Thanks, that helped. I had bison, but flex was missing. vmcore was fine with locally compiled crash.

Comment 4 Fedora Update System 2017-04-11 08:40:58 UTC
kexec-tools-2.0.13-7.fc25.2 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-257bdbacc7

Comment 7 Fedora Update System 2017-04-11 19:25:13 UTC
kexec-tools-2.0.13-7.fc25.2 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-257bdbacc7

Comment 9 Fedora Update System 2017-04-12 20:23:42 UTC
kexec-tools-2.0.13-7.fc25.2 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 10 Dave Anderson 2017-04-26 13:59:34 UTC
Note that the most recent (crash-7.1.9) crash utility does not work
with dumpfiles created with kexec-tools-2.0.13-7.fc25.2.x86_64:

 https://bugzilla.redhat.com/show_bug.cgi?id=1441452#c4

Like this bugzilla, I've got a fc25 update for crash in play as well.
We should probably have coordinated our updates to make sure everything
worked before pushing Fedora updates.

Comment 11 Dave Anderson 2017-04-26 18:33:07 UTC
(In reply to Dave Anderson from comment #10)
> Note that the most recent (crash-7.1.9) crash utility does not work
> with dumpfiles created with kexec-tools-2.0.13-7.fc25.2.x86_64:
> 
>  https://bugzilla.redhat.com/show_bug.cgi?id=1441452#c4
> 
> Like this bugzilla, I've got a fc25 update for crash in play as well.
> We should probably have coordinated our updates to make sure everything
> worked before pushing Fedora updates.

The failure reported by Qiao in the bugzilla above was on a 4.10.8-200.fc25 
kernel.  However, in my testing using a 4.10.11-200.fc25 kernel, crash and
the kdump vmcore works OK:

  https://bugzilla.redhat.com/show_bug.cgi?id=1441452#c7

I put a NEEDINFO there asking any of you who have worked on the kernel
if there is any kernel/kdump differences between 4.10.8-200.fc25 and
4.10.11-200.fc25.


Note You need to log in before you can comment on or make changes to this bug.