Bug 1421657

Summary: crash: invalid structure size: tnt
Product: [Fedora] Fedora Reporter: Jitendra Khasdev <jkhasdev>
Component: crashAssignee: Dave Anderson <anderson>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: anderson
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: crash-7.1.8-1.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-23 21:45:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jitendra Khasdev 2017-02-13 11:37:18 UTC
Description of problem:

crash utility crashed with following error:

[root@localhost 127.0.0.1-2017-02-13-15:31:16]# crash ~/btrfs/build/vmlinux vmcore

crash 7.1.7-1.fc25
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

                                                
crash: invalid structure size: tnt
       FILE: kernel.c  LINE: 10458  FUNCTION: show_kernel_taints()

[/usr/bin/crash] error trace: 4ea605 => 4dbc45 => 4d801c => 536752

  536752: SIZE_verify+146
  4d801c: (undetermined)
  4dbc45: (undetermined)
  4ea605: display_sys_stats+2213



Version-Release number of selected component (if applicable):

[root@localhost 127.0.0.1-2017-02-13-15:31:16]# crash -version

crash 7.1.7-1.fc25



How reproducible:

Here is my setup information. I am running this Fedora on qemu vm with x86_64 arch

Please system information. I using latest btrfs tree 
[root@localhost linux]# git remote -v
origin	https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git (fetch)
origin	https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git (push)

[root@localhost 127.0.0.1-2017-02-13-15:31:16]# cat /etc/redhat-release 
Fedora release 25 (Twenty Five)
[root@localhost 127.0.0.1-2017-02-13-15:31:16]# uname -a
Linux localhost.localdomain 4.10.0-rc4+ #3 SMP Mon Feb 13 15:12:36 IST 2017 x86_64 x86_64 x86_64 GNU/Linux


I had follow the steps into this webinar http://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=1124714&sessionid=1&username=&partnerref=null&format=fhvideo1&mobile=false&flashsupportedmobiledevice=false&helpcenter=false&key=F89730775AFDAE0516F5A3B733D4884F&text_language_id=en&playerwidth=1000&playerheight=650&overwritelobby=y&eventuserid=162505198&contenttype=A&mediametricsessionid=129592780&mediametricid=1653962&usercd=162505198&mode=launch

Actual results:
It just crashed with above message.


Expected results:
It show open shell for debugging and show something useful information.

Additional info:

Comment 1 Dave Anderson 2017-02-13 13:53:04 UTC
Fixed in crash-utility upstream:

https://github.com/crash-utility/crash/commit/24a696228c56fd4354d29abe05b206373e0c8bfb

  commit 24a696228c56fd4354d29abe05b206373e0c8bfb
  Author: Dave Anderson <anderson>
  Date:   Thu Jan 5 14:55:18 2017 -0500

    Fix for Linux 4.10 commit 7fd8329ba502ef76dd91db561c7aed696b2c7720
    "taint/module: Clean up global and module taint flags handling".
    Without the patch, when running against Linux 4.10-rc1 and later
    kernels, the crash utility fails during session initialization with
    the message "crash: invalid structure size: tnt".
    (panand)

I plan to do a package rebase when the next upstream release is done,
but in the interim, you can build the package from the github sources:

 $ git clone git://github.com/crash-utility/crash.git
 $ cd crash
 $ make lzo snappy
 $ make install

Comment 2 Jitendra Khasdev 2017-02-13 15:12:21 UTC
Okay Dave, 

I tried building crash, it also got failed with following logs

ase.o inline-frame.o gnu-v2-abi.o gnu-v3-abi.o cp-abi.o cp-support.o cp-namespace.o reggroups.o regset.o trad-frame.o tramp-frame.o solib.o solib-target.o prologue-value.o memory-map.o memrange.o xml-support.o xml-syscall.o xml-utils.o target-descriptions.o target-memory.o xml-tdesc.o xml-builtin.o inferior.o osdata.o gdb_usleep.o record.o record-full.o gcore.o gdb_vecs.o jit.o progspace.o skip.o probe.o common-utils.o buffer.o ptid.o gdb-dlfcn.o common-agent.o format.o registry.o btrace.o record-btrace.o inflow.o    init.o \
	   ../readline/libreadline.a ../opcodes/libopcodes.a ../bfd/libbfd.a  ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a    -ldl -lncurses -lz -lm   ../libiberty/libiberty.a  build-gnulib/import/libgnu.a -ldl -Wl,--dynamic-list=./proc-service.list -llzo2 -lz -ldl -rdynamic
c-exp.o: In function `main':
/root/btrfs/crash/gdb-7.6/gdb/c-exp.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/btrfs/crash/main.c:81: first defined here
cp-name-parser.o: In function `main':
/root/btrfs/crash/gdb-7.6/gdb/cp-name-parser.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/btrfs/crash/main.c:1948: first defined here
ada-exp.o: In function `main':
/root/btrfs/crash/gdb-7.6/gdb/ada-exp.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/btrfs/crash/main.c:1948: first defined here
f-exp.o: In function `main':
/root/btrfs/crash/gdb-7.6/gdb/f-exp.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/btrfs/crash/main.c:1948: first defined here
p-exp.o: In function `main':
/root/btrfs/crash/gdb-7.6/gdb/p-exp.c:1: multiple definition of `main'
../../crashlib.a(main.o):/root/btrfs/crash/main.c:1948: first defined here
go-exp.o: In function `parse_string_or_char':
/root/btrfs/crash/gdb-7.6/gdb/go-exp.y:943: undefined reference to `c_parse_escape'
macroexp.o: In function `get_string_literal':
/root/btrfs/crash/gdb-7.6/gdb/macroexp.c:418: undefined reference to `c_parse_escape'
macroexp.o: In function `get_character_constant':
/root/btrfs/crash/gdb-7.6/gdb/macroexp.c:364: undefined reference to `c_parse_escape'
ada-lang.o: In function `parse':
/root/btrfs/crash/gdb-7.6/gdb/ada-lang.c:12703: undefined reference to `ada_parse'
ada-lang.o:(.rodata+0x1b90): undefined reference to `ada_error'
c-lang.o:(.rodata+0x28): undefined reference to `c_parse'
c-lang.o:(.rodata+0x30): undefined reference to `c_error'
c-lang.o:(.rodata+0x148): undefined reference to `c_parse'
c-lang.o:(.rodata+0x150): undefined reference to `c_error'
c-lang.o:(.rodata+0x268): undefined reference to `c_parse'
c-lang.o:(.rodata+0x270): undefined reference to `c_error'
c-lang.o:(.rodata+0x388): undefined reference to `c_parse'
c-lang.o:(.rodata+0x390): undefined reference to `c_error'
d-lang.o:(.rodata+0x1e8): undefined reference to `c_parse'
d-lang.o:(.rodata+0x1f0): undefined reference to `c_error'
f-lang.o:(.rodata+0x28): undefined reference to `f_parse'
f-lang.o:(.rodata+0x30): undefined reference to `f_error'
objc-lang.o:(.rodata+0xe8): undefined reference to `c_parse'
objc-lang.o:(.rodata+0xf0): undefined reference to `c_error'
opencl-lang.o:(.rodata+0x308): undefined reference to `c_parse'
opencl-lang.o:(.rodata+0x310): undefined reference to `c_error'
p-lang.o:(.rodata+0x28): undefined reference to `pascal_parse'
p-lang.o:(.rodata+0x30): undefined reference to `pascal_error'
cp-support.o: In function `mangled_name_to_comp':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:644: undefined reference to `cp_demangled_name_to_comp'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:629: undefined reference to `cp_new_demangle_parse_info'
cp-support.o: In function `cp_canonicalize_string_full':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:534: undefined reference to `cp_demangled_name_to_comp'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:541: undefined reference to `cp_comp_to_string'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:545: undefined reference to `cp_demangled_name_parse_free'
cp-support.o: In function `inspect_type':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:260: undefined reference to `cp_demangled_name_to_comp'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:264: undefined reference to `cp_merge_demangle_parse_infos'
cp-support.o: In function `replace_typedefs':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:441: undefined reference to `cp_comp_to_string'
cp-support.o: In function `replace_typedefs_qualified_name':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:368: undefined reference to `cp_comp_to_string'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:343: undefined reference to `cp_comp_to_string'
cp-support.o: In function `cp_canonicalize_string':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:582: undefined reference to `cp_demangled_name_to_comp'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:587: undefined reference to `cp_comp_to_string'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:588: undefined reference to `cp_demangled_name_parse_free'
cp-support.o: In function `cp_class_name_from_physname':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:743: undefined reference to `cp_demangled_name_parse_free'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:738: undefined reference to `cp_comp_to_string'
cp-support.o: In function `method_name_from_physname':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:826: undefined reference to `cp_comp_to_string'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:830: undefined reference to `cp_demangled_name_parse_free'
cp-support.o: In function `cp_func_name':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:847: undefined reference to `cp_demangled_name_to_comp'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:855: undefined reference to `cp_comp_to_string'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:857: undefined reference to `cp_demangled_name_parse_free'
cp-support.o: In function `cp_remove_params':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:876: undefined reference to `cp_demangled_name_to_comp'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:903: undefined reference to `cp_demangled_name_parse_free'
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:901: undefined reference to `cp_comp_to_string'
cp-support.o: In function `do_demangled_name_parse_free_cleanup':
/root/btrfs/crash/gdb-7.6/gdb/cp-support.c:108: undefined reference to `cp_demangled_name_parse_free'
collect2: error: ld returned 1 exit status
Makefile:1186: recipe for target 'gdb' failed
make[4]: *** [gdb] Error 1
Makefile:8264: recipe for target 'all-gdb' failed
make[3]: *** [all-gdb] Error 2
Makefile:834: recipe for target 'all' failed
make[2]: *** [all] Error 2

crash build failed

Makefile:229: recipe for target 'gdb_merge' failed
make[1]: *** [gdb_merge] Error 1
Makefile:319: recipe for target 'lzo' failed
make: *** [lzo] Error 2

Comment 3 Dave Anderson 2017-02-13 15:50:27 UTC
What happens when you try this:

$ wget https://kojipkgs.fedoraproject.org//packages/crash/7.1.7/2.fc26/src/crash-7.1.7-2.fc26.src.rpm
$ rpm -ivh crash-7.1.7-2.fc26.src.rpm
$ cd $HOME/rpmbuild/SPECS
$ rpmbuild -a crash.spec

Comment 4 Dave Anderson 2017-02-13 15:51:26 UTC
Make that last command

$ rpmbuild -ba crash.spec

Comment 5 Dave Anderson 2017-02-13 16:01:36 UTC
I'm thinking that it's this patch that went into crash-7.0.2, but which
only modified the crash.spec file that is used when building with
rpmbuild.  Building from the tar.gz file directly would show this:

    - Added "bison" to the BuildRequires line of the crash.spec file.
      Without the patch, the build of the embedded gdb-7.6 module will fail
      unless either /usr/bin/bison or /usr/bin/yacc are available.  The
      failure will result in a stream of error messages from different
      files that indicate:

        multiple definition of 'main'
        undefined reference to 'c_parse_escape'
        undefined reference to 'ada_parse'
        undefined reference to 'ada_error'
        undefined reference to 'c_parse'
        undefined reference to 'c_error'
        undefined reference to 'cp_demangled_name_to_comp'
        undefined reference to 'cp_demangled_name_parse_free'
        undefined reference to 'cp_comp_to_string'
        undefined reference to 'cp_new_demangle_parse_info'

      and the build fails like so:

        collect2: ld returned 1 exit status
        make[4]: *** [gdb] Error 1
        crash build failed

      If building with rpmbuild, the new BuildRequires "bison" entry will
      prevent the build from initiating unless the bison package has been
      installed.  If building with the tar.gz file, the build attempt will
      proceed and fail unless either the bison or byacc (Berkeley Yacc)
      package is installed.
      (anderson)

Comment 6 Jitendra Khasdev 2017-02-13 16:27:45 UTC
I need to install following packages
yum  install bison readline-devel snappy-devel

then crash build success and got the binary into BUILD dir.

but the problem still remain same with same error as,

[root@localhost crash-7.1.7]# ./crash ~/btrfs/build/vmlinux /var/crash/127.0.0.1-2017-02-13-15\:31\:16/vmcore

crash 7.1.7-2.fc25
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

                                                
crash: invalid structure size: tnt
       FILE: kernel.c  LINE: 10458  FUNCTION: show_kernel_taints()

[./crash] error trace: 4ea605 => 4dbc45 => 4d801c => 536752

  536752: SIZE_verify+146
  4d801c: show_kernel_taints+380
  4dbc45: is_livepatch+37
  4ea605: display_sys_stats+2213

Comment 7 Dave Anderson 2017-02-13 16:39:39 UTC
You can build the upstream tar.gz file now since you have installed the
required packages.

Comment 8 Jitendra Khasdev 2017-02-13 17:47:57 UTC
I have download tar from https://people.redhat.com/anderson/ .

But still seems like same error although build is successful as followed steps during build.


[root@localhost 127.0.0.1-2017-02-13-23:12:43]# crash ~/btrfs/build/vmlinux vmcore

crash 7.1.7
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

                                                
crash: invalid structure size: tnt
       FILE: kernel.c  LINE: 10458  FUNCTION: show_kernel_taints()

[/usr/bin/crash] error trace: 4d61a1 => 4d2c4c => 52958a => 529509

  529509: SIZE_verify.part.31+73
  52958a: SIZE_verify+58
  4d2c4c: show_kernel_taints+300
  4d61a1: is_livepatch+17

Comment 9 Dave Anderson 2017-02-13 18:19:05 UTC
In order to run against a 4.10-rc (rawhide) kernel, you need to build
from the current github sources, so try this again:
 
 $ git clone git://github.com/crash-utility/crash.git
 $ cd crash
 $ make lzo snappy
 $ make install

Comment 10 Dave Anderson 2017-02-23 21:44:41 UTC
Information for build crash-7.1.8-1.fc26
https://koji.fedoraproject.org/koji/buildinfo?buildID=861562