Bug 1441452 - crash: invalid structure size: tnt with newer fc25 4.10-based kernels
Summary: crash: invalid structure size: tnt with newer fc25 4.10-based kernels
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: crash
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Dave Anderson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1440587
Blocks: 1448115
TreeView+ depends on / blocked
 
Reported: 2017-04-12 01:07 UTC by Qiao Zhao
Modified: 2017-05-12 12:02 UTC (History)
6 users (show)

Fixed In Version: crash-7.1.9-1.fc25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1440587
: 1448115 (view as bug list)
Environment:
Last Closed: 2017-05-12 12:02:14 UTC
Type: Bug

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Comment 1 Dave Anderson 2017-04-24 20:16:34 UTC 
Rawhide build:

 Information for build crash-7.1.9-1.fc27
 https://koji.fedoraproject.org/koji/buildinfo?buildID=882902
Comment 2 Dave Anderson 2017-04-25 17:44:20 UTC 
fc25 build:

 Information for build crash-7.1.9-1.fc25
 https://koji.fedoraproject.org/koji/buildinfo?buildID=883098
Comment 3 Fedora Update System 2017-04-25 17:46:18 UTC 
crash-7.1.9-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-06f4084b29
Comment 4 Qiao Zhao 2017-04-26 05:52:27 UTC 
(In reply to Dave Anderson from comment #2)
> fc25 build:
> 
>  Information for build crash-7.1.9-1.fc25
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=883098

Hi Dave,

I retest this problem use new package,
# rpm -q crash kexec-tools
crash-7.1.9-1.fc25.x86_64
kexec-tools-2.0.13-7.fc25.2.x86_64
# crash /lib/debug/lib/modules/4.10.8-200.fc25.x86_64/vmlinux /var/crash/127.0.0.1-2017-04-11-17\:14\:12/vmcore

crash 7.1.9-1.fc25
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [768MB]: patching 85669 gdb minimal_symbol values

crash: seek error: kernel virtual address: ffffffffb1e3d328  type: "page_offset_base"
# uname -r
4.10.8-200.fc25.x86_64

Looks another error occur.
Comment 5 Dave Anderson 2017-04-26 12:55:47 UTC 
Can you please make the vmlinux/vmcore pair available?
Comment 6 Dave Anderson 2017-04-26 14:14:09 UTC 
Also, can you please verify that crash works on the live system running
the 4.10.8-200.fc25 kernel?
Comment 7 Dave Anderson 2017-04-26 18:27:19 UTC 
(In reply to Qiao Zhao from comment #4)
> ...
> 
> crash: seek error: kernel virtual address: ffffffffb1e3d328  type: "page_offset_base"
> # uname -r
> 4.10.8-200.fc25.x86_64
>
> Looks another error occur.

(In reply to Dave Anderson from comment #6)
> Also, can you please verify that crash works on the live system running
> the 4.10.8-200.fc25 kernel?
  
I provisioned an fc25 kernel from beaker, which is running 4.10.11-200.fc25:
  
  [root@ibm-hs22-02 127.0.0.1-2017-04-26-20:06:26]# uname -r
  4.10.11-200.fc25.x86_64
  [root@ibm-hs22-02 127.0.0.1-2017-04-26-20:06:26]# 
  
It runs OK on a live system:
  
  [root@ibm-hs22-02 127.0.0.1-2017-04-26-20:06:26]# crash
  
  crash 7.1.9-1.fc25
  Copyright (C) 2002-2016  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005, 2011  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.6
  Copyright (C) 2013 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
  WARNING: kernel relocated [560MB]: patching 85635 gdb minimal_symbol values
  
        KERNEL: /usr/lib/debug/lib/modules/4.10.11-200.fc25.x86_64/vmlinux
      DUMPFILE: /dev/crash
          CPUS: 16
          DATE: Wed Apr 26 20:12:12 2017
        UPTIME: 00:02:41
  LOAD AVERAGE: 0.48, 0.26, 0.11
         TASKS: 306
      NODENAME: ibm-hs22-02.rhts.eng.brq.redhat.com
       RELEASE: 4.10.11-200.fc25.x86_64
       VERSION: #1 SMP Tue Apr 18 17:25:01 UTC 2017
       MACHINE: x86_64  (2400 Mhz)
        MEMORY: 16 GB
           PID: 2347
       COMMAND: "crash"
          TASK: ffff9379359f2440  [THREAD_INFO: ffff9379359f2440]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)
  
  crash> 
  
And it also runs OK on a kdump vmcore:
  
  # crash /usr/lib/debug/lib/modules/4.10.11-200.fc25.x86_64/vmlinux vmcore 
  
  crash 7.1.9-1.fc25
  Copyright (C) 2002-2016  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005, 2011  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.6
  Copyright (C) 2013 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
  WARNING: kernel relocated [784MB]: patching 85635 gdb minimal_symbol values
  
        KERNEL: /usr/lib/debug/lib/modules/4.10.11-200.fc25.x86_64/vmlinux
      DUMPFILE: vmcore  [PARTIAL DUMP]
          CPUS: 16
          DATE: Wed Apr 26 20:06:01 2017
        UPTIME: 03:15:43
  LOAD AVERAGE: 0.03, 0.05, 0.02
         TASKS: 296
      NODENAME: ibm-hs22-02.rhts.eng.brq.redhat.com
       RELEASE: 4.10.11-200.fc25.x86_64
       VERSION: #1 SMP Tue Apr 18 17:25:01 UTC 2017
       MACHINE: x86_64  (2399 Mhz)
        MEMORY: 16 GB
         PANIC: "sysrq: SysRq : Trigger a crash"
           PID: 12455
       COMMAND: "bash"
          TASK: ffff8a28afb44880  [THREAD_INFO: ffff8a28afb44880]
           CPU: 2
         STATE: TASK_RUNNING (SYSRQ)
  
  crash> 
  
Is there a difference in the kdump implementation in the 4.10.8-200.fc25
vs the newer 4.10.11-200.fc25 kernel?
Comment 8 Qiao Zhao 2017-04-27 02:10:09 UTC 
Hi Dave,

Maybe i have some mistake before. When i try to dump a new vmcore and try to use crash to analysis, it is works well now.

# crash /lib/debug/lib/modules/4.10.8-200.fc25.x86_64/vmlinux /var/crash/127.0.0.1-2017-04-27-10\:05\:18/vmcore

crash 7.1.9-1.fc25
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [336MB]: patching 85669 gdb minimal_symbol values

      KERNEL: /lib/debug/lib/modules/4.10.8-200.fc25.x86_64/vmlinux    
    DUMPFILE: /var/crash/127.0.0.1-2017-04-27-10:05:18/vmcore  [PARTIAL DUMP]
        CPUS: 2
        DATE: Thu Apr 27 10:05:15 2017
      UPTIME: 00:09:58
LOAD AVERAGE: 0.19, 0.07, 0.02
       TASKS: 148
    NODENAME: localhost.localdomain
     RELEASE: 4.10.8-200.fc25.x86_64
     VERSION: #1 SMP Fri Mar 31 13:20:22 UTC 2017
     MACHINE: x86_64  (2593 Mhz)
      MEMORY: 2 GB
       PANIC: "sysrq: SysRq : Trigger a crash"
         PID: 1064
     COMMAND: "bash"
        TASK: ffff9cececd24b00  [THREAD_INFO: ffff9cececd24b00]
         CPU: 1
       STATE: TASK_RUNNING (SYSRQ)

crash> 

Thanks for your information. I will vote this package in here https://bodhi.fedoraproject.org/updates/FEDORA-2017-06f4084b29

--
Thanks,
Qiao
Comment 9 Fedora Update System 2017-04-27 06:48:43 UTC 
crash-7.1.9-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-06f4084b29
Comment 10 Dave Young 2017-04-27 07:40:25 UTC 
(In reply to Dave Anderson from comment #7)
> (In reply to Qiao Zhao from comment #4)
> > ...
> > 
> > crash: seek error: kernel virtual address: ffffffffb1e3d328  type: "page_offset_base"
> > # uname -r
> > 4.10.8-200.fc25.x86_64
> >
> > Looks another error occur.
> 
> (In reply to Dave Anderson from comment #6)
> > Also, can you please verify that crash works on the live system running
> > the 4.10.8-200.fc25 kernel?
>   
> I provisioned an fc25 kernel from beaker, which is running 4.10.11-200.fc25:
>   
>   [root@ibm-hs22-02 127.0.0.1-2017-04-26-20:06:26]# uname -r
>   4.10.11-200.fc25.x86_64
>   [root@ibm-hs22-02 127.0.0.1-2017-04-26-20:06:26]# 
>   
> It runs OK on a live system:
>   
>   [root@ibm-hs22-02 127.0.0.1-2017-04-26-20:06:26]# crash
>   
>   crash 7.1.9-1.fc25
>   Copyright (C) 2002-2016  Red Hat, Inc.
>   Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>   Copyright (C) 1999-2006  Hewlett-Packard Co
>   Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>   Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>   Copyright (C) 2005, 2011  NEC Corporation
>   Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>   Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>   This program is free software, covered by the GNU General Public License,
>   and you are welcome to change it and/or distribute copies of it under
>   certain conditions.  Enter "help copying" to see the conditions.
>   This program has absolutely no warranty.  Enter "help warranty" for
> details.
>    
>   GNU gdb (GDB) 7.6
>   Copyright (C) 2013 Free Software Foundation, Inc.
>   License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
>   This is free software: you are free to change and redistribute it.
>   There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>   and "show warranty" for details.
>   This GDB was configured as "x86_64-unknown-linux-gnu"...
>   
>   WARNING: kernel relocated [560MB]: patching 85635 gdb minimal_symbol values
>   
>         KERNEL: /usr/lib/debug/lib/modules/4.10.11-200.fc25.x86_64/vmlinux
>       DUMPFILE: /dev/crash
>           CPUS: 16
>           DATE: Wed Apr 26 20:12:12 2017
>         UPTIME: 00:02:41
>   LOAD AVERAGE: 0.48, 0.26, 0.11
>          TASKS: 306
>       NODENAME: ibm-hs22-02.rhts.eng.brq.redhat.com
>        RELEASE: 4.10.11-200.fc25.x86_64
>        VERSION: #1 SMP Tue Apr 18 17:25:01 UTC 2017
>        MACHINE: x86_64  (2400 Mhz)
>         MEMORY: 16 GB
>            PID: 2347
>        COMMAND: "crash"
>           TASK: ffff9379359f2440  [THREAD_INFO: ffff9379359f2440]
>            CPU: 0
>          STATE: TASK_RUNNING (ACTIVE)
>   
>   crash> 
>   
> And it also runs OK on a kdump vmcore:
>   
>   # crash /usr/lib/debug/lib/modules/4.10.11-200.fc25.x86_64/vmlinux vmcore 
>   
>   crash 7.1.9-1.fc25
>   Copyright (C) 2002-2016  Red Hat, Inc.
>   Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>   Copyright (C) 1999-2006  Hewlett-Packard Co
>   Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>   Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>   Copyright (C) 2005, 2011  NEC Corporation
>   Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>   Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>   This program is free software, covered by the GNU General Public License,
>   and you are welcome to change it and/or distribute copies of it under
>   certain conditions.  Enter "help copying" to see the conditions.
>   This program has absolutely no warranty.  Enter "help warranty" for
> details.
>    
>   GNU gdb (GDB) 7.6
>   Copyright (C) 2013 Free Software Foundation, Inc.
>   License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
>   This is free software: you are free to change and redistribute it.
>   There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>   and "show warranty" for details.
>   This GDB was configured as "x86_64-unknown-linux-gnu"...
>   
>   WARNING: kernel relocated [784MB]: patching 85635 gdb minimal_symbol values
>   
>         KERNEL: /usr/lib/debug/lib/modules/4.10.11-200.fc25.x86_64/vmlinux
>       DUMPFILE: vmcore  [PARTIAL DUMP]
>           CPUS: 16
>           DATE: Wed Apr 26 20:06:01 2017
>         UPTIME: 03:15:43
>   LOAD AVERAGE: 0.03, 0.05, 0.02
>          TASKS: 296
>       NODENAME: ibm-hs22-02.rhts.eng.brq.redhat.com
>        RELEASE: 4.10.11-200.fc25.x86_64
>        VERSION: #1 SMP Tue Apr 18 17:25:01 UTC 2017
>        MACHINE: x86_64  (2399 Mhz)
>         MEMORY: 16 GB
>          PANIC: "sysrq: SysRq : Trigger a crash"
>            PID: 12455
>        COMMAND: "bash"
>           TASK: ffff8a28afb44880  [THREAD_INFO: ffff8a28afb44880]
>            CPU: 2
>          STATE: TASK_RUNNING (SYSRQ)
>   
>   crash> 
>   
> Is there a difference in the kdump implementation in the 4.10.8-200.fc25
> vs the newer 4.10.11-200.fc25 kernel?

Dave, I'm not sure what is the difference, this should be an increase from stable kernel 4.10.8 to 4.10.11, I do not remember there are any kdump patches from Greg's stable patch review mails, but I may miss something.
Comment 11 Dave Anderson 2017-05-04 14:51:57 UTC 
fc25 push to stable failed because:

============================================================
crash-7.1.9-1.fc25 into f25-updates
============================================================
[ OK ] f24
        Latest package:         crash-7.1.4-1.fc24
[ OK ] f24-updates
        Latest package:         crash-7.1.7-1.fc24
[ OK ] f25
        Latest package:         crash-7.1.5-2.fc25
[--->] f25-updates
        Proposed package:       crash-7.1.9-1.fc25
[FAIL] f26 + f26-updates
        Latest package:         crash-7.1.8-1.fc26
        Latest pending package: None
        Error: Condition not satisfied: crash-7.1.9-1.fc25 <= crash-7.1.8-1.fc26
        Error: Proposed package version must be lesser than or equal to the latest (pending) package version.
[ OK ] f27
        Latest package:         crash-7.1.9-1.fc27
        Latest pending package: None
RESULT: FAILED

SUMMARY: 1 FAILED
Comment 12 Fedora Update System 2017-05-12 12:02:14 UTC 
crash-7.1.9-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.