RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1483934 - [Pegas] crash: crash cmd reports "WARNING: cannot access vmalloc'd module memory"
Summary: [Pegas] crash: crash cmd reports "WARNING: cannot access vmalloc'd module mem...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: crash
Version: 7.4-Alt
Hardware: ppc64le
OS: Linux
unspecified
high
Target Milestone: rc
: 7.4-Alt
Assignee: Dave Anderson
QA Contact: Yuming Liu
URL:
Whiteboard:
Depends On:
Blocks: 1381083 1440030 1485391 1516606
TreeView+ depends on / blocked
 
Reported: 2017-08-22 10:07 UTC by Emma Wu
Modified: 2019-02-19 21:41 UTC (History)
14 users (show)

Fixed In Version: crash-7.1.9-2.p1.el7a
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1485391 (view as bug list)
Environment:
Last Closed: 2017-11-09 11:33:54 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
crash analysis log on kernel-4.11.0-28.el7a.ppc64le.rpm with crash-7.1.9-2.p1.el7a.ppc64le.rpm (8.61 KB, application/octet-stream)
2017-09-06 19:30 UTC, IBM Bug Proxy 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 157924 0 None None None 2019-07-22 21:24:06 UTC
Red Hat Product Errata RHEA-2017:3178 0 normal SHIPPED_LIVE crash enhancement update 2017-11-09 16:13:43 UTC

Description Emma Wu 2017-08-22 10:07:15 UTC 
Description of problem:

The error is found since kernel-4.11.0-24.el7a.ppc64le.
No warning/error message is reported in kernel-4.11.0-23.el7a.ppc64le.

It also report errors like "read error: kernel virtual address: d00000000347ba70  type: "XXX"" when running some crash commands 

The error is seen on both a live system and a vmcore. 


Version-Release number of selected component (if applicable):

distro: RHEL-ALT-7.4-20170808.0
kernel-4.11.0-24.el7a.ppc64le
crash-7.1.9-2.el7.ppc64le
crash-trace-command-2.0-12.el7.ppc64le


How reproducible:
Always

Steps to Reproduce:
1.Install a machine with RHEL-ALT-7.4-20170808.0 ppc64le
2.Upgrade kernel to kernel-4.11.0-24.el7a.ppc64le
3.Install crash, kernel-debuginfo, crash-trace-command
4.Run crash on live system. 


Actual results:
# crash

crash 7.1.9-2.el7
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu"...

WARNING: cannot access vmalloc'd module memory

      KERNEL: /usr/lib/debug/lib/modules/4.11.0-24.el7a.ppc64le/vmlinux
    DUMPFILE: /dev/crash
        CPUS: 2
        DATE: Tue Aug 22 05:45:34 2017
      UPTIME: 00:07:00
LOAD AVERAGE: 0.15, 0.03, 0.01
       TASKS: 139
    NODENAME: ibm-p8-kvm-06-guest-01.rhts.eng.bos.redhat.com
     RELEASE: 4.11.0-24.el7a.ppc64le
     VERSION: #1 SMP Wed Aug 16 12:00:32 EDT 2017
     MACHINE: ppc64le  (3425 Mhz)
      MEMORY: 19 GB
         PID: 3864
     COMMAND: "crash"
        TASK: c0000000030b3400  [THREAD_INFO: c0000004afd34000]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)

crash> mod
mod: cannot access vmalloc'd module memory
crash> mount
     MOUNT           SUPERBLK     TYPE   DEVNAME   DIRNAME
c0000004b83a0000 c0000004be0d1000 rootfs rootfs    /         
c0000004b86d0300 c0000004b87a0800 sysfs  sysfs     /sys      
c0000004b86d0480 c0000004be0d4000 proc   proc      /proc     
c0000004b86d0600 c0000004b87a0000 devtmpfs devtmpfs /dev      
c0000004b86d0780 c0000004be0d5800 securityfs securityfs /sys/kernel/security
c0000004b86d0900 c0000004b87a1000 tmpfs  tmpfs     /dev/shm  
c0000004b86d0a80 c0000004b87a1800 devpts devpts    /dev/pts  
c0000004b86d0c00 c0000004b87a2000 tmpfs  tmpfs     /run      
c0000004b86d0d80 c0000004b87a2800 tmpfs  tmpfs     /sys/fs/cgroup
c0000004b86d0f00 c0000004b87a3000 cgroup cgroup    /sys/fs/cgroup/systemd
c0000004b86d1080 c0000004b87a3800 pstore pstore    /sys/fs/pstore
c0000004b86d1200 c0000004b87a4800 cgroup cgroup    /sys/fs/cgroup/devices
c0000004b86d1380 c0000004b87a4000 cgroup cgroup    /sys/fs/cgroup/cpu,cpuacct
c0000004b86d1500 c0000004b87a5000 cgroup cgroup    /sys/fs/cgroup/net_cls,net_prio
c0000004b86d1680 c0000004b87a5800 cgroup cgroup    /sys/fs/cgroup/memory
c0000004b86d1800 c0000004b87a6000 cgroup cgroup    /sys/fs/cgroup/blkio
c0000004b86d1980 c0000004b87a6800 cgroup cgroup    /sys/fs/cgroup/cpuset
c0000004b86d1b00 c0000004b87a7000 cgroup cgroup    /sys/fs/cgroup/hugetlb
c0000004b86d1c80 c0000004b87a7800 cgroup cgroup    /sys/fs/cgroup/pids
c0000004b86d1e00 c0000004b87a8000 cgroup cgroup    /sys/fs/cgroup/freezer
c0000004b86d1f80 c0000004b87a8800 cgroup cgroup    /sys/fs/cgroup/perf_event
c0000004b83a1680 c0000000034f6000 configfs configfs /sys/kernel/config
c0000004ad7c1c80 c0000000034ff800 mount: read error: kernel virtual address: d0000000046aba70  type: "file_system_type name"
crash> extend trace.so
extend: read error: kernel virtual address: d000000006068858  type: "read ftrace_event_call id"
extend: /usr/lib64/crash/extensions/trace.so: no commands registered: shared object unloaded


Expected results:


Additional info:
Comment 2 Emma Wu 2017-08-22 10:09:42 UTC 
Created attachment 1316571 [details]
log_live.txt

attached log (log_live.txt) of cmd `crash -d7`

Seems the warning is reported after:
....
<readmem: c0000004b9280000, KVADDR, "page table", 65536, (FOE), 100285a7eb0>
<read_memory_device: addr: c0000004b9280000 paddr: 4b9280000 cnt: 65536>
<read_memory_device: addr: d0000000078e0900 paddr: 800004b19a0900 cnt: 896>
crash: read error: kernel virtual address: d0000000078e0900  type: "module struct"
WARNING: cannot access vmalloc'd module memory
...
Comment 4 Dave Anderson 2017-08-22 13:05:48 UTC 
(In reply to Emma Wu from comment #3)
> Feel free to change the component. Just not very sure where I should place
> it..

The crash-trace-command is irrelevant, it is simply trying to read
a vmalloc'd memory address, and the base crash utility's virtual
address translation (page-table-walkthrough) is failing for some
reason.
Comment 5 Dave Anderson 2017-08-22 14:00:52 UTC 
(In reply to Emma Wu from comment #0)
> 
> Version-Release number of selected component (if applicable):
> 
> distro: RHEL-ALT-7.4-20170808.0
> kernel-4.11.0-24.el7a.ppc64le


Kernel version 4.11.0-23.el7a.ppc64le works OK:

  [root@ibm-p8-05-lp6 ~]# crash

  crash 7.1.9-2.el7
  Copyright (C) 2002-2016  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005, 2011  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.6
  Copyright (C) 2013 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "powerpc64le-unknown-linux-gnu"...
  
        KERNEL: /usr/lib/debug/lib/modules/4.11.0-23.el7a.ppc64le/vmlinux
      DUMPFILE: /dev/crash
          CPUS: 16
          DATE: Tue Aug 22 09:48:52 2017
        UPTIME: 00:04:45
  LOAD AVERAGE: 0.17, 0.05, 0.01
         TASKS: 282
      NODENAME: ibm-p8-05-lp6.rhts.eng.bos.redhat.com
       RELEASE: 4.11.0-23.el7a.ppc64le
       VERSION: #1 SMP Fri Aug 11 11:52:59 EDT 2017
       MACHINE: ppc64le  (3425 Mhz)
        MEMORY: 24 GB
           PID: 2169
       COMMAND: "crash"
          TASK: c0000005eaae9400  [THREAD_INFO: c0000005f463c000]
           CPU: 9
         STATE: TASK_RUNNING (ACTIVE)
  
  crash>

So it looks like this kernel patch:

  * Wed Aug 16 2017 Herton R. Krzesinski <herton> [4.11.0-24.el7a]
  ... [ cut ] ...
  - [powerpc] mm: Move hash specific pte bits to be top bits of RPN (Scott Wood) [1475882]
  - [powerpc] mm: Lower the max real address to 53 bits (Scott Wood) [1475882]
  - [powerpc] mm: Define all PTE bits based on radix definitions (Scott Wood) [1475882]
  - [powerpc] mm: Define _PAGE_SOFT_DIRTY unconditionally (Scott Wood) [1475882]
  - [powerpc] mm: Cleanup bits definition between hash and radix (Scott Wood) [1475882]
  ...
Comment 6 Dave Anderson 2017-08-22 14:10:44 UTC 
This is going to require assistance from IBM.  It really needs to be
viewed by Hari Krishna Bathini (hbathini.com), but he does not
have a bugzilla account under that email address.  Hopefully the cc:
additions of bugproxy.com or mahesh.salgaonkar.com
can get this to his attention.
Comment 7 PaulB 2017-08-22 18:51:03 UTC 
pbunyan adds  hkrzesin  so its on his radar.
Comment 8 Mahesh Salgaonkar 2017-08-23 02:50:19 UTC 
(In reply to Dave Anderson from comment #6)
> This is going to require assistance from IBM.  It really needs to be
> viewed by Hari Krishna Bathini (hbathini.com), but he does not
> have a bugzilla account under that email address.  Hopefully the cc:
> additions of bugproxy.com or mahesh.salgaonkar.com
> can get this to his attention.

Can you mirror this bug to IBM bugzilla so that Hari can take a look ?
Comment 9 IBM Bug Proxy 2017-08-23 06:10:21 UTC 
------- Comment From vaish123.com 2017-08-23 02:04 EDT-------
Please reverse mirror Red Hat Bug 1483934 - crash: crash cmd reports "WARNING: cannot access vmalloc'd module memory"
Comment 10 IBM Bug Proxy 2017-08-24 11:10:19 UTC 
------- Comment From hbathini.com 2017-08-24 07:06 EDT-------
Hello Dave,

The below patch should fix the error during vmalloc address translation:

https://www.redhat.com/archives/crash-utility/2017-August/msg00005.html
..which reads "[Crash-utility] [PATCH] book3s/ppc64: Lower the max real address to 53 bits"

Also, there seem be slight changes in how hugepages are handled. Will look into it and
post a separate patch for it but this fix should at least help ppl not using huge pages..

Thanks
Hari
Comment 11 Dave Anderson 2017-08-24 14:35:12 UTC 
Hari,

Thanks for the quick response!  The patch is queued upstream:

https://github.com/crash-utility/crash/commit/d7b43c2a527503415dd66dff20daec01c139c6f1

  PPC64 kernel commit 2f18d533757da3899f4bedab0b2c051b080079dc lowered
  the max real address to 53 bits.  Without this patch, the warning
  message "WARNING: cannot access vmalloc'd module memory" appears
  during initialization, and any command that attempts to read a
  vmalloc'd kernel virtual address will fail and display "read error"
  messages.
  (hbathini.ibm.com)
Comment 12 Dave Anderson 2017-08-24 14:55:06 UTC 
Brendan and/or John,

Despite my best efforts to keep the RHEL7.4 version of the crash utility
working with Pegas kernels, the recent kernel commit referenced in 
comment #5 has crippled the crash utility.  

Can you give me a pointer to the process for creating a Pegas branch
of a package?

Thanks,
  Dave
Comment 18 IBM Bug Proxy 2017-09-06 19:30:23 UTC 
Created attachment 1322785 [details]
crash analysis log on kernel-4.11.0-28.el7a.ppc64le.rpm with crash-7.1.9-2.p1.el7a.ppc64le.rpm


------- Comment on attachment From hbathini.com 2017-09-06 15:23 EDT-------


Able to access vmalloc region successfully on kernel-4.11.0-28.el7a.ppc64le.rpm
with crash-7.1.9-2.p1.el7a.ppc64le.rpm

Thanks
Hari
Comment 21 errata-xmlrpc 2017-11-09 11:33:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3178
Comment 22 IBM Bug Proxy 2017-11-27 10:01:13 UTC 
------- Comment From sthoufee.com 2017-11-27 04:56 EDT-------
*** Bug 161663 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.