1786497 – [RHEL-8.2.0 | x86_64] crash-ptdump-command test failed : ptdump: invalid size request: 0 type: "read page for write"

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1786497 - [RHEL-8.2.0 | x86_64] crash-ptdump-command test failed : ptdump: invalid size request: 0 type: "read page for write"

Summary: [RHEL-8.2.0 | x86_64] crash-ptdump-command test failed : ptdump: invalid size...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	crash-ptdump-command
Sub Component:
Version:	8.2
Hardware:	x86_64
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	8.0
Assignee:	Dave Anderson
QA Contact:	xiaoying yan
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1732733 1776957 1791525
TreeView+	depends on / blocked

Reported:	2019-12-26 02:49 UTC by xiaoying yan
Modified:	2020-12-20 06:33 UTC (History)
CC List:	9 users (show)
Fixed In Version:	crash-ptdump-command-1.0.3-5.el8
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Clones:	1791525 (view as bug list)
Environment:
Last Closed:	2020-04-28 16:04:55 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:
Flags:	pm-rhel: mirror+

Attachments	(Terms of Use)
patch (3.08 KB, application/mbox) 2020-01-28 01:42 UTC, fj-lsoft-rh-dump	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	5188161	0	None	None	None	2020-06-28 14:56:23 UTC
Red Hat Product Errata	RHBA-2020:1749	0	None	None	None	2020-04-28 16:04:59 UTC

Description xiaoying yan 2019-12-26 02:49:46 UTC

Description of problem:

Tested ptdump on RHEL-8.2.0-20191219.0 (beta 1.0),got the error info ' ptdump: invalid size request: 0  type: "read page for write" '  when analysed the vmcore file .


Version-Release number of selected component (if applicable):
RHEL-8.2.0-20191219.0 (beta 1.0)
kernel-4.18.0-167.el8  
kexec-tools-2.0.20-9.el8_1.1   
crash-7.2.7-2.el8

RHEL-8.1.0 GA (kernel-4.18.0-147.el8) also can reproduce this issue.


How reproducible:

repeatedly tested ptdump 10 times,maybe reproduce this issue.

below machine can reproduced this issue easily (1 out of 10 crash)  :
fujitsu-3800e.rhts.eng.pek2.redhat.com
dell-per640-03.dell2.lab.eng.bos.redhat.com
hpe-dl120gen10-01.hpe2.lab.eng.bos.redhat.com


Steps to Reproduce:

1. # cat  /etc/kdump.conf  | grep  -v  '#'
path /var/crash
core_collector makedumpfile -l --message-level 1 -d 31
2. #  rpm -q perf || yum install -y perf
3. #  perf record -vv -a -T -e intel_pt// -S -o /dev/null &

4. Wait a few seconds ，trigger a crash,then , got the vmcore file: 
#  echo c > /proc/sysrq-c   

you can got  my the vmcore file "vmcore-ptdump" :   ibm-x3250m4-03.rhts.eng.pek2.redhat.com/vmcore/yiyan/rhel-8.2.0-beta1.0-ptdump/vmcore-ptdump

5. analyse this vmcore file by crash ptdump command:

# crash   /usr/lib/debug/lib/modules/4.18.0-167.el8.x86_64/vmlinux   vmcore-ptdump


Actual results:

beaker job:https://beaker.engineering.redhat.com/recipes/7734186#task104157254
console log:   https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2019/12/39810/3981012/7734186/104157254/crash.vmcore.log

# crash   /usr/lib/debug/lib/modules/4.18.0-167.el8.x86_64/vmlinux   vmcore-ptdump

crash 7.2.7-2.el8

Copyright (C) 2002-2019  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: stdin: not a tty

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [214MB]: patching 94877 gdb minimal_symbol values

please wait... (gathering kmem slab cache data)
                                                

please wait... (gathering module symbol data)
                                              

please wait... (gathering task table data)
                                           

please wait... (determining panic task)
                                        
      KERNEL: /usr/lib/debug/lib/modules/4.18.0-167.el8.x86_64/vmlinux
    DUMPFILE: /var/crash/127.0.0.1-2019-12-22-20:49:38/vmcore  [PARTIAL DUMP]
        CPUS: 64
        DATE: Sun Dec 22 20:49:19 2019
      UPTIME: 00:12:04
LOAD AVERAGE: 0.18, 0.37, 0.28
       TASKS: 738
    NODENAME: fujitsu-3800e.rhts.eng.pek2.redhat.com
     RELEASE: 4.18.0-167.el8.x86_64
     VERSION: #1 SMP Sun Dec 15 01:24:23 UTC 2019
     MACHINE: x86_64  (2000 Mhz)
      MEMORY: 255.9 GB
       PANIC: "sysrq: SysRq : Trigger a crash"
         PID: 44221
     COMMAND: "runtest.sh"
        TASK: ffff9efe2f5d5c40  [THREAD_INFO: ffff9efe2f5d5c40]
         CPU: 29
       STATE: TASK_RUNNING (SYSRQ)
       
crash> extend ptdump.so
/usr/lib64/crash/extensions/ptdump.so: shared object loaded
crash> ptdump /root/ptdump
ptdump: invalid size request: 0  type: "read page for write"


Expected results:

Dump log buffer of Intel(R) processor trace  successfully


Additional info:

Comment 1 Dave Anderson 2019-12-26 14:05:40 UTC

This will require Fujitsu to look into this bug.

By the way, Which version of crash-ptdump-command are you using?

Comment 2 Emma Wu 2019-12-26 14:13:37 UTC

Hi, it's crash-ptdump-command-1.0.3-4.el8

Comment 3 Dave Anderson 2019-12-26 14:14:32 UTC

Fujitsu,

Can you please investigate this issue?

Thanks,
  Dave

Comment 4 Dave Anderson 2020-01-09 17:15:19 UTC

Hello Fujitsu,

Using the supplied vmcore-ptdump file, here is the full command output:

  crash> ptdump /root/ptdump
  [0] buffer dump: dump.0
  [0] packet decode: decode.0
  [1] buffer dump: dump.1
  [1] packet decode: decode.1
  [2] buffer dump: dump.2
  [2] packet decode: decode.2
  ptdump: invalid size request: 0  type: "read page for write"
  crash> 
  
The fatal error is coming the readmem() call here in the write_buffer_wrapped() function:

        ...

        idx = start_idx;
        page = pt_info_ptr->buffer_ptr[idx];
        offset = pt_info_ptr->output_off & mask;
        len = offset;

        readmem(page, KVADDR, buf, len, "read page for write",
                FAULT_ON_ERROR);

        ...

So I added this debug patch to display some information that might 
show why a readmem() of size 0 was being attempted:

  --- ptdump-1.0.3/ptdump.c.orig
  +++ ptdump-1.0.3/ptdump.c
  @@ -351,6 +351,9 @@ int write_buffer_wrapped(int cpu, FILE *
   	offset = pt_info_ptr->output_off & mask;
   	len = offset;
   
  +error(INFO, "write_buffer_wrapped: start_idx: %d len: %d output_off: %lx (%lx) page: %lx\n", 
  +	start_idx, len, pt_info_ptr->output_off, offset, page);
  +
   	readmem(page, KVADDR, buf, len, "read page for write",
   		FAULT_ON_ERROR);
   
The debug info output seems normal leading up to the fatal readmem() attempt, other than 
the pt_info_ptr->output_off happens to be on a page boundary:
  
  crash> ptdump /root/ptdump
  ptdump: write_buffer_wrapped: start_idx: 86 len: 512 output_off: 56200 (200) page: ffff937f59456000
  [0] buffer dump: dump.0
  [0] packet decode: decode.0
  ptdump: write_buffer_wrapped: start_idx: 823 len: 2944 output_off: 337b80 (b80) page: ffff937ebaf37000
  [1] buffer dump: dump.1
  [1] packet decode: decode.1
  [2] buffer dump: dump.2
  [2] packet decode: decode.2
  ptdump: write_buffer_wrapped: start_idx: 698 len: 0 output_off: 2ba000 (0) page: ffff937ebaaba000
  ptdump: invalid size request: 0  type: "read page for write"
  crash> 

So then I simply avoided the readmem() attempt in that case, with this
addition to the debug code:
  
  --- ptdump-1.0.3/ptdump.c.orig
  +++ ptdump-1.0.3/ptdump.c
  @@ -351,6 +351,11 @@ int write_buffer_wrapped(int cpu, FILE *
   	offset = pt_info_ptr->output_off & mask;
   	len = offset;
   
  +error(INFO, "write_buffer_wrapped: start_idx: %d len: %d output_off: %lx (%lx) page: %lx\n", 
  +	start_idx, len, pt_info_ptr->output_off, offset, page);
  +	if (!len)
  +		goto done;
  +
   	readmem(page, KVADDR, buf, len, "read page for write",
   		FAULT_ON_ERROR);
   
  @@ -364,6 +369,7 @@ int write_buffer_wrapped(int cpu, FILE *
   		return FALSE;
   	}
   
  +done:
   	free(buf);
   	return TRUE;
   }
  
And then it seems to work as expected:
  
  crash> ptdump /root/ptdump
  ptdump: write_buffer_wrapped: start_idx: 86 len: 512 output_off: 56200 (200) page: ffff937f59456000
  [0] buffer dump: dump.0
  [0] packet decode: decode.0
  ptdump: write_buffer_wrapped: start_idx: 823 len: 2944 output_off: 337b80 (b80) page: ffff937ebaf37000
  [1] buffer dump: dump.1
  [1] packet decode: decode.1
  [2] buffer dump: dump.2
  [2] packet decode: decode.2
  ptdump: write_buffer_wrapped: start_idx: 698 len: 0 output_off: 2ba000 (0) page: ffff937ebaaba000
  [3] buffer dump: dump.3
  [3] packet decode: decode.3
  [4] buffer dump: dump.4
  [4] packet decode: decode.4
  [5] buffer dump: dump.5
  [5] packet decode: decode.5
  [6] buffer dump: dump.6
  [6] packet decode: decode.6
  [7] buffer dump: dump.7
  [7] packet decode: decode.7
  [8] buffer dump: dump.8
  [8] packet decode: decode.8
  [9] buffer dump: dump.9
  [9] packet decode: decode.9
  ptdump: write_buffer_wrapped: start_idx: 264 len: 304 output_off: 108130 (130) page: ffff937f4c108000
  [10] buffer dump: dump.10
  [10] packet decode: decode.10
  [11] buffer dump: dump.11
  [11] packet decode: decode.11
  [12] buffer dump: dump.12
  [12] packet decode: decode.12
  [13] buffer dump: dump.13
  [13] packet decode: decode.13
  [14] buffer dump: dump.14
  [14] packet decode: decode.14
  [15] buffer dump: dump.15
  [15] packet decode: decode.15
  ptdump: write_buffer_wrapped: start_idx: 426 len: 3872 output_off: 1aaf20 (f20) page: ffff937f4d9aa000
  [16] buffer dump: dump.16
  [16] packet decode: decode.16
  [17] buffer dump: dump.17
  [17] packet decode: decode.17
  ptdump: write_buffer_wrapped: start_idx: 598 len: 2192 output_off: 256890 (890) page: ffff937f4e656000
  [18] buffer dump: dump.18
  [18] packet decode: decode.18
  [19] buffer dump: dump.19
  [19] packet decode: decode.19
  [20] buffer dump: dump.20
  [20] packet decode: decode.20
  [21] buffer dump: dump.21
  [21] packet decode: decode.21
  ptdump: write_buffer_wrapped: start_idx: 57 len: 624 output_off: 39270 (270) page: ffff937f4f439000
  [22] buffer dump: dump.22
  [22] packet decode: decode.22
  ptdump: write_buffer_wrapped: start_idx: 723 len: 2736 output_off: 2d3ab0 (ab0) page: ffff937f4fad3000
  [23] buffer dump: dump.23
  [23] packet decode: decode.23
  ptdump: write_buffer_wrapped: start_idx: 117 len: 2000 output_off: 757d0 (7d0) page: ffff937f4fc75000
  [24] buffer dump: dump.24
  [24] packet decode: decode.24
  [25] buffer dump: dump.25
  [25] packet decode: decode.25
  [26] buffer dump: dump.26
  [26] packet decode: decode.26
  [27] buffer dump: dump.27
  [27] packet decode: decode.27
  [28] buffer dump: dump.28
  [28] packet decode: decode.28
  ptdump: write_buffer_wrapped: start_idx: 670 len: 1072 output_off: 29e430 (430) page: ffff937f5169e000
  [29] buffer dump: dump.29
  [29] packet decode: decode.29
  ptdump: write_buffer_wrapped: start_idx: 42 len: 576 output_off: 2a240 (240) page: ffff937f5182a000
  [30] buffer dump: dump.30
  [30] packet decode: decode.30
  [31] buffer dump: dump.31
  [31] packet decode: decode.31
  [32] buffer dump: dump.32
  [32] packet decode: decode.32
  ptdump: write_buffer_wrapped: start_idx: 659 len: 1104 output_off: 293450 (450) page: ffff937f52a93000
  [33] buffer dump: dump.33
  [33] packet decode: decode.33
  [34] buffer dump: dump.34
  [34] packet decode: decode.34
  ptdump: write_buffer_wrapped: start_idx: 516 len: 4048 output_off: 204fd0 (fd0) page: ffff937f53204000
  [35] buffer dump: dump.35
  [35] packet decode: decode.35
  [36] buffer dump: dump.36
  [36] packet decode: decode.36
  [37] buffer dump: dump.37
  [37] packet decode: decode.37
  [38] buffer dump: dump.38
  [38] packet decode: decode.38
  ptdump: write_buffer_wrapped: start_idx: 699 len: 3776 output_off: 2bbec0 (ec0) page: ffff937f542bb000
  [39] buffer dump: dump.39
  [39] packet decode: decode.39
  ptdump: write_buffer_wrapped: start_idx: 44 len: 960 output_off: 2c3c0 (3c0) page: ffff937f5482c000
  [40] buffer dump: dump.40
  [40] packet decode: decode.40
  ptdump: write_buffer_wrapped: start_idx: 740 len: 2208 output_off: 2e48a0 (8a0) page: ffff937f54ee4000
  [41] buffer dump: dump.41
  [41] packet decode: decode.41
  ptdump: write_buffer_wrapped: start_idx: 512 len: 2544 output_off: 2009f0 (9f0) page: ffff937f55200000
  [42] buffer dump: dump.42
  [42] packet decode: decode.42
  ptdump: write_buffer_wrapped: start_idx: 60 len: 1856 output_off: 3c740 (740) page: ffff937f5543c000
  [43] buffer dump: dump.43
  [43] packet decode: decode.43
  crash> 

That's probably not the correct way to fix the bug, but is it legitimate
for the pt_info_ptr->output_off to be on a page boundary?

Comment 5 Dave Anderson 2020-01-10 15:03:53 UTC

Adding Takao to the NEEDINFO list.

Takao is the author/maintainer of crash-ptdump-command package:

  http://people.redhat.com/anderson/extensions.html#PTDUMP

Comment 6 fj-lsoft-rh-dump 2020-01-14 05:13:11 UTC

Dave,

> Adding Takao to the NEEDINFO list.
> 
> Takao is the author/maintainer of crash-ptdump-command package:
> 
>   http://people.redhat.com/anderson/extensions.html#PTDUMP

Sorry for the delayed response.

I have started investigating the issue. Please wait for a while.

By the way, Indoh moved to another team some years ago and he now doesn't maintain ptdump module.
Could you replace the author information at extensions.html by the following signature?

    HATAYAMA Daisuke <d.hatayama>

Thanks.
HATAYAMA, Daisuke

Comment 7 Dave Anderson 2020-01-14 15:03:45 UTC

(In reply to fj-lsoft-rh-dump from comment #6)
> Dave,
> 
> > Adding Takao to the NEEDINFO list.
> > 
> > Takao is the author/maintainer of crash-ptdump-command package:
> > 
> >   http://people.redhat.com/anderson/extensions.html#PTDUMP
> 
> Sorry for the delayed response.
> 
> I have started investigating the issue. Please wait for a while.
> 
> By the way, Indoh moved to another team some years ago and he now doesn't maintain ptdump module.
> Could you replace the author information at extensions.html by the following signature?
> 
>     HATAYAMA Daisuke <d.hatayama>

Ok thanks -- done!

Dave

Comment 8 fj-lsoft-rh-dump 2020-01-16 02:44:18 UTC

Dave,

> Ok thanks -- done!

Thanks for the update. I confirmed that.
I'll post the next version of ptdump that includes the fix for this issue in crash devel ML after testing.

Oguma-san,

Could you monitor this bugzilla ticket?

Could you also clone this bugzilla ticket for RHEL7.8?
This issue occurs in the RHEL7 series, too.

Thanks.
HATAYAMA, Daisuke

Comment 9 Yoko Oguma 2020-01-16 03:08:15 UTC

(In reply to fj-lsoft-rh-dump from comment #8)
Hatayama-san,

> Could you monitor this bugzilla ticket?

OK. I will monitor this ticket.

> Could you also clone this bugzilla ticket for RHEL7.8?
> This issue occurs in the RHEL7 series, too.

For RHEL7 - https://bugzilla.redhat.com/show_bug.cgi?id=1791525

Best Regards,
Yoko Oguma

Comment 10 Dave Anderson 2020-01-27 20:02:12 UTC

A patch to address this issue has been posted upstream:

  [Crash-utility] Extensions: ptdump update v1.0.7
  https://www.redhat.com/archives/crash-utility/2020-January/msg00007.html

  Hi Dave,

  The attached file is updated extension module ptdump v1.0.7.
  Please replace old one in extension site [1] with this tarball.

  Changelog:
  - ptdump: fix build warning: warning: this ‘if’ clause does not guard
  - ptdump: fix failure: ptdump: invalid size request: 0 type: "read page for write" 
  - ptdump: fix heap memory and fd leak when fault happens

  md5sum:
  b548afa3c44b6e7f81bce020297a1572

  [1] http://people.redhat.com/anderson/extensions.html#PTDUMP

  --
  Thanks.
  HATAYAMA, Daisuke

Comment 11 fj-lsoft-rh-dump 2020-01-28 01:42:18 UTC

Created attachment 1655845 [details]
patch

Dave,

> A patch to address this issue has been posted upstream:
> 
>   [Crash-utility] Extensions: ptdump update v1.0.7
>   https://www.redhat.com/archives/crash-utility/2020-January/msg00007.html

I attached the patch, 0001-ptdump-fix-failure-ptdump-invalid-size-request-0-typ.patch, to fix the issue tracked here.
Could you include it in the crash-ptdump-command package?

Thanks.
HATAYAMA, Daisuke

Comment 12 Dave Anderson 2020-01-28 14:10:05 UTC

I've been keeping the RHEL package in sync with the upstream version,
so I've already created the patch.  However, I will add your explanation
to the patch file.

Comment 19 errata-xmlrpc 2020-04-28 16:04:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1749

Note You need to log in before you can comment on or make changes to this bug.