Bug 438056 - makedumpfile -c on an existing vmcore fails on ppc
makedumpfile -c on an existing vmcore fails on ppc
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kexec-tools (Show other bugs)
ppc64 Linux
low Severity low
: rc
: ---
Assigned To: Neil Horman
Depends On:
  Show dependency treegraph
Reported: 2008-03-18 16:22 EDT by Mike Gahagan
Modified: 2009-01-20 16:00 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2009-01-20 16:00:18 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
strace of failing makedumpfile on ppc (313.28 KB, text/plain)
2008-03-18 16:22 EDT, Mike Gahagan
no flags Details
patch to make read continue until error or complete read (8.07 KB, patch)
2008-03-19 14:04 EDT, Neil Horman
no flags Details | Diff
strace from failed makedumpfile on ppc with the patch. (323.46 KB, text/plain)
2008-03-20 10:17 EDT, Mike Gahagan
no flags Details
new makedumpfile patch (8.08 KB, patch)
2008-03-20 13:23 EDT, Neil Horman
no flags Details | Diff
dump initrd from x86_64 (3.24 MB, application/octet-stream)
2008-04-04 14:36 EDT, Mike Gahagan
no flags Details
patch to force rtas section to be a multiple of PAGE_SIZE (1.47 KB, patch)
2008-05-19 15:46 EDT, Neil Horman
no flags Details | Diff

  None (edit)
Description Mike Gahagan 2008-03-18 16:22:28 EDT
Description of problem:
[root@squad7-lp1 2008-03-18-15:18]# makedumpfile -c ./vmcore ./vmcore-compressed
[  0 %]write_kdump_pages: Can't read the dump memory(./vmcore). Success

makedumpfile Failed.

[root@squad7-lp1 2008-03-18-15:18]# ll -h
total 13G
-r-------- 1 root root 7.6G Mar 18 15:19 vmcore
-rw------- 1 root root 9.1M Mar 18 15:37 vmcore-compressed

It wrote about 9MB before failing. The crashdump itself is from kernel
2.6.18-85.el5 and was created by a systemtap hook into __oom_kill_task that
calls panic() just in case it makes any difference. 

Version-Release number of selected component (if applicable):

How reproducible:
always on ppc with this particular dump file, not sure if the dump file itself
makes any difference. 

I was able to use the same makedumpfile command on an x86_64 box to create a
compressed dump usable  by crash. 

Steps to Reproduce:
1.run 'makedumpfile -c vmcore newcompressedvmcore' on a ppc system
Actual results:
makedumpfile fails

Expected results:
makedumpfile completes sucessfully

Additional info:

From Neil H.

[15:56] <nhorman> given that the error code above is 0 (success), I'm guessing
read just returned less than it was requested to read, and I need to implement a
safe_read to loop around and get the rest of the data
Comment 1 Mike Gahagan 2008-03-18 16:22:28 EDT
Created attachment 298450 [details]
strace of failing makedumpfile on ppc
Comment 3 Neil Horman 2008-03-19 14:04:24 EDT
Created attachment 298559 [details]
patch to make read continue until error or complete read

here mike, give this patch a try. It should guarantee that read reads the size

Also, this may not need to be a blocker/exception.  You're using mkdumpfile on
a running system, I've not observed this same issue while using makedumpfile
from within kdump (see the core_collector option in kdump.conf).  Not saying it
won't happen, just that I've not seen it.
Comment 4 Mike Gahagan 2008-03-20 10:17:52 EDT
Created attachment 298698 [details]
strace from failed makedumpfile on ppc with the patch.

I'm getting into an infinate loop with the patch, it appears that the
compressed dump file is getting up to about 9mb in size as before, but then we
get stuck in read() and don't make any more progress. I edited the strace file
and cut off most of the read() calls at the end to make the file smaller.

strace -o makedumpfile-ppc.strace makedumpfile -c ./vmcore.orig
[  0 %]

[1]+  Stopped		      strace -o makedumpfile-ppc.strace makedumpfile -c
./vmcore.orig ./vmcore-compressed
[root@squad7-lp1 2008-03-18-15:18]# bg
[1]+ strace -o makedumpfile-ppc.strace makedumpfile -c ./vmcore.orig
./vmcore-compressed &
[root@squad7-lp1 2008-03-18-15:18]# kill %1
[root@squad7-lp1 2008-03-18-15:18]# ll
total 7169416
-rw-r--r-- 1 root root	 27841322 Mar 20 10:10 makedumpfile-ppc.strace
-rw------- 1 root root	  9506320 Mar 20 10:10 vmcore-compressed
-r-------- 1 root root	821446247 Mar 18 15:19 vmcore.gz
-r-------- 1 root root 8062906676 Mar 18 15:33 vmcore.orig
Comment 5 Neil Horman 2008-03-20 10:50:26 EDT
doh!  Forgot the EOF condition check.  I'll fix that up shortly.  Thanks!
Comment 6 Neil Horman 2008-03-20 13:23:33 EDT
Created attachment 298718 [details]
new makedumpfile patch

Here you go, Mike, give this a shot, but I'm starting to wonder if perhaps
theres not something else going on here.  Either something is bad about this
core file, or we're reading more data than we need to be.  Anywho, give this a
shot, and we'll go from there with the results.
Comment 7 Neil Horman 2008-03-20 13:35:14 EDT
Mike, if that doesn't fix your problem, please do me a favor and point me to
this vmcore file that is failing.  This may be a kernel bug.  Its possible that
a note header in the elf file is reporting a size larger than it actually is,
running us out to the end of the file.  Thanks
Comment 8 Mike Gahagan 2008-03-28 16:02:37 EDT
It looks like this latest patch is also failing in a different way, here is the

[root@ibm-p5-185-01 2008-03-28-15:48]# cat strace.out
execve("/sbin/makedumpfile", ["makedumpfile", "-c", "./vmcore",
"vmcore-compressed"], [/* 23 vars */]) = 0
uname({...})                            = 0
brk(0)                                  = 0x100e0000
brk(0x100e0f44)                         = 0x100e0f44
brk(0x10110f44)                         = 0x10110f44
brk(0x10120000)                         = 0x10120000
open(0xffe5fbd7, O_RDONLY|O_LARGEFILE)  = 3
open(0xffe5fbe0, O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 4
gettimeofday({...}, NULL)               = 0
getpid()                                = 2542
open("/tmp/kdump_bitmapmcjyGu", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 5
unlink("/tmp/kdump_bitmapmcjyGu")       = 0
lseek(3, 0, SEEK_SET)                   = 0
read(3, "", 0)                          = 0
write(2, "check_elf_format", 16)        = 16
write(2, ": ", 2)                       = 2
write(2, 0xffe5c898, 29)                = 29
write(2, "\n", 1)                       = 1
write(2, "makedumpfile Failed.\n", 21)  = 21
close(3)                                = 0
close(4)                                = 0
close(5)                                = 0
exit_group(1)                           = ?

I have tried 2 vmcores, the one I found this bug with originally as well as a
new vmcore I created with sysrq-c. makedumpfile seems to have the same problem
with both files. 
Comment 10 Neil Horman 2008-03-28 17:56:32 EDT
mike, jsut out of curiosity, the nominal use case for makedumpfile is within the
kdump initramfs, using /proc/vmcore as the input file.  Can you tell me if it
works in that case?  It would help me narrow down whats going on here.
Comment 11 Mike Gahagan 2008-04-02 13:40:19 EDT
I have tried this a couple of times both with and without the kexec-tools patch
from comment 6. It looks like when enabled via the config file, the option is
just being ignored and is doing nothing. I don't get any errors or anything, but
the vmcore is about 7.5GB which would be the expected uncompressed size. 

[root@squad3 2008-04-02-13:24]# ll -h
total 6.1G
-r-------- 1 root root 7.4G Apr  2 13:30 vmcore

perhaps we are hitting another bug when enabling the -c option to makedumpfile
because when I look at the output when the initrd gets rebuilt, I am seeing:
[root@squad3 ~]# /etc/init.d/kdump restart
Stopping kdump:[  OK  ]
Detected change(s) the following file(s):

Rebuilding /boot/initrd-2.6.18-87.el5kdump.img
ls: /etc/ld.so.conf.d/*: No such file or directory
Starting kdump:[  OK  ]

In the config file, I just uncommented the example line that turns on page

[root@squad3 2008-04-02-13:24]# grep make /etc/kdump.conf
#                         NOTE: make sure user has necessary write
# core_collector makedumpfile <options>
#                         program makedumpfile to retrieve your core, which on
#                         See /sbin/makedumpfile --help for a list of options.
core_collector makedumpfile -c

Comment 12 Neil Horman 2008-04-02 20:30:35 EDT
ok, but makedumpfile works when we use it against /proc/vmcore, but not with a
real file.  I wonder if one of the elf notes is malformed or something, with a
bogus offset, leading to a seek to a non-existant part of the file.  How was the
vmcore file that you are trying to compress on the filesystem created?  Did
kdump use makedumpfile to filter it during capture?  was it generated with cp in
kdump?  Or did it come from a netdump/diskdump crash?
Comment 13 Mike Gahagan 2008-04-03 09:42:16 EDT

From looking at the file sizes, it looks like either kdump is just ignoring the
config file, or for whatever reason it doesn't compress the file or it is
falling back to just copying the file over because makedumpfile is failing in
the same way as before. I didn't see any error messages or any other output
aside from what is comment 11, but I definately did not get a compressed core file.

Comment 14 Neil Horman 2008-04-04 09:52:53 EDT
ok, but how was this vmcore created?  kdump/netdump/diskdump?  If it was kdump,
was it already passed through makedumpfile, or was it just copied out of
Comment 15 Mike Gahagan 2008-04-04 11:28:22 EDT
All the vmcores mentioned in this bug on ppc were created with kdump, using the
default configuration which puts the vmcore in /var/crash/whatever. The only
change I made to the default configuration was to try and get a compressed dump
in comment# 11. I can also create the exact behavior described in comment 11 on
an x86_64 system (core_collector makedumpfile -c in kdump.conf is ignored). As
mentioned before, makedumpfile -c on an existing vmcore on x86_64 works just
fine. See below:

[root@test177 2008-04-04-10:49]# makedumpfile -c vmcore.orig vmcore-compressed
[100 %]

The dumpfile is saved to vmcore-compressed.

makedumpfile Completed.
[root@test177 2008-04-04-10:49]# echo $?
[root@test177 2008-04-04-10:49]# ll
total 4336576
-rw------- 1 root root 1259350675 Apr  4 10:58 vmcore-compressed
-r-------- 1 root root 1241337138 Apr  4 10:50 vmcore.gz
-r-------- 1 root root 2016968784 Apr  4 10:55 vmcore.orig
[root@test177 2008-04-04-10:49]# uname -a
Linux test177.test.redhat.com 2.6.18-88.el5 #1 SMP Tue Apr 1 19:01:18 EDT 2008
x86_64 x86_64 x86_64 GNU/Linux

I think we have two bugs here:

1.) makedumpfile -c on ppc doesn't work (this bug)
2.) core_collector makedumpfile -c (the commented example in /etc/kdump.conf)
either is an improper configuration (which needs to be fixed in the default
kdump.conf file) or kdump/kexec is broken in some way that is causing it to
ignore the makedumpfile line completely. (no bug filed yet)

I can't verify if makedumpfile -c even works from the kexec envionment on ppc
due to problem #2, however I can verify that makedumpfile -c works on x86_64 on
an existing vmcore file and problem #2 also occurs on x86_64.
Comment 16 Neil Horman 2008-04-04 11:50:02 EDT
Ok, thats what I needed to know, thanks.  I think this is still one bug,
specifically, as you origionally describe, I think our two bugs are:

1) makedumpfile doesn't work on real files, only on /proc/vmcore at the moment
(this may be specific to ppc at the moment, I'm not sure).

2) compression doesn't work (or just works very very poorly) on ppc

Given that makedumpfile is only usually meant to be used in kdump environments,
I think we can put (1) on the back burner for now, and focus on (2).  I'll
reserve a ppc system and see if I can track down whats going on here shortly
Comment 17 Mike Gahagan 2008-04-04 12:18:49 EDT
So if I were to look into an initrd created by mkdumprd, where would I find the
command that actually gets the dump from /proc/ to /var/crash? I looked at the
initrd that /etc/init.d/kdump made when I reconfigured kdump.conf and I can't
find anything that actually copies the file over. I figured it put this in /init
or something it called, but I can't find it after doing find/grep searches for
makedumpfile, vmcore, and cp. 

from looking at mkdumprd, we fall back to using cp for the vmcore thinking that
somewhere in all the regrex stuff we do in mkdumprd we are clobbering the
makedumpfile command and going back to cp, but I can't find where we actually
copy the dump over to make sure.

Comment 18 Neil Horman 2008-04-04 13:27:49 EDT
well, thats interesting, you should have found at least a cp or makedumpfile
command in the init script of the initramfs.  Can you send me a copy of the
initramfs that was generated on your system?  Thanks!
Comment 19 Mike Gahagan 2008-04-04 14:36:47 EDT
Created attachment 301320 [details]
dump initrd from x86_64

Here is the initrd created by '/etc/init.d/kdump restart' after I enabled core
dump compression in the config file.

[root@test177 ~]# grep -v '#' /etc/kdump.conf

core_collector makedumpfile -c
Comment 20 Neil Horman 2008-04-04 15:25:34 EDT
you never specified a dump target, Mike :)

You need to spcify a location to dump the vmcore too.  Otherwise, it takes the
default action which is to mount the root filesystem and capture the core via
the initscript which only uses cp.  You need to specify a filesystem to dump to
(via scp/nfs/etc or locally using a raw disk or ext[2|3] filesystem.  If you
don't do that then nothing gets filtered by makedumpfile
Comment 21 Mike Gahagan 2008-04-04 16:00:21 EDT
ahh.. so now you tell me :)

anyways.. my x86_64 test system has /var/crash on its own partition (long story)
so I did:

[root@test177 initrd]# grep -v '#' /etc/kdump.conf

ext3 /dev/VolGroup00/LogVolCrash
core_collector makedumpfile -c

and after rebooting the system I got what appears to be a compressed vmcore in:

close enough for this purpose... nothing really to do with problem #2 aside from
fixing documentation no one reads anyway :). If I can get my hands on a ppc
system next week I can try some things on it again, but it will be a while
before I get to it again. 

I also opened up the kdump initrd and this time I can see the expected
makedumpfile -c command. 

Comment 22 Neil Horman 2008-04-04 16:27:25 EDT
ok, good, so it works properly on the x86_64 system.  Does the same thing happen
on the ppc system?
Comment 23 CAI Qian 2008-04-07 06:20:42 EDT
I observed the simiar problem within kdump only on ppc before, makedumpfile just
got an incomplete vmcore, and then failed with exit code 1, so kdump then
attempted to enter user-space to capture vmcore. I have tried in kdump kernel
manually running,

makedumpfile --message-level 1 -c /proc/vmcore vmcore-incomplete

it also returned 1 after a little while. The machine in question was
Comment 24 CAI Qian 2008-04-07 10:29:31 EDT
From testing logs, the above failure can be found for machine
ibm-js21-01.lab.boston.redhat.com too.
Comment 25 CAI Qian 2008-04-07 11:22:33 EDT
Comment 26 CAI Qian 2008-04-07 18:55:34 EDT
core_collector does work for a few ppc machines like
ibm-js20-01.lab.boston.redhat.com though,

- filtered vmcore -
-rw------- 1 root root  124938124 Apr  7 13:13

- unfiltered vmcore -
-r-------- 1 root root 1817780228 Apr  7 13:08
Comment 27 CAI Qian 2008-04-07 23:09:22 EDT
The problem can be reliably reproduced on kernel-2.6.18-87.el5 and
kexec-tools-1.102pre-18.el5 by,

- reserve one of the affected ppc64 machines.
- add crashkernel=256M@32M and reboot
- start kdump service with default setup.
- turn off kdump daemon by default (chkconfig kdump off)
- SysRq-C

When entering into kdump kernel, issue the following command,

makedumpfile --message-level 15 /proc/vmcore vmcore-incomplete

failing messages and strace logs are almost identical to Mike's,

[  0 %]write_kdump_pages: Can't read the dump memory(/proc/vmcore). Success

makedumpfile Failed.

However, it can successfully dump vmcore in ELF format (-E).
Comment 28 Neil Horman 2008-05-01 16:03:15 EDT
make, I just tried to recreate this on a ppc box I reserved from rhts
(ibm-l4b-lp1.test.redhat.com), and using kernel-2.6.18-91.el5 and
kexec-tools-1.102pre-21.el5), I was able to :
1) capture a vmcore
2) manually issue the following command:
   makedumpfile -c ./vmcore ./vmcore-compressed
3) read the resultant dump file with crash

Can you please test on the system you origionally had this problem with using
that kernel and kexec-tools package and verify that the problem still exists? 
Comment 29 Mike Gahagan 2008-05-15 17:06:39 EDT

I'm seeing the same problem as before on squad6-lp1 running 5.2:

[root@squad6-lp1 2008-05-15-16:50]# makedumpfile -c ./vmcore ./vmcore-compressed
[  0 %]write_kdump_pages: Can't read the dump memory(./vmcore). Success

makedumpfile Failed.
[root@squad6-lp1 2008-05-15-16:50]# rpm -q kernel
[root@squad6-lp1 2008-05-15-16:50]# rpm -q kexec-tools
[root@squad6-lp1 2008-05-15-16:50]# hostname
Comment 30 Neil Horman 2008-05-19 10:58:31 EDT
Note to self: In researching this, it looks like the failing systems are trying
to do a paddR_to_offset conversion on an address that falls  within the range of
a section header in the origional vmcore file.  The problem is that the intended
subsequent read of PAGE_SIZE bytes (65536 bytes on ppc64) extends beyond the end
of the section.  Given that its the last section in the uncompresed vmcore, we
read off the end of the file.  Not sure if this is a problem in the logic of
makedumpfile yet, or if the kernel is formatting this last section of the elf
file incorrectly.
Comment 31 Neil Horman 2008-05-19 11:28:50 EDT
Note to self: Looks like the last section in the offending vmcore file isn't a
multiple of PAGE_SIZE in length, and thats where this problem is stemming from.
 I need to check to see if LOAD sections in ELF files that end on non PAGE_SIZE
boundaries are legal to make sure, but that looks like our problem here.
Comment 32 Neil Horman 2008-05-19 13:51:41 EDT
Note to self: Looks like the RTAS interface is at fault here.  We create an ELF
note for the RTAS memory region based on the information in /proc/device-tree/rtas

linux,rtas-base seems to be page aligned, but its size is not.  Not sure if the
error is that size needs to be a multiple of PAGE_SIZE, or if userspace needs to
cope with this yet.
Comment 33 Neil Horman 2008-05-19 15:46:18 EDT
Created attachment 306002 [details]
patch to force rtas section to be a multiple of PAGE_SIZE

Ok, I think we found the problem.

The problem stems from there being a section in the uncompressed vmcore file
which is, while aligned to a PAGE_SIZE boundary, not of a size that is a
multiple of PAGE_SIZE.	Since makedumpfile attempts to read things in PAGE_SIZE
chunks, we read off the end of this section, and given that its the last
section in the vmcore file, we read off the end of the file itself.

This short section is formed when we try to build an extra section in kexec for
the rtas memory area.  We form the section by reading
/proc/device-tree/rtas/linux,rtas-base and rtas-size.  rtas-size doesn't have
any requirement to be a multiple of PAGE_SIZE, even though its memory mapped. 
The remainder of the page jsut reads 0 or some other value, so its safe to
read, but because its not accounterd for in the proc file size, we need to make
up for that in kexec.  This patch rounds the rtas area up to the next page size
so that makedumpfile can read it properly.  I just testing this out on squad6
and it worked fine.  Mike, if you could test an d please confirm, I'd
appreciate it.	Just so that I remember, this will need to go upstream as well.
Comment 34 Neil Horman 2008-05-27 08:01:12 EDT
mike, I just tested this locally again, and it worked fine, so I'm going to
assume its good.  I've gotten it in upstream, so I'm going to check it in
pending PM approval.  Thanks!
Comment 35 Neil Horman 2008-05-28 14:06:59 EDT
fixed in -23.el5.
Comment 39 errata-xmlrpc 2009-01-20 16:00:18 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.