Description of problem: [root@squad7-lp1 2008-03-18-15:18]# makedumpfile -c ./vmcore ./vmcore-compressed [ 0 %]write_kdump_pages: Can't read the dump memory(./vmcore). Success makedumpfile Failed. [root@squad7-lp1 2008-03-18-15:18]# ll -h total 13G -r-------- 1 root root 7.6G Mar 18 15:19 vmcore -rw------- 1 root root 9.1M Mar 18 15:37 vmcore-compressed It wrote about 9MB before failing. The crashdump itself is from kernel 2.6.18-85.el5 and was created by a systemtap hook into __oom_kill_task that calls panic() just in case it makes any difference. Version-Release number of selected component (if applicable): kexec-tools-1.102pre-11.el5 How reproducible: always on ppc with this particular dump file, not sure if the dump file itself makes any difference. I was able to use the same makedumpfile command on an x86_64 box to create a compressed dump usable by crash. Steps to Reproduce: 1.run 'makedumpfile -c vmcore newcompressedvmcore' on a ppc system 2. 3. Actual results: makedumpfile fails Expected results: makedumpfile completes sucessfully Additional info: From Neil H. [15:56] <nhorman> given that the error code above is 0 (success), I'm guessing read just returned less than it was requested to read, and I need to implement a safe_read to loop around and get the rest of the data
Created attachment 298450 [details] strace of failing makedumpfile on ppc
Created attachment 298559 [details] patch to make read continue until error or complete read here mike, give this patch a try. It should guarantee that read reads the size requested. Also, this may not need to be a blocker/exception. You're using mkdumpfile on a running system, I've not observed this same issue while using makedumpfile from within kdump (see the core_collector option in kdump.conf). Not saying it won't happen, just that I've not seen it.
Created attachment 298698 [details] strace from failed makedumpfile on ppc with the patch. I'm getting into an infinate loop with the patch, it appears that the compressed dump file is getting up to about 9mb in size as before, but then we get stuck in read() and don't make any more progress. I edited the strace file and cut off most of the read() calls at the end to make the file smaller. strace -o makedumpfile-ppc.strace makedumpfile -c ./vmcore.orig ./vmcore-compressed [ 0 %] [1]+ Stopped strace -o makedumpfile-ppc.strace makedumpfile -c ./vmcore.orig ./vmcore-compressed [root@squad7-lp1 2008-03-18-15:18]# bg [1]+ strace -o makedumpfile-ppc.strace makedumpfile -c ./vmcore.orig ./vmcore-compressed & [root@squad7-lp1 2008-03-18-15:18]# kill %1 [root@squad7-lp1 2008-03-18-15:18]# ll total 7169416 -rw-r--r-- 1 root root 27841322 Mar 20 10:10 makedumpfile-ppc.strace -rw------- 1 root root 9506320 Mar 20 10:10 vmcore-compressed -r-------- 1 root root 821446247 Mar 18 15:19 vmcore.gz -r-------- 1 root root 8062906676 Mar 18 15:33 vmcore.orig
doh! Forgot the EOF condition check. I'll fix that up shortly. Thanks!
Created attachment 298718 [details] new makedumpfile patch Here you go, Mike, give this a shot, but I'm starting to wonder if perhaps theres not something else going on here. Either something is bad about this core file, or we're reading more data than we need to be. Anywho, give this a shot, and we'll go from there with the results.
Mike, if that doesn't fix your problem, please do me a favor and point me to this vmcore file that is failing. This may be a kernel bug. Its possible that a note header in the elf file is reporting a size larger than it actually is, running us out to the end of the file. Thanks
It looks like this latest patch is also failing in a different way, here is the strace: [root@ibm-p5-185-01 2008-03-28-15:48]# cat strace.out execve("/sbin/makedumpfile", ["makedumpfile", "-c", "./vmcore", "vmcore-compressed"], [/* 23 vars */]) = 0 uname({...}) = 0 brk(0) = 0x100e0000 brk(0x100e0f44) = 0x100e0f44 brk(0x10110f44) = 0x10110f44 brk(0x10120000) = 0x10120000 open(0xffe5fbd7, O_RDONLY|O_LARGEFILE) = 3 open(0xffe5fbe0, O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 4 gettimeofday({...}, NULL) = 0 getpid() = 2542 open("/tmp/kdump_bitmapmcjyGu", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 5 unlink("/tmp/kdump_bitmapmcjyGu") = 0 lseek(3, 0, SEEK_SET) = 0 read(3, "", 0) = 0 write(2, "check_elf_format", 16) = 16 write(2, ": ", 2) = 2 write(2, 0xffe5c898, 29) = 29 write(2, "\n", 1) = 1 write(2, "makedumpfile Failed.\n", 21) = 21 close(3) = 0 close(4) = 0 close(5) = 0 exit_group(1) = ? I have tried 2 vmcores, the one I found this bug with originally as well as a new vmcore I created with sysrq-c. makedumpfile seems to have the same problem with both files.
mike, jsut out of curiosity, the nominal use case for makedumpfile is within the kdump initramfs, using /proc/vmcore as the input file. Can you tell me if it works in that case? It would help me narrow down whats going on here.
I have tried this a couple of times both with and without the kexec-tools patch from comment 6. It looks like when enabled via the config file, the option is just being ignored and is doing nothing. I don't get any errors or anything, but the vmcore is about 7.5GB which would be the expected uncompressed size. [root@squad3 2008-04-02-13:24]# ll -h total 6.1G -r-------- 1 root root 7.4G Apr 2 13:30 vmcore perhaps we are hitting another bug when enabling the -c option to makedumpfile because when I look at the output when the initrd gets rebuilt, I am seeing: [root@squad3 ~]# /etc/init.d/kdump restart Stopping kdump:[ OK ] Detected change(s) the following file(s): /etc/kdump.conf Rebuilding /boot/initrd-2.6.18-87.el5kdump.img ls: /etc/ld.so.conf.d/*: No such file or directory Starting kdump:[ OK ] In the config file, I just uncommented the example line that turns on page compression: [root@squad3 2008-04-02-13:24]# grep make /etc/kdump.conf # NOTE: make sure user has necessary write # core_collector makedumpfile <options> # program makedumpfile to retrieve your core, which on # See /sbin/makedumpfile --help for a list of options. core_collector makedumpfile -c
ok, but makedumpfile works when we use it against /proc/vmcore, but not with a real file. I wonder if one of the elf notes is malformed or something, with a bogus offset, leading to a seek to a non-existant part of the file. How was the vmcore file that you are trying to compress on the filesystem created? Did kdump use makedumpfile to filter it during capture? was it generated with cp in kdump? Or did it come from a netdump/diskdump crash?
From looking at the file sizes, it looks like either kdump is just ignoring the config file, or for whatever reason it doesn't compress the file or it is falling back to just copying the file over because makedumpfile is failing in the same way as before. I didn't see any error messages or any other output aside from what is comment 11, but I definately did not get a compressed core file.
ok, but how was this vmcore created? kdump/netdump/diskdump? If it was kdump, was it already passed through makedumpfile, or was it just copied out of /proc/vmcore?
All the vmcores mentioned in this bug on ppc were created with kdump, using the default configuration which puts the vmcore in /var/crash/whatever. The only change I made to the default configuration was to try and get a compressed dump in comment# 11. I can also create the exact behavior described in comment 11 on an x86_64 system (core_collector makedumpfile -c in kdump.conf is ignored). As mentioned before, makedumpfile -c on an existing vmcore on x86_64 works just fine. See below: [root@test177 2008-04-04-10:49]# makedumpfile -c vmcore.orig vmcore-compressed [100 %] The dumpfile is saved to vmcore-compressed. makedumpfile Completed. [root@test177 2008-04-04-10:49]# echo $? 0 [root@test177 2008-04-04-10:49]# ll total 4336576 -rw------- 1 root root 1259350675 Apr 4 10:58 vmcore-compressed -r-------- 1 root root 1241337138 Apr 4 10:50 vmcore.gz -r-------- 1 root root 2016968784 Apr 4 10:55 vmcore.orig [root@test177 2008-04-04-10:49]# uname -a Linux test177.test.redhat.com 2.6.18-88.el5 #1 SMP Tue Apr 1 19:01:18 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux I think we have two bugs here: 1.) makedumpfile -c on ppc doesn't work (this bug) 2.) core_collector makedumpfile -c (the commented example in /etc/kdump.conf) either is an improper configuration (which needs to be fixed in the default kdump.conf file) or kdump/kexec is broken in some way that is causing it to ignore the makedumpfile line completely. (no bug filed yet) I can't verify if makedumpfile -c even works from the kexec envionment on ppc due to problem #2, however I can verify that makedumpfile -c works on x86_64 on an existing vmcore file and problem #2 also occurs on x86_64.
Ok, thats what I needed to know, thanks. I think this is still one bug, specifically, as you origionally describe, I think our two bugs are: 1) makedumpfile doesn't work on real files, only on /proc/vmcore at the moment (this may be specific to ppc at the moment, I'm not sure). 2) compression doesn't work (or just works very very poorly) on ppc Given that makedumpfile is only usually meant to be used in kdump environments, I think we can put (1) on the back burner for now, and focus on (2). I'll reserve a ppc system and see if I can track down whats going on here shortly
So if I were to look into an initrd created by mkdumprd, where would I find the command that actually gets the dump from /proc/ to /var/crash? I looked at the initrd that /etc/init.d/kdump made when I reconfigured kdump.conf and I can't find anything that actually copies the file over. I figured it put this in /init or something it called, but I can't find it after doing find/grep searches for makedumpfile, vmcore, and cp. from looking at mkdumprd, we fall back to using cp for the vmcore thinking that somewhere in all the regrex stuff we do in mkdumprd we are clobbering the makedumpfile command and going back to cp, but I can't find where we actually copy the dump over to make sure.
well, thats interesting, you should have found at least a cp or makedumpfile command in the init script of the initramfs. Can you send me a copy of the initramfs that was generated on your system? Thanks!
Created attachment 301320 [details] dump initrd from x86_64 Here is the initrd created by '/etc/init.d/kdump restart' after I enabled core dump compression in the config file. [root@test177 ~]# grep -v '#' /etc/kdump.conf core_collector makedumpfile -c
you never specified a dump target, Mike :) You need to spcify a location to dump the vmcore too. Otherwise, it takes the default action which is to mount the root filesystem and capture the core via the initscript which only uses cp. You need to specify a filesystem to dump to (via scp/nfs/etc or locally using a raw disk or ext[2|3] filesystem. If you don't do that then nothing gets filtered by makedumpfile
ahh.. so now you tell me :) anyways.. my x86_64 test system has /var/crash on its own partition (long story) so I did: [root@test177 initrd]# grep -v '#' /etc/kdump.conf ext3 /dev/VolGroup00/LogVolCrash core_collector makedumpfile -c and after rebooting the system I got what appears to be a compressed vmcore in: /var/crash/var/crash/127.0.0.1-2008-04-04-15:46:47 close enough for this purpose... nothing really to do with problem #2 aside from fixing documentation no one reads anyway :). If I can get my hands on a ppc system next week I can try some things on it again, but it will be a while before I get to it again. I also opened up the kdump initrd and this time I can see the expected makedumpfile -c command.
ok, good, so it works properly on the x86_64 system. Does the same thing happen on the ppc system?
I observed the simiar problem within kdump only on ppc before, makedumpfile just got an incomplete vmcore, and then failed with exit code 1, so kdump then attempted to enter user-space to capture vmcore. I have tried in kdump kernel manually running, makedumpfile --message-level 1 -c /proc/vmcore vmcore-incomplete it also returned 1 after a little while. The machine in question was squad3.lab.boston.redhat.com.
From testing logs, the above failure can be found for machine ibm-js21-01.lab.boston.redhat.com too.
squad6-lp1.lab.boston.redhat.com
core_collector does work for a few ppc machines like ibm-js20-01.lab.boston.redhat.com though, - filtered vmcore - -rw------- 1 root root 124938124 Apr 7 13:13 /mnt/var/crash/ibm-js20-01.lab.boston.redhat.com/192.168.76.180-2008-04-07-13:12:02/vmcore - - unfiltered vmcore - -r-------- 1 root root 1817780228 Apr 7 13:08 /mnt/var/crash/ibm-js20-01.lab.boston.redhat.com/vmcore-full -
The problem can be reliably reproduced on kernel-2.6.18-87.el5 and kexec-tools-1.102pre-18.el5 by, - reserve one of the affected ppc64 machines. - add crashkernel=256M@32M and reboot - start kdump service with default setup. - turn off kdump daemon by default (chkconfig kdump off) - SysRq-C When entering into kdump kernel, issue the following command, makedumpfile --message-level 15 /proc/vmcore vmcore-incomplete failing messages and strace logs are almost identical to Mike's, [ 0 %]write_kdump_pages: Can't read the dump memory(/proc/vmcore). Success makedumpfile Failed. However, it can successfully dump vmcore in ELF format (-E).
make, I just tried to recreate this on a ppc box I reserved from rhts (ibm-l4b-lp1.test.redhat.com), and using kernel-2.6.18-91.el5 and kexec-tools-1.102pre-21.el5), I was able to : 1) capture a vmcore 2) manually issue the following command: makedumpfile -c ./vmcore ./vmcore-compressed 3) read the resultant dump file with crash Can you please test on the system you origionally had this problem with using that kernel and kexec-tools package and verify that the problem still exists? Thanks!
I'm seeing the same problem as before on squad6-lp1 running 5.2: RHEL5.2-Server-20080430.0 [root@squad6-lp1 2008-05-15-16:50]# makedumpfile -c ./vmcore ./vmcore-compressed [ 0 %]write_kdump_pages: Can't read the dump memory(./vmcore). Success makedumpfile Failed. [root@squad6-lp1 2008-05-15-16:50]# rpm -q kernel kernel-2.6.18-92.el5 [root@squad6-lp1 2008-05-15-16:50]# rpm -q kexec-tools kexec-tools-1.102pre-21.el5 [root@squad6-lp1 2008-05-15-16:50]# hostname squad6-lp1.rhts.bos.redhat.com
Note to self: In researching this, it looks like the failing systems are trying to do a paddR_to_offset conversion on an address that falls within the range of a section header in the origional vmcore file. The problem is that the intended subsequent read of PAGE_SIZE bytes (65536 bytes on ppc64) extends beyond the end of the section. Given that its the last section in the uncompresed vmcore, we read off the end of the file. Not sure if this is a problem in the logic of makedumpfile yet, or if the kernel is formatting this last section of the elf file incorrectly.
Note to self: Looks like the last section in the offending vmcore file isn't a multiple of PAGE_SIZE in length, and thats where this problem is stemming from. I need to check to see if LOAD sections in ELF files that end on non PAGE_SIZE boundaries are legal to make sure, but that looks like our problem here.
Note to self: Looks like the RTAS interface is at fault here. We create an ELF note for the RTAS memory region based on the information in /proc/device-tree/rtas linux,rtas-base seems to be page aligned, but its size is not. Not sure if the error is that size needs to be a multiple of PAGE_SIZE, or if userspace needs to cope with this yet.
Created attachment 306002 [details] patch to force rtas section to be a multiple of PAGE_SIZE Ok, I think we found the problem. The problem stems from there being a section in the uncompressed vmcore file which is, while aligned to a PAGE_SIZE boundary, not of a size that is a multiple of PAGE_SIZE. Since makedumpfile attempts to read things in PAGE_SIZE chunks, we read off the end of this section, and given that its the last section in the vmcore file, we read off the end of the file itself. This short section is formed when we try to build an extra section in kexec for the rtas memory area. We form the section by reading /proc/device-tree/rtas/linux,rtas-base and rtas-size. rtas-size doesn't have any requirement to be a multiple of PAGE_SIZE, even though its memory mapped. The remainder of the page jsut reads 0 or some other value, so its safe to read, but because its not accounterd for in the proc file size, we need to make up for that in kexec. This patch rounds the rtas area up to the next page size so that makedumpfile can read it properly. I just testing this out on squad6 and it worked fine. Mike, if you could test an d please confirm, I'd appreciate it. Just so that I remember, this will need to go upstream as well.
mike, I just tested this locally again, and it worked fine, so I'm going to assume its good. I've gotten it in upstream, so I'm going to check it in pending PM approval. Thanks!
fixed in -23.el5.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2009-0105.html