Description of problem: abrt deletes dumps for crashes induced in an older version of shotwell: "Corrupted or bad dump ... deleting". By stopping abrtd and setting "ulimit -c unlimited", I was able to save a core file. gdb read it fine, and there were symbols in the backtrace, although gdb said: Missing separate debuginfos, use: debuginfo-install shotwell-0.9.1-1.fc15.x86_64 That debuginfo package is indeed not available, but "debuginfo-install" finds all other debuginfo packages either installed or available. I was actually trying to test the fix for Bug 672023 by downgrading to an older version of shotwell, for which there would be no debuginfo. Version-Release number of selected component (if applicable): abrt-2.0.1-2.fc15.x86_64 Installed: shotwell-0.9.1-1.fc15.x86_64 Latest: shotwell-0.9.2-1.fc15.x86_64 gdb-7.2.90.20110411-34.fc15.x86_64 How reproducible: Always. Steps to Reproduce: [joeblow@fir ~]$ shotwell & [1] 1841 [joeblow@fir ~]$ kill -6 1841 Actual results: See below. Expected results: Crash saved, or possibly a more detailed message in /var/log/messages. Additional info: Snippet from /var/log/messages: Apr 23 09:39:32 fir abrt[1852]: saved core dump of pid 1841 (/usr/bin/shotwell) to /var/spool/abrt/ccpp-2011-04-23-09:39:31-1841.new/coredump (49704960 bytes) Apr 23 09:39:32 fir abrtd: Directory 'ccpp-2011-04-23-09:39:31-1841' creation detected Apr 23 09:39:32 fir abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2011-04-23-09:39:31-1841 (res:2), deleting
While investigating this problem, I temporarily moved a ccpp directory to $HOME. When I moved it back to /var/spool/abrt/, abrtd immediately deleted it. The directory that was deleted had a shotwell coredump in it. In /var/spool/abrt/ [joeblow@fir abrt]$ sudo mv -i ~/ccpp-2011-04-23-08\:41\:02-1749/ . $ sudo tail -f /var/log/messages immediately showed: Apr 23 11:20:19 fir abrtd: Directory 'ccpp-2011-04-23-08:41:02-1749' creation detected Apr 23 11:20:19 fir abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2011-04-23-08:41:02-1749 (res:2), deleting
I tried the same thing with another ccpp directory ("sleep 1000" coredump). abrtd deleted it too. In /var/spool/abrt/ [joeblow@fir abrt]$ sudo mv -i /tmp/ccpp-2011-04-22-12\:07\:04-2100/ . Snippet from /var/log/messages: Apr 23 11:32:19 fir abrtd: Directory 'ccpp-2011-04-22-12:07:04-2100' creation detected Apr 23 11:32:20 fir abrtd: Dump directory is a duplicate of /var/spool/abrt/ccpp-2011-04-22-12:07:04-2100 Apr 23 11:32:20 fir abrtd: Deleting dump directory ccpp-2011-04-22-12:07:04-2100 (dup of ccpp-2011-04-22-12:07:04-2100), sending dbus signal
Whoop Dee Doo! [joeblow@fir ~]$ sudo mkdir /var/spool/abrt/foo [sudo] password for joeblow: [joeblow@fir ~]$ Apr 23 16:17:11 fir abrtd: '/var/spool/abrt/foo' is not a crash dump directory Apr 23 16:17:11 fir abrtd: Corrupted or bad dump /var/spool/abrt/foo (res:2), deleting Apr 23 16:17:12 fir abrtd: '/var/spool/abrt/foo' is not a crash dump directory
The directory foo does not get deleted ... [joeblow@fir ~]$ sudo mkdir /var/spool/abrt/foo [joeblow@fir ~]$ ls -lFd /var/spool/abrt/foo drwxr-xr-x. 2 root root 4096 Apr 23 23:10 /var/spool/abrt/foo/ Apr 23 23:10:12 fir abrtd: Directory 'foo' creation detected Apr 23 23:10:13 fir abrtd: '/var/spool/abrt/foo' is not a crash dump directory Apr 23 23:10:13 fir abrtd: Corrupted or bad dump /var/spool/abrt/foo (res:2), deleting Apr 23 23:10:13 fir abrtd: '/var/spool/abrt/foo' is not a crash dump directory
Even with matching debuginfo, the crash directory is removed: Apr 24 00:04:12 fir yum[1940]: Installed: shotwell-0.9.1-1.fc15.x86_64 Apr 24 00:04:14 fir yum[1940]: Installed: shotwell-debuginfo-0.9.1-1.fc15.x86_64 Apr 24 00:05:46 fir abrt[1982]: saved core dump of pid 1976 (/usr/bin/shotwell) to /var/spool/abrt/ccpp-2011-04-24-00:05:46-1976.new/coredump (49684480 bytes) Apr 24 00:05:46 fir abrtd: Directory 'ccpp-2011-04-24-00:05:46-1976' creation detected Apr 24 00:05:46 fir abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2011-04-24-00:05:46-1976 (res:2), deleting [joeblow@fir ~]$ ls -lFd /var/spool/abrt/ccpp-2011-04-24-00:05:46-1976 ls: cannot access /var/spool/abrt/ccpp-2011-04-24-00:05:46-1976: No such file or directory
strace on abrtd: read(11, "Package 'shotwell' isn't signed "..., 4096) = 48 shotwell-0.9.1-1.fc15.x86_64 is from koji: http://koji.fedoraproject.org/koji/buildinfo?buildID=237585
This message isn't getting logged: ./src/daemon/abrt-action-save-package-data.c: log("Package '%s' isn't signed with proper key", package_short_name); $ sudo strace -p 845 -s 256 ... mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14d548f000 lseek(10, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) read(10, "Package 'shotwell' isn't signed with proper key\n", 4096) = 48 read(10, "", 4096) = 0 close(10) = 0 ...
So the bug here is with the logging. By disabling the gpg check, I can get a crash dump with an unsigned package. Retitling. $ cat /etc/abrt/abrt.conf ... OpenGPGCheck = no ...
Steve, thanks for extensive analysis! And sorry for the delay on my part. Fixed in git: commit 92a230c7f996c67f32106f906dec0dd8e9996aca Author: Denys Vlasenko <dvlasenk> Date: Sun Jul 17 11:00:44 2011 +0200 abrtd: pipe post-create output to log. Now log will contain information why dump gets deleted: Jul 17 10:58:32 abrt[19991]: saved core dump of pid 19989 (/app/busybox-1.18.4/busybox) to /var/spool/abrt/ccpp-2011-07-17-10:58:32-19989.new/coredump (159744 bytes) Jul 17 10:58:32 abrtd: Directory 'ccpp-2011-07-17-10:58:32-19989' creation detected Jul 17 10:58:32 abrtd: Executable '/app/busybox-1.18.4/busybox' doesn't belong to any package Jul 17 10:58:32 abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2011-07-17-10:58:32-19989 (res:2), deleting
Thanks, Denys. http://git.fedorahosted.org/git/?p=abrt.git;a=commit;h=92a230c7f996c67f32106f906dec0dd8e9996aca
abrt-2.0.3-2.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/abrt-2.0.3-2.fc15
Package abrt-2.0.3-2.fc15: * should fix your issue, * was pushed to the Fedora 15 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing abrt-2.0.3-2.fc15' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/abrt-2.0.3-2.fc15 then log in and leave karma (feedback).
abrt-2.0.3-3.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/abrt-2.0.3-3.fc15
abrt-2.0.3-4.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/abrt-2.0.3-4.fc15
abrt-2.0.3-4.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.