Bug 1169774
Summary: | failure to extract debuginfo | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Brian J. Murrell <brian> |
Component: | libreport | Assignee: | Matej Habrnal <mhabrnal> |
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | urgent | Docs Contact: | |
Priority: | unspecified | ||
Version: | 21 | CC: | abrt-devel-list, brian, dvlasenk, iprikryl, jfilak, maurizio.antillon, mhabrnal, michal.toman, mmilata, rvokal |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | gnome-abrt-1.0.0-3.fc21 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-06-30 20:20:04 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Brian J. Murrell
2014-12-02 12:28:10 UTC
Thank you for the report! Isn't this report a dupe of bug #1036918? If you have time to run the following commands $ cd /var/cache/abrt-di $ cpio -idu < /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio , please do so and report the results here. (In reply to Jakub Filak from comment #1) > Thank you for the report! Isn't this report a dupe of bug #1036918? Wow. I really mean wow. You are right, this is a duplicate of my own bug #1036918 and filed exactly 1 year later, to the day. What are the odds? > If you have time to run the following commands > > $ cd /var/cache/abrt-di > $ cpio -idu < > /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio Do you really mean for me to run this an a non-root user as the "$" prompt on your commands suggest? In any case, /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio does not exist here. But surely it's a temporary directory that is removed when abrt is done doing what it needs to with that dir, yes? (In reply to Brian J. Murrell from comment #2) > (In reply to Jakub Filak from comment #1) > > If you have time to run the following commands > > > > $ cd /var/cache/abrt-di > > $ cpio -idu < > > /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio > > Do you really mean for me to run this an a non-root user as the "$" prompt > on your commands suggest? I have to remember that the "$" prompt means non-root user. Could you please run those commands as root? > > In any case, > /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio does not > exist here. But surely it's a temporary directory that is removed when abrt > is done doing what it needs to with that dir, yes? The file should exist, because abrt removes it only after successful unpacking [1], but your log messages says ".*Can't extract files from.*" [2] (printed out in the else branch) Do you use a temporary data cleaner? I am afraid that something removed /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio before abrt got a chance to unpack it. 1: https://github.com/abrt/libreport/blob/master/src/client-python/debuginfo.py#L132 2: https://github.com/abrt/libreport/blob/master/src/client-python/debuginfo.py#L134 (In reply to Jakub Filak from comment #3) > Could you please run those commands as root? Well, I can't though because of... > The file should exist, because abrt removes it only after successful > unpacking [1], Clearly that's not the case. I added some debug (note the "ls -l" commands and their results below showing when the file is present): Downloading 78.57Mb, installed size: 371.56Mb. Continue? 'YES' Downloading (1 of 8) mate-control-center-debuginfo-1.8.3-1.fc20.x86_64.rpm: 100% Extracting cpio from /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/mate-control-center-debuginfo-1.8.3-1.fc20.x86_64.rpm # ls -l /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio -rw-rw-r--. 1 abrt abrt 4035356 Dec 2 10:53 /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio Caching files from unpacked.cpio made from mate-control-center-debuginfo-1.8.3-1.fc20.x86_64.rpm running ['cpio', '-idu'] < /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio in /var/cache/abrt-di out: err: cpio: ./usr/lib/debug: Cannot change mode to rwxr-xr-x: Operation not permitted cpio: ./usr/lib/debug/.build-id: Cannot change mode to rwxr-xr-x: Operation not permitted cpio: ./usr/lib/debug/.build-id/48: Cannot mkdir: Permission denied cpio: cannot make directory `./usr/lib/debug/.build-id/48': Permission denied cpio: ../../../../bin/mate-display-properties: Cannot symlink to `./usr/lib/debug/.build-id/48/9fedcd40c723f5947fe6002a66ef3759f6eee5': No such file or directory cpio: cannot make directory `./usr/lib/debug/.build-id/48': Permission denied cpio: ../../usr/bin/mate-display-properties.debug: Cannot symlink to `./usr/lib/debug/.build-id/48/9fedcd40c723f5947fe6002a66ef3759f6eee5.debug': No such file or directory cpio: ./usr/lib/debug/.build-id/71: Cannot mkdir: Permission denied cpio: cannot make directory `./usr/lib/debug/.build-id/71': Permission denied cpio: ../../../../lib64/libslab.so.0.0.0: Cannot symlink to `./usr/lib/debug/.build-id/71/8d2b95f4501fbf2308794739663e5aad2b12af': No such file or directory cpio: cannot make directory `./usr/lib/debug/.build-id/71': Permission denied cpio: ../../usr/lib64/libslab.so.0.0.0.debug: Cannot symlink to `./usr/lib/debug/.build-id/71/8d2b95f4501fbf2308794739663e5aad2b12af.debug': No such file or directory cpio: ./usr/lib/debug/.build-id/cd: Cannot mkdir: Permission denied cpio: cannot make directory `./usr/lib/debug/.build-id/cd': Permission denied cpio: ../../../../sbin/mate-display-properties-install-systemwide: Cannot symlink to `./usr/lib/debug/.build-id/cd/10783aa2e44351bd9fc19245c9bb7efc7728de': No such file or directory cpio: cannot make directory `./usr/lib/debug/.build-id/cd': Permission denied cpio: ../../usr/sbin/mate-display-properties-install-systemwide.debug: Cannot symlink to `./usr/lib/debug/.build-id/cd/10783aa2e44351bd9fc19245c9bb7efc7728de.debug': No such file or directory cpio: ./usr/lib/debug/.dwz: Cannot change mode to rwxr-xr-x: Operation not permitted cpio: ./usr/lib/debug/.dwz/mate-control-center-1.8.3-1.fc20.x86_64: Cannot open: Permission denied cpio: ./usr/lib/debug/usr: Cannot change mode to rwxr-xr-x: Operation not permitted cpio: ./usr/lib/debug/usr/lib64: Cannot change mode to rwxr-xr-x: Operation not permitted cpio: ./usr/lib/debug/usr/lib64/libmate-window-settings.so.1.0.0.debug: Cannot open: Permission denied cpio: libmate-window-settings.so.1.0.0.debug: Cannot symlink to `./usr/lib/debug/usr/lib64/libmate-window-settings.so.1.debug': Permission denied cpio: libmate-window-settings.so.1.0.0.debug: Cannot symlink to `./usr/lib/debug/usr/lib64/libmate-window-settings.so.debug': Permission denied cpio: ./usr/lib/debug/usr/lib64/libslab.so.0.0.0.debug: Cannot open: Permission denied cpio: libslab.so.0.0.0.debug: Cannot symlink to `./usr/lib/debug/usr/lib64/libslab.so.0.debug': Permission denied cpio: libslab.so.0.0.0.debug: Cannot symlink to `./usr/lib/debug/usr/lib64/libslab.so.debug': Permission denied cpio: ./usr/lib/debug/usr/lib64/window-manager-settings: Cannot mkdir: Permission denied cpio: cannot make directory `./usr/lib/debug/usr/lib64/window-manager-settings': Permission denied cpio: ./usr/lib/debug/usr/lib64/window-manager-settings/libmarco.so.debug: Cannot open: No such file or directory 7882 blocks rc: 2 Can't extract files from '/var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio' # ls -l /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio -rw-rw-r--. 1 abrt abrt 4035356 Dec 2 10:53 /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio Unpacking failed, aborting download... Missing debuginfo file: /usr/lib/debug/.build-id/4e/fe28aadfe0724926357e90872845dd5f714dcf.debug Missing debuginfo file: /usr/lib/debug/.build-id/75/691a21b24ec54372155190638a47ff80672304.debug Missing debuginfo file: /usr/lib/debug/.build-id/6d/9aabd27354e47d6b4c07f73c482691eee85ff5.debug Missing debuginfo file: /usr/lib/debug/.build-id/0c/2881db5c93250f1576dab3d17864d22031ef70.debug Missing debuginfo file: /usr/lib/debug/.build-id/ec/42b661db38fbf53d01e5f43e60f3e72e27d20b.debug Missing debuginfo file: /usr/lib/debug/.build-id/66/b06d49f01cd823398fac7b75b688b81736a5b6.debug Missing debuginfo file: /usr/lib/debug/.build-id/1c/d1edc030658c29de7035a6b83cf44443358ab3.debug Missing debuginfo file: /usr/lib/debug/.build-id/04/663fc1335f018d20f5494226de543d65eec92f.debug Missing debuginfo file: /usr/lib/debug/.build-id/60/cdd7495cbd92ed3b6f24c29162e08689ac9c85.debug Missing debuginfo file: /usr/lib/debug/.build-id/ba/ef53aa03759d3a6a2b9e41115afd54dab36d2e.debug Missing debuginfo file: /usr/lib/debug/.build-id/a9/ae2f275e5daffd09bada4aa4b8e9c3e859fcb1.debug Missing debuginfo file: /usr/lib/debug/.build-id/21/10a7c97d990f5197bf27acbb8f11e090612077.debug Missing debuginfo file: /usr/lib/debug/.build-id/c6/67bb13e948ae8f23b7a10e1dba7e5ebd3232a9.debug Missing debuginfo file: /usr/lib/debug/.build-id/25/4e182d3b0d9190b51de85c3fa1dcecdb844c83.debug Missing debuginfo file: /usr/lib/debug/.build-id/96/d5dc0e8574e3bdfcd7329340eeb7bb9b6b87c0.debug Missing debuginfo file: /usr/lib/debug/.build-id/48/9fedcd40c723f5947fe6002a66ef3759f6eee5.debug Missing debuginfo file: /usr/lib/debug/.build-id/13/900c8170b59a5239df1f79db31a51cfbaadf66.debug Missing debuginfo file: /usr/lib/debug/.build-id/30/d104d9cc4c8d997c5ef50b470ebeb68eaee683.debug Missing debuginfo file: /usr/lib/debug/.build-id/7c/eb1818a46ca58650559c9aebaea11cc91be0b6.debug Missing debuginfo file: /usr/lib/debug/.build-id/f8/988e18c02dc9eb82d025a00065a5eee5318e9c.debug Generating backtrace Backtrace is generated and saved, 139403 bytes Looking for similar problems in bugzilla Searching for updates No updates for this package found And yet, with the window that the above was pasted from still open, the file is gone: $ ls -l /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio ls: cannot access /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio: No such file or directory So as you can see, it was there, but removed by the time abrt was done processing that failure. But the debug has revealed the actual problem. The cpio fails due to EPERMs and ENOENTs. My /var/cache/abrt-di/ looks like this: $ ls -ldZ /var/cache/abrt-di/{,usr/{,lib/{,debug}}} drwxrwxr-x. abrt abrt system_u:object_r:abrt_var_cache_t:s0 /var/cache/abrt-di/ drwxrwxr-x. abrt abrt unconfined_u:object_r:abrt_var_cache_t:s0 /var/cache/abrt-di/usr/ drwxrwxr-x. abrt abrt unconfined_u:object_r:abrt_var_cache_t:s0 /var/cache/abrt-di/usr/lib/ drwxrwxr-x. root root unconfined_u:object_r:abrt_var_cache_t:s0 /var/cache/abrt-di/usr/lib/debug I'm guessing that last one, /var/cache/abrt-di/usr/lib/debug, should be owned by abrt.abrt, yes? > Do you use a temporary data cleaner? No. And even if I did, it certainly would not be one that removed anything that was not clearly stale (i.e. many days old). I have lots of stuff in /var/tmp/ that is way old: drwx------. 3 root root 4096 Jul 12 2013 systemd-private-dr11ns drwx------. 3 root root 4096 Jul 12 2013 systemd-private-cB4Hup drwx------. 3 root root 4096 Jul 12 2013 systemd-private-f6RyVm > I am afraid that something removed > /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio before > abrt got a chance to unpack it. Not at all, given the above analysis. abrt is doing some clean-up somewhere. Some better error reporting than just "it failed" would be nice here I think. Knowing it was EPERM and ENOENT would have helped here without having to smatter debug into the tool. Just providing the cpio stdout and stderr in the case of failure would likely be enough for most sysadmins to go figure out what the problem is. Thank you for the excellent analysis! You are right, abrt removes the temporary directory here [1]. I apologize for misleading you. Could you please clean /var/cache/abrt-di/, re-run reporting and report the results here? abrt uses /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache to unpack the missing debuginfo packages into the abrt cache and the executable should have 's' flags for both owner and group. If everything works as expected, then all files within /var/cache/abrt-di/ are owned by abrt.abrt. 1: https://github.com/abrt/libreport/blob/master/src/client-python/debuginfo.py#L492 (In reply to Jakub Filak from comment #5) > Thank you for the excellent analysis! NP. Sometimes the easiest way to resolve uncertainties. > Could you please clean /var/cache/abrt-di/, re-run reporting and report the > results here? I don't have to. I did a chown -R abrt.abrt /var/cache/abrt-di/usr/lib/debug and now everything seems to be working. > abrt uses /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache to unpack > the missing debuginfo packages into the abrt cache and the executable should > have 's' flags for both owner and group. It does and the owner and group are abrt and abrt. That is an important detail when dealing with set[gu]id bits. :-) > If everything works as expected, > then all files within /var/cache/abrt-di/ are owned by abrt.abrt. Right. And my chown above achieved that and abrt seems to be installing the necessary debuginfos now. Now if abrt would just manage the /var/cache/abrt-di cache like it purports to. But that's another ticket. :-) I like your idea of providing cpio output to sysadmins in the case of failure. We will forward cpio stdout and stderr to user visible temporary files and will remove them only if cpio succeeds in unpacking. I've created a pull request which fixes this bug. https://github.com/abrt/libreport/pull/347 This message is a reminder that Fedora 20 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 20. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '20'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 20 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. gnome-abrt-1.0.0-2.fc21,abrt-2.3.0-5.fc21,libreport-2.3.0-7.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/gnome-abrt-1.0.0-2.fc21,abrt-2.3.0-5.fc21,libreport-2.3.0-7.fc21 Package gnome-abrt-1.0.0-2.fc21, abrt-2.3.0-6.fc21, libreport-2.3.0-8.fc21: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing gnome-abrt-1.0.0-2.fc21 abrt-2.3.0-6.fc21 libreport-2.3.0-8.fc21' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-10193/gnome-abrt-1.0.0-2.fc21,abrt-2.3.0-6.fc21,libreport-2.3.0-8.fc21 then log in and leave karma (feedback). Package gnome-abrt-1.0.0-3.fc21, abrt-2.3.0-7.fc21, libreport-2.3.0-8.fc21: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing gnome-abrt-1.0.0-3.fc21 abrt-2.3.0-7.fc21 libreport-2.3.0-8.fc21' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-10193/gnome-abrt-1.0.0-3.fc21,abrt-2.3.0-7.fc21,libreport-2.3.0-8.fc21 then log in and leave karma (feedback). gnome-abrt-1.0.0-3.fc21, abrt-2.3.0-7.fc21, libreport-2.3.0-8.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report. |