Bug 1169774 - failure to extract debuginfo
Summary: failure to extract debuginfo
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libreport
Version: 21
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Matej Habrnal
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-02 12:28 UTC by Brian J. Murrell
Modified: 2015-06-30 20:20 UTC (History)
10 users (show)

Fixed In Version: gnome-abrt-1.0.0-3.fc21
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-30 20:20:04 UTC


Attachments (Terms of Use)

Description Brian J. Murrell 2014-12-02 12:28:10 UTC
Description of problem:
abrt got an error unpacking a debuginfo and did not install all 12 required debuginfos

Version-Release number of selected component (if applicable):
abrt-2.2.2-1.fc20.x86_64

How reproducible:
Unknown, but suspected fairly reproducible.

Steps to Reproduce:
1. Wait for something to crash
2. Start the abrt GUI tool
3. Say no to uploading a coredump and allow it to (attempt to) install required debug packages

Actual results:
Doesn't install all packages

Expected results:
Should install all packages and backtraces should be fully resolved.

Additional info:
event_log of an abrt session that failed to download all packages:

2014-12-02-06:59:44> ('report_uReport' completed successfully)
2014-12-02-06:59:47> Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'NO'
2014-12-02-06:59:47  Analyzing coredump 'coredump'
2014-12-02-06:59:48  Missing build id: libsane-brother3.so.1
2014-12-02-06:59:49  Coredump references 157 debuginfo files, 96 of them are not installed
2014-12-02-06:59:50  Setting up yum repositories
2014-12-02-07:00:05  Can't setup rpmfusion-free-rawhide-debuginfo: failure: repodata/repomd.xml from rpmfusion-free-rawhide-debuginfo: [Errno 256] No more mirrors to try.
2014-12-02-07:00:05  http://linux.interlinx.bc.ca:3142/rpmfusion/free/fedora/development/x86_64/os/debug/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found, disabling
2014-12-02-07:00:06  Can't setup rpmfusion-nonfree-rawhide-debuginfo: failure: repodata/repomd.xml from rpmfusion-nonfree-rawhide-debuginfo: [Errno 256] No more mirrors to try.
2014-12-02-07:00:06  http://linux.interlinx.bc.ca:3142/rpmfusion/nonfree/fedora/development/x86_64/os/debug/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found, disabling
2014-12-02-07:00:14  Looking for needed packages in repositories
2014-12-02-07:00:41  Can't find packages for 8 debuginfo files
2014-12-02-07:00:41  Packages to download: 12
2014-12-02-07:05:12  Downloading 88.12Mb, installed size: 421.18Mb. Continue? 'YES'
2014-12-02-07:05:13  Downloading (1 of 12) v4l-utils-debuginfo-1.0.0-1.fc20.x86_64.rpm: 100%
2014-12-02-07:05:13  Extracting cpio from /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/v4l-utils-debuginfo-1.0.0-1.fc20.x86_64.rpm
2014-12-02-07:05:13  Caching files from unpacked.cpio made from v4l-utils-debuginfo-1.0.0-1.fc20.x86_64.rpm
2014-12-02-07:05:13  Can't extract files from '/var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio'
2014-12-02-07:05:13  Unpacking failed, aborting download...
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/b9/8358069b9515eea3caa3a7984c6c0b5d475c6b.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/27/b2540a6bdfe98730e83d6c2e3b741c1e3bb0f1.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/48/449558009178acc72b07ce5c314dd0eef7ca75.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/0c/2881db5c93250f1576dab3d17864d22031ef70.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/47/d705265057a2c90c9ed15120a4203d39ef6ee8.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/84/6c0da4dfe0f34d53b850e79661ab6f5c6dfa7d.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/5e/2465ae4bf49bff9bdf5ef48cadcaf8567e354a.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/4c/a76c4bc8180e226d80f873eff0cbfb8cb7bd7e.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/e3/4a8b35a70d831ceefc0e13b7cfdd44933d4901.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/4e/a6af7e56de2fe9d08608ccbe8112ba5c505d44.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/8d/c7cc988d6e125583ee81947e057e849d712902.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/f8/9e2db5e3aefe4e20f572f84642b24ebf7acaf0.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/d9/d64e056bd38a421b7b17ef0b3f8218524d3c20.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/9f/f6a694bea9ea6b2e87b10d15215b8f11dbc8ba.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/85/8f9e59db5d614185b76829c2dc2adf22837199.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/db/1085a5b849a361d9064a84060ff4c4ba0510c5.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/08/6d7d7232c20d0d3ef273bb7701d4bd3526aa17.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/6f/74542d3ccc0cd129d05e7362e41d90c55649a2.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/d8/7a0ca1d7c6c7906d9b3ea561befd72495c9a3b.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/7c/eb1818a46ca58650559c9aebaea11cc91be0b6.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/ee/225b525d26ae13e3f43d6cb3c2bdd5f850f0f3.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/c1/357822d569656b0dcac4bf42cbf2ab446e318d.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/06/ed4487c3ddbfacdc08ec3256e422ae046f47cc.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/4e/fe28aadfe0724926357e90872845dd5f714dcf.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/48/1cd47efeab17a29923b46c078397ecbddfbafc.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/33/1a9a487f9da0f4dc4648f64bd18f5aab854f5a.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/ba/ef53aa03759d3a6a2b9e41115afd54dab36d2e.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/4c/1d292e58aa5b7ab1a3fc54ed727c51903b14ac.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/72/fa258656502943eed4931c3cb9cd26d16ea42f.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/dc/809994869eef3b8a99c91f481973f5b27c143c.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/8d/08792e631265191218ed527586ccffe60c82e6.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/80/bd356d44954557a0a1d488516763837fb97ee4.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/9b/d26ae8d9f6c9590991d45046c093987501c877.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/b0/a0527b21de82b359880a25926d2effe9e365ff.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/8a/c8ddf1648ca478117b1d92e91fb2fd1417da12.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/a6/5be18ba96c28624ec772520bdacb92f045e64c.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/cd/71158d55ebf07cb10923663c031ddbf72a2571.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/21/10a7c97d990f5197bf27acbb8f11e090612077.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/e0/af70d7e3aa565509f3ec97d287639363de7148.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/7d/a590964b311dd0e660046b398b983bda487800.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/d2/49c9921df32a5c177662b923482aeddbe1ceb7.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/27/b8771f94ef2d502a7e009977bd1d20120a1ea7.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/af/f5c329c258e7a121240bc5273bea39bcdc1ad3.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/87/7c368b60d8721d0093f92a2594fa278bdd4732.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/6e/2eabb295bc2e7e91a96ee3cbb0a3a44011adaf.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/27/b27058479a0d6cbf04d97e8aafd41570b2fbad.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/e9/0b93ccd59dbd2d16990809108aa3a076ecffd2.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/d8/540efdbf2b97f0cd4932dcad452cfe465df554.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/2a/edb747f9cdb75bc6b7b75d7c66c22e92535617.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/13/900c8170b59a5239df1f79db31a51cfbaadf66.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/99/a35f77e52cadc4028f3c5874dd1329280dda57.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/66/b06d49f01cd823398fac7b75b688b81736a5b6.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/5d/2fb4b989e99a33cba15b418e3eb9e7329efc3c.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/60/cdd7495cbd92ed3b6f24c29162e08689ac9c85.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/4b/14503c993a60b9fec133aa254260bfa756e6b2.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/0e/35aaa2bd5ae89828cb546cf2c919fdff41243d.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/e9/eb8e7d76553efd00b908582dfc7ce211c7b76c.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/01/9039560420ffe008648430552a14c9c0011917.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/25/4e182d3b0d9190b51de85c3fa1dcecdb844c83.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/a5/9a201ee5389067dd6258e1dfffae439e5f207e.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/48/31b89a8a940b47a81ceb7d5150f57f9cf81705.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/51/c218a7bb54c51b0ac9c636ef1e343910b55e27.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/77/0b2b484caafce2418c8fbe5e044c399e8c9d7d.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/97/6e540e2f8a9a17df36981b76526623fcf066cc.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/7a/23c68fdca37972e8fa19d5bed990bd1c3bee76.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/59/4ff8e719a02ab4281e1eca593986895d37a33a.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/7c/9ad9446cf07c4fb8369ab6dd631358113756b4.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/a7/2d18871b8e06a41d59de372f868f3bed0a2cc3.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/92/05e26148bf99786da44a48309cd05f5943bf76.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/75/691a21b24ec54372155190638a47ff80672304.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/ee/b3adc2d2d0ccc9808f5f218d88097313f48555.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/cc/3c0811b2f351f333cc221b60a306c9ccb44ce0.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/77/d74db05ab0361f229083e51ed7ec6c5270bc5c.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/ec/42b661db38fbf53d01e5f43e60f3e72e27d20b.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/60/17ac7d05ba51c23e719153fe872227642d59a3.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/96/d5dc0e8574e3bdfcd7329340eeb7bb9b6b87c0.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/c4/5eaa6cd80370c09176d26468b3019991772f00.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/04/663fc1335f018d20f5494226de543d65eec92f.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/29/de42766ea30b1b22020537cd01280f88260031.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/c8/7ba0aac1663725014a7ff0884e7c6fcafe7e40.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/c6/67bb13e948ae8f23b7a10e1dba7e5ebd3232a9.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/42/fa86e3abe84cb1310c7a82bc1f9904b0257c33.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/04/806f818b4045a25415233ddf6d153416737b08.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/cc/e35ead8640e6632f5fd489f30a864614992d1c.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/a9/ae2f275e5daffd09bada4aa4b8e9c3e859fcb1.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/20/f76613a2daf8e0fe2ef08fbe3b773431285adf.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/20/4002a427abd98fb619537cc701e9ccfb802542.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/e8/b7fce46c6c6399633fb0607fe50ebcba67a4d8.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/46/27f9a269725739999a1bd9f125852555037f23.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/21/d557e2d033da9e020ed70d611c79afef3a0836.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/eb/07a99babab9c97a6544de130952d678049990a.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/8c/30a731ea98194aa37e1a6a60d3b51b28144b9b.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/bf/5643420a294ad8cbcf32dd3c397771dcfc1038.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/59/b34a10be008b8999858a78130e4d77bfd1fe93.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/de/6062cf611d18c7d9611196a3d040095cdcdfc4.debug
2014-12-02-07:05:13  Missing debuginfo file: /usr/lib/debug/.build-id/54/4a7cbb0b3896c12f05b650b8fab35d5f3bfdf9.debug
2014-12-02-07:05:13  Generating backtrace
2014-12-02-07:05:18  Backtrace is generated and saved, 33210 bytes
2014-12-02-07:05:19  Looking for similar problems in bugzilla

As you can see, for whatever (unreported) reason abrt failed to extract the debuginfo rpm.  But doing so manually seems to work just fine:

$ yumdownloader v4l-utils-debuginfo-1.0.0-1.fc20.x86_64
Loaded plugins: auto-update-debuginfo, langpacks
v4l-utils-debuginfo-1.0.0-1.fc20.x86_64.rpm                 | 1.6 MB  00:00     
$ rpm2cpio < v4l-utils-debuginfo-1.0.0-1.fc20.x86_64.rpm | cpio -itv
drwxr-xr-x   5 root     root            0 Aug  3  2013 ./usr/lib/debug
drwxr-xr-x  26 root     root            0 Aug  3  2013 ./usr/lib/debug/.build-id
drwxr-xr-x   2 root     root            0 Aug  3  2013 ./usr/lib/debug/.build-id/00
lrwxrwxrwx   1 root     root           24 Aug  3  2013 ./usr/lib/debug/.build-id/00/b6d4480482db7b92a12c115350e6d7084f873e -> ../../../../bin/cx18-ctl
lrwxrwxrwx   1 root     root           28 Aug  3  2013 ./usr/lib/debug/.build-id/00/b6d4480482db7b92a12c115350e6d7084f873e.debug -> ../../usr/bin/cx18-ctl.debug
drwxr-xr-x   2 root     root            0 Aug  3  2013 ./usr/lib/debug/.build-id/1e
lrwxrwxrwx   1 root     root           31 Aug  3  2013 ./usr/lib/debug/.build-id/1e/d184883e32e85b478949caa3152c0da54d13c9 -> ../../../../bin/v4l2-sysfs-path
...

Related filesystems have enough space:

$ df -h /var/tmp/ /var/cache/abrt-di/
Filesystem                               Size  Used Avail Use% Mounted on
/dev/mapper/rootvol-fedora_var           8.4G  5.0G  3.0G  63% /var
/dev/mapper/rootvol-fedora_abrt--di--20  4.4G  608M  3.5G  15% /var/cache/abrt-di

Comment 1 Jakub Filak 2014-12-02 13:42:20 UTC
Thank you for the report! Isn't this report a dupe of bug #1036918?

If you have time to run the following commands

$ cd /var/cache/abrt-di
$ cpio -idu < /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio

, please do so and report the results here.

Comment 2 Brian J. Murrell 2014-12-02 14:13:09 UTC
(In reply to Jakub Filak from comment #1)
> Thank you for the report! Isn't this report a dupe of bug #1036918?

Wow.  I really mean wow.  You are right, this is a duplicate of my own bug #1036918 and filed exactly 1 year later, to the day.  What are the odds?
 
> If you have time to run the following commands
> 
> $ cd /var/cache/abrt-di
> $ cpio -idu <
> /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio

Do you really mean for me to run this an a non-root user as the "$" prompt on your commands suggest?

In any case,  /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio does not exist here.  But surely it's a temporary directory that is removed when abrt is done doing what it needs to with that dir, yes?

Comment 3 Jakub Filak 2014-12-02 14:39:43 UTC
(In reply to Brian J. Murrell from comment #2)
> (In reply to Jakub Filak from comment #1)
> > If you have time to run the following commands
> > 
> > $ cd /var/cache/abrt-di
> > $ cpio -idu <
> > /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio
> 
> Do you really mean for me to run this an a non-root user as the "$" prompt
> on your commands suggest?

I have to remember that the "$" prompt means non-root user.
Could you please run those commands as root?

> 
> In any case, 
> /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio does not
> exist here.  But surely it's a temporary directory that is removed when abrt
> is done doing what it needs to with that dir, yes?

The file should exist, because abrt removes it only after successful unpacking [1], but your log messages says ".*Can't extract files from.*" [2] (printed out in the else branch)

Do you use a temporary data cleaner? I am afraid that something removed /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio before abrt got a chance to unpack it.


1: https://github.com/abrt/libreport/blob/master/src/client-python/debuginfo.py#L132
2: https://github.com/abrt/libreport/blob/master/src/client-python/debuginfo.py#L134

Comment 4 Brian J. Murrell 2014-12-02 16:05:42 UTC
(In reply to Jakub Filak from comment #3)
> Could you please run those commands as root?

Well, I can't though because of...

> The file should exist, because abrt removes it only after successful
> unpacking [1],

Clearly that's not the case.  I added some debug (note the "ls -l" commands and their results below showing when the file is present):

Downloading 78.57Mb, installed size: 371.56Mb. Continue? 'YES'
Downloading (1 of 8) mate-control-center-debuginfo-1.8.3-1.fc20.x86_64.rpm: 100%
Extracting cpio from /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/mate-control-center-debuginfo-1.8.3-1.fc20.x86_64.rpm
# ls -l /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio
-rw-rw-r--. 1 abrt abrt 4035356 Dec  2 10:53 /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio

Caching files from unpacked.cpio made from mate-control-center-debuginfo-1.8.3-1.fc20.x86_64.rpm
running ['cpio', '-idu'] < /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio in /var/cache/abrt-di
out:  err: cpio: ./usr/lib/debug: Cannot change mode to rwxr-xr-x: Operation not permitted
cpio: ./usr/lib/debug/.build-id: Cannot change mode to rwxr-xr-x: Operation not permitted
cpio: ./usr/lib/debug/.build-id/48: Cannot mkdir: Permission denied
cpio: cannot make directory `./usr/lib/debug/.build-id/48': Permission denied
cpio: ../../../../bin/mate-display-properties: Cannot symlink to `./usr/lib/debug/.build-id/48/9fedcd40c723f5947fe6002a66ef3759f6eee5': No such file or directory
cpio: cannot make directory `./usr/lib/debug/.build-id/48': Permission denied
cpio: ../../usr/bin/mate-display-properties.debug: Cannot symlink to `./usr/lib/debug/.build-id/48/9fedcd40c723f5947fe6002a66ef3759f6eee5.debug': No such file or directory
cpio: ./usr/lib/debug/.build-id/71: Cannot mkdir: Permission denied
cpio: cannot make directory `./usr/lib/debug/.build-id/71': Permission denied
cpio: ../../../../lib64/libslab.so.0.0.0: Cannot symlink to `./usr/lib/debug/.build-id/71/8d2b95f4501fbf2308794739663e5aad2b12af': No such file or directory
cpio: cannot make directory `./usr/lib/debug/.build-id/71': Permission denied
cpio: ../../usr/lib64/libslab.so.0.0.0.debug: Cannot symlink to `./usr/lib/debug/.build-id/71/8d2b95f4501fbf2308794739663e5aad2b12af.debug': No such file or directory
cpio: ./usr/lib/debug/.build-id/cd: Cannot mkdir: Permission denied
cpio: cannot make directory `./usr/lib/debug/.build-id/cd': Permission denied
cpio: ../../../../sbin/mate-display-properties-install-systemwide: Cannot symlink to `./usr/lib/debug/.build-id/cd/10783aa2e44351bd9fc19245c9bb7efc7728de': No such file or directory
cpio: cannot make directory `./usr/lib/debug/.build-id/cd': Permission denied
cpio: ../../usr/sbin/mate-display-properties-install-systemwide.debug: Cannot symlink to `./usr/lib/debug/.build-id/cd/10783aa2e44351bd9fc19245c9bb7efc7728de.debug': No such file or directory
cpio: ./usr/lib/debug/.dwz: Cannot change mode to rwxr-xr-x: Operation not permitted
cpio: ./usr/lib/debug/.dwz/mate-control-center-1.8.3-1.fc20.x86_64: Cannot open: Permission denied
cpio: ./usr/lib/debug/usr: Cannot change mode to rwxr-xr-x: Operation not permitted
cpio: ./usr/lib/debug/usr/lib64: Cannot change mode to rwxr-xr-x: Operation not permitted
cpio: ./usr/lib/debug/usr/lib64/libmate-window-settings.so.1.0.0.debug: Cannot open: Permission denied
cpio: libmate-window-settings.so.1.0.0.debug: Cannot symlink to `./usr/lib/debug/usr/lib64/libmate-window-settings.so.1.debug': Permission denied
cpio: libmate-window-settings.so.1.0.0.debug: Cannot symlink to `./usr/lib/debug/usr/lib64/libmate-window-settings.so.debug': Permission denied
cpio: ./usr/lib/debug/usr/lib64/libslab.so.0.0.0.debug: Cannot open: Permission denied
cpio: libslab.so.0.0.0.debug: Cannot symlink to `./usr/lib/debug/usr/lib64/libslab.so.0.debug': Permission denied
cpio: libslab.so.0.0.0.debug: Cannot symlink to `./usr/lib/debug/usr/lib64/libslab.so.debug': Permission denied
cpio: ./usr/lib/debug/usr/lib64/window-manager-settings: Cannot mkdir: Permission denied
cpio: cannot make directory `./usr/lib/debug/usr/lib64/window-manager-settings': Permission denied
cpio: ./usr/lib/debug/usr/lib64/window-manager-settings/libmarco.so.debug: Cannot open: No such file or directory
7882 blocks
 rc: 2
Can't extract files from '/var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio'
# ls -l /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio
-rw-rw-r--. 1 abrt abrt 4035356 Dec  2 10:53 /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio

Unpacking failed, aborting download...
Missing debuginfo file: /usr/lib/debug/.build-id/4e/fe28aadfe0724926357e90872845dd5f714dcf.debug
Missing debuginfo file: /usr/lib/debug/.build-id/75/691a21b24ec54372155190638a47ff80672304.debug
Missing debuginfo file: /usr/lib/debug/.build-id/6d/9aabd27354e47d6b4c07f73c482691eee85ff5.debug
Missing debuginfo file: /usr/lib/debug/.build-id/0c/2881db5c93250f1576dab3d17864d22031ef70.debug
Missing debuginfo file: /usr/lib/debug/.build-id/ec/42b661db38fbf53d01e5f43e60f3e72e27d20b.debug
Missing debuginfo file: /usr/lib/debug/.build-id/66/b06d49f01cd823398fac7b75b688b81736a5b6.debug
Missing debuginfo file: /usr/lib/debug/.build-id/1c/d1edc030658c29de7035a6b83cf44443358ab3.debug
Missing debuginfo file: /usr/lib/debug/.build-id/04/663fc1335f018d20f5494226de543d65eec92f.debug
Missing debuginfo file: /usr/lib/debug/.build-id/60/cdd7495cbd92ed3b6f24c29162e08689ac9c85.debug
Missing debuginfo file: /usr/lib/debug/.build-id/ba/ef53aa03759d3a6a2b9e41115afd54dab36d2e.debug
Missing debuginfo file: /usr/lib/debug/.build-id/a9/ae2f275e5daffd09bada4aa4b8e9c3e859fcb1.debug
Missing debuginfo file: /usr/lib/debug/.build-id/21/10a7c97d990f5197bf27acbb8f11e090612077.debug
Missing debuginfo file: /usr/lib/debug/.build-id/c6/67bb13e948ae8f23b7a10e1dba7e5ebd3232a9.debug
Missing debuginfo file: /usr/lib/debug/.build-id/25/4e182d3b0d9190b51de85c3fa1dcecdb844c83.debug
Missing debuginfo file: /usr/lib/debug/.build-id/96/d5dc0e8574e3bdfcd7329340eeb7bb9b6b87c0.debug
Missing debuginfo file: /usr/lib/debug/.build-id/48/9fedcd40c723f5947fe6002a66ef3759f6eee5.debug
Missing debuginfo file: /usr/lib/debug/.build-id/13/900c8170b59a5239df1f79db31a51cfbaadf66.debug
Missing debuginfo file: /usr/lib/debug/.build-id/30/d104d9cc4c8d997c5ef50b470ebeb68eaee683.debug
Missing debuginfo file: /usr/lib/debug/.build-id/7c/eb1818a46ca58650559c9aebaea11cc91be0b6.debug
Missing debuginfo file: /usr/lib/debug/.build-id/f8/988e18c02dc9eb82d025a00065a5eee5318e9c.debug
Generating backtrace
Backtrace is generated and saved, 139403 bytes
Looking for similar problems in bugzilla
Searching for updates
No updates for this package found

And yet, with the window that the above was pasted from still open, the file is gone:

$ ls -l /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio
ls: cannot access /var/tmp/abrt-tmp-debuginfo-2014-12-02-10:52:55.11610/unpacked.cpio: No such file or directory

So as you can see, it was there, but removed by the time abrt was done processing that failure.

But the debug has revealed the actual problem.  The cpio fails due to EPERMs and ENOENTs.

My /var/cache/abrt-di/ looks like this:

$ ls -ldZ /var/cache/abrt-di/{,usr/{,lib/{,debug}}}
drwxrwxr-x. abrt abrt system_u:object_r:abrt_var_cache_t:s0 /var/cache/abrt-di/
drwxrwxr-x. abrt abrt unconfined_u:object_r:abrt_var_cache_t:s0 /var/cache/abrt-di/usr/
drwxrwxr-x. abrt abrt unconfined_u:object_r:abrt_var_cache_t:s0 /var/cache/abrt-di/usr/lib/
drwxrwxr-x. root root unconfined_u:object_r:abrt_var_cache_t:s0 /var/cache/abrt-di/usr/lib/debug

I'm guessing that last one, /var/cache/abrt-di/usr/lib/debug, should be owned by abrt.abrt, yes?

> Do you use a temporary data cleaner?

No.  And even if I did, it certainly would not be one that removed anything that was not clearly stale (i.e. many days old).  I have lots of stuff in /var/tmp/ that is way old:

drwx------.  3 root  root  4096 Jul 12  2013 systemd-private-dr11ns
drwx------.  3 root  root  4096 Jul 12  2013 systemd-private-cB4Hup
drwx------.  3 root  root  4096 Jul 12  2013 systemd-private-f6RyVm

> I am afraid that something removed
> /var/tmp/abrt-tmp-debuginfo-2014-12-02-06:59:48.5124/unpacked.cpio before
> abrt got a chance to unpack it.

Not at all, given the above analysis.  abrt is doing some clean-up somewhere.

Some better error reporting than just "it failed" would be nice here I think.  Knowing it was EPERM and ENOENT would have helped here without having to smatter debug into the tool.  Just providing the cpio stdout and stderr in the case of failure would likely be enough for most sysadmins to go figure out what the problem is.

Comment 5 Jakub Filak 2014-12-02 16:27:48 UTC
Thank you for the excellent analysis!

You are right, abrt removes the temporary directory here [1]. I apologize for misleading you.

Could you please clean /var/cache/abrt-di/, re-run reporting and report the results here?

abrt uses /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache to unpack the missing debuginfo packages into the abrt cache and the executable should have 's' flags for both owner and group. If everything works as expected, then all files within /var/cache/abrt-di/ are owned by abrt.abrt.


1: https://github.com/abrt/libreport/blob/master/src/client-python/debuginfo.py#L492

Comment 6 Brian J. Murrell 2014-12-02 17:14:09 UTC
(In reply to Jakub Filak from comment #5)
> Thank you for the excellent analysis!

NP.  Sometimes the easiest way to resolve uncertainties.
 
> Could you please clean /var/cache/abrt-di/, re-run reporting and report the
> results here?

I don't have to.  I did a chown -R abrt.abrt /var/cache/abrt-di/usr/lib/debug and now everything seems to be working.
 
> abrt uses /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache to unpack
> the missing debuginfo packages into the abrt cache and the executable should
> have 's' flags for both owner and group.

It does and the owner and group are abrt and abrt.  That is an important detail when dealing with set[gu]id bits.  :-)

> If everything works as expected,
> then all files within /var/cache/abrt-di/ are owned by abrt.abrt.

Right.  And my chown above achieved that and abrt seems to be installing the necessary debuginfos now.

Now if abrt would just manage the /var/cache/abrt-di cache like it purports to.  But that's another ticket.  :-)

Comment 7 Jakub Filak 2014-12-05 13:22:36 UTC
I like your idea of providing cpio output to sysadmins in the case of failure. We will forward cpio stdout and stderr to user visible temporary files and will remove them only if cpio succeeds in unpacking.

Comment 8 Matej Habrnal 2015-04-22 13:13:39 UTC
I've created a pull request which fixes this bug. https://github.com/abrt/libreport/pull/347

Comment 9 Fedora End Of Life 2015-05-29 13:26:10 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 10 Fedora Update System 2015-06-18 08:58:49 UTC
gnome-abrt-1.0.0-2.fc21,abrt-2.3.0-5.fc21,libreport-2.3.0-7.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/gnome-abrt-1.0.0-2.fc21,abrt-2.3.0-5.fc21,libreport-2.3.0-7.fc21

Comment 11 Fedora Update System 2015-06-21 00:32:54 UTC
Package gnome-abrt-1.0.0-2.fc21, abrt-2.3.0-6.fc21, libreport-2.3.0-8.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing gnome-abrt-1.0.0-2.fc21 abrt-2.3.0-6.fc21 libreport-2.3.0-8.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-10193/gnome-abrt-1.0.0-2.fc21,abrt-2.3.0-6.fc21,libreport-2.3.0-8.fc21
then log in and leave karma (feedback).

Comment 12 Fedora Update System 2015-06-23 09:08:10 UTC
Package gnome-abrt-1.0.0-3.fc21, abrt-2.3.0-7.fc21, libreport-2.3.0-8.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing gnome-abrt-1.0.0-3.fc21 abrt-2.3.0-7.fc21 libreport-2.3.0-8.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-10193/gnome-abrt-1.0.0-3.fc21,abrt-2.3.0-7.fc21,libreport-2.3.0-8.fc21
then log in and leave karma (feedback).

Comment 13 Fedora Update System 2015-06-30 20:20:04 UTC
gnome-abrt-1.0.0-3.fc21, abrt-2.3.0-7.fc21, libreport-2.3.0-8.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.