Bug 555205 - RFE: include "rpm -qf" output for all memory-mapped DSOs
Summary: RFE: include "rpm -qf" output for all memory-mapped DSOs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: abrt
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Denys Vlasenko
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 623062 (view as bug list)
Depends On:
Blocks: 655783
TreeView+ depends on / blocked
 
Reported: 2010-01-14 00:19 UTC by Dave Malcolm
Modified: 2011-04-26 16:19 UTC (History)
8 users (show)

Fixed In Version: abrt-2.0.1-2.fc15
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-04-26 16:19:14 UTC


Attachments (Terms of Use)

Description Dave Malcolm 2010-01-14 00:19:15 UTC
Description of problem:
At the end of each ABRT backtrace is a handy table of the shared libraries that are mapped into the process.

See e.g. attachment 383547 [details] to bug 555147, which has lines like this one:
0x00cc48e0  0x00ce1598  Yes (*)     /usr/lib/libibus.so.1

This is already useful (for example, I can see what compiled C modules have been imported into a Python process).

RFE: it would be nice if these lines had an additional column, giving the result of "rpm -qf" upon that shared library, so that the crash report automatically embedded this pertinent information.

Version-Release number of selected component (if applicable):
abrt 1.0.3

Comment 1 Dave Malcolm 2010-04-30 16:34:17 UTC
Even better would be if ABRT could report if any of the RPMs of the DSOs have updates available: that way if a DSO has a known crasher, we might be able to point the user at a fix.  For that case where a fix is still in testing, perhaps get him or her to test the fix in Bodhi.

Comment 2 Jiri Moskovcak 2011-01-05 15:49:02 UTC
I have a patch for this which parses /proc/PID/maps file and lists all libraries with their rpm package. The output format is:

library_path -> NVR (RPM vendor)

/usr/lib64/python2.7/site-packages/rpm/_rpmmodule.so -> rpm-python-4.8.1-5.fc14 (Fedora Project)
/usr/lib64/python2.7/site-packages/_dbus_glib_bindings.so -> dbus-python-0.83.0-7.fc14 (Fedora Project)
/usr/lib64/python2.7/lib-dynload/pyexpat.so -> python-2.7-8.fc14.1 (Fedora Project)
/usr/lib64/python2.7/site-packages/_dbus_bindings.so -> dbus-python-0.83.0-7.fc14 (Fedora Project)
/usr/lib64/python2.7/site-packages/gtk-2.0/gnomekeyring.so -> gnome-python2-gnomekeyring-2.32.0-1.fc14 (Fedora Project)
/usr/lib64/python2.7/site-packages/gtk-2.0/gtk/glade.so -> pygtk2-libglade-2.22.0-1.fc15 ((none))
/lib64/libnss_files-2.12.90.so -> glibc-2.12.90-21 (Fedora Project)

The automatic checking for the known crashes would require more coding, 
but is definitely a good idea worth implementing, so it's in our todo. The patch is not in git yet, so if you have any additional ideas/comments for dso->rpm mapping (liek better format) I'd be happy to change it..

Comment 3 Jiri Moskovcak 2011-01-05 15:50:01 UTC
one question do you prefer it to be in a separate attachment or in the comment feild?

Comment 4 Dave Malcolm 2011-01-05 18:03:42 UTC
Thanks for looking at this; it should be a significant improvement!

To slightly revise/clarify my original request, note that it's not just "libraries"; the "main" binary should be included in the list as well ("DSO" is the jargon for "Dynamic Shared Object").

Please add the %{arch} to the rpm output, giving
   %{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}
and
  %{VENDOR}

I may also be interesting to run "rpm --verify" on all rpms contributing a DSO to the process, and to add a warning next to any DSOs that have verification issues (though I don't know to what extent prelink screws us here).


I'm not sure what the best presentation of the data is, and I suspect that some experimentation/prototyping is worth doing (developers/triagers spend a great deal of time looking at these reports, so it's worth spending time to make them as good as possible).

I think the most pertinent information when looking at a process that has crashed at the machine code level (SIGSEGV, SIGABRT, etc) is: "which components are providing machine code (or data) within this process?"

There are two ways of seeing this, and I think I'd like both.

They are:
(i) a summary report, showing all RPMs that contributed DSOs to the crashing process, and a list of any DSOs aren't coming from an RPM.  I believe that this would typically be a relatively short amount of highly pertinent data, so this could be added as a comment to the bug, or in the initial report.  If there is more that one vendor, then I think the RPMs should be grouped by vendor.

(ii) the full details: all mapped DSOs, together with the RPM NVR.ARCH, and vendor for each DSO.    I believe that this would typically be a large amount of detailed data, so it should probably be done in an attachment form.  I think I'd like to see the original request, which was for the data to be merged into the "maps" table, included inline in the original attachment, as part of the memory map of the process.  This already contains a useful table of details, and it would be easier on developers/triagers to capture all of this information in one place.

It would two additional columns, listing the rpms N-V-R.A, and the "vendor" field as well (I really like your idea of including the vendor!)

(What language are you using to generate the reports?  I can help if you're doing it in Python :) )


It's definitely interesting to see if a DSO isn't coming from an rpm, or if an rpm has a non-standard vendor.   One other thought on this:
I wonder what happens if a user has rebuilt a package locally.  I don't know if the Vendor field captures this, but it may be worth looking at the "Build Host" field, and noting if the host is non-standard; this suggests that someone has rebuilt a package locally.  For Fedora, these typically match
  ".+\.fedoraproject\.org"
and for RHEL these match:
 ".+\.redhat.com" (CentOS etc would have different values here).

(Each "Vendor" could have a regex of expected build hosts.  But this may be redundant; I suspect that checking the Vendor field is enough here.  Actually capturing the "Build Host" field may be regarded by some users as a change in privacy expectations)


I'd still love to see the "pending updates" idea from comment #1; should we break that out into a separate RFE?

(Note that it's something of a misfeature to have multiple comments in the initial report; ideally the initial report should somehow be done as a single transaction with the bug tracking tool, to minimize/consolidate the notification emails it sends out - but I appreciate that it's hard to do that with Bugzilla).

Comment 5 Nikola Pajkovsky 2011-01-06 10:26:10 UTC
(In reply to comment #4)
> (Note that it's something of a misfeature to have multiple comments in the
> initial report; ideally the initial report should somehow be done as a single
> transaction with the bug tracking tool, to minimize/consolidate the
> notification emails it sends out - but I appreciate that it's hard to do that
> with Bugzilla).

Bugzilla is insanely stupid via xmlrpc. We talk about that too and it's on my todo list.

Comment 6 Jiri Moskovcak 2011-01-26 14:31:19 UTC
(In reply to comment #4)
> Thanks for looking at this; it should be a significant improvement!
> 
> To slightly revise/clarify my original request, note that it's not just
> "libraries"; the "main" binary should be included in the list as well ("DSO" is
> the jargon for "Dynamic Shared Object").
> 
> Please add the %{arch} to the rpm output, giving
>    %{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}
> and
>   %{VENDOR}
> 
> I may also be interesting to run "rpm --verify" on all rpms contributing a DSO
> to the process, and to add a warning next to any DSOs that have verification
> issues (though I don't know to what extent prelink screws us here).
> 

- we used to have this feature, but it was claimed to be dangerous, because it runs prelink (to un-prelink bins) and it might screw things a lot if we do some mistake so we didn't get an exception in SELinux policy for that - maybe they will allow us at least call rpm --verify as a separate process to check the packages (we used to do this using librpm)

> 
> I'm not sure what the best presentation of the data is, and I suspect that some
> experimentation/prototyping is worth doing (developers/triagers spend a great
> deal of time looking at these reports, so it's worth spending time to make them
> as good as possible).

- I send an email about this fedora-{devel, triage} let's hope it wont turn into a flame against ABRT as usual :)

> 
> I think the most pertinent information when looking at a process that has
> crashed at the machine code level (SIGSEGV, SIGABRT, etc) is: "which components
> are providing machine code (or data) within this process?"
> 
> There are two ways of seeing this, and I think I'd like both.
> 
> They are:
> (i) a summary report, showing all RPMs that contributed DSOs to the crashing
> process, and a list of any DSOs aren't coming from an RPM.  I believe that this
> would typically be a relatively short amount of highly pertinent data, so this
> could be added as a comment to the bug, or in the initial report.  If there is
> more that one vendor, then I think the RPMs should be grouped by vendor.
> 
> (ii) the full details: all mapped DSOs, together with the RPM NVR.ARCH, and
> vendor for each DSO.    I believe that this would typically be a large amount
> of detailed data, so it should probably be done in an attachment form.  I think
> I'd like to see the original request, which was for the data to be merged into
> the "maps" table, included inline in the original attachment, as part of the
> memory map of the process.  This already contains a useful table of details,
> and it would be easier on developers/triagers to capture all of this
> information in one place.
> 
> It would two additional columns, listing the rpms N-V-R.A, and the "vendor"
> field as well (I really like your idea of including the vendor!)
> 
> (What language are you using to generate the reports?  I can help if you're
> doing it in Python :) )
> 

- actually in the latest git it can be any language, there is a list of actions to be run when the crash happens and these action can be any script/executable specified in the config file
- by default the script/executable is called with one option which is the directory where the data about crash are saved
- so my idea is to save the file maps from /proc/<PID>/ of the crashed process and then call the script to process it and all we have to do in ABRT is to add line to conf file:

EVENT=post-create analyzer=CCpp         get-nevra-for-dsos

which means run get-nevra-for-dsos right after the new crash is detected (right after the code processing /proc/<PID> is done)

- so if you'd like help, you're more than welcome ;)

> 
> It's definitely interesting to see if a DSO isn't coming from an rpm, or if an
> rpm has a non-standard vendor.   One other thought on this:
> I wonder what happens if a user has rebuilt a package locally.  I don't know if
> the Vendor field captures this, but it may be worth looking at the "Build Host"
> field, and noting if the host is non-standard; this suggests that someone has
> rebuilt a package locally.  For Fedora, these typically match
>   ".+\.fedoraproject\.org"
> and for RHEL these match:
>  ".+\.redhat.com" (CentOS etc would have different values here).
> 
> (Each "Vendor" could have a regex of expected build hosts.  But this may be
> redundant; I suspect that checking the Vendor field is enough here.  Actually
> capturing the "Build Host" field may be regarded by some users as a change in
> privacy expectations)
> 
> 

- for local build the vendor is (typically) empty and it's definitely not signed with Fedora/RHEL gpg key, so we can detect such rpms

> I'd still love to see the "pending updates" idea from comment #1; should we
> break that out into a separate RFE?
> 

- this already is on our TODO list
- hm, is there a library to talk to bodhi? it it is we can implement this also as a separate app, which can be run during the event call analyze and ask user to update even before he press "report"...

Comment 7 Jiri Moskovcak 2011-03-27 16:54:17 UTC
*** Bug 623062 has been marked as a duplicate of this bug. ***

Comment 8 Fedora Update System 2011-03-27 17:35:59 UTC
abrt-2.0.0-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/abrt-2.0.0-1.fc15

Comment 9 Fedora Update System 2011-03-29 20:01:20 UTC
abrt-2.0.0-2.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/abrt-2.0.0-2.fc15

Comment 10 Fedora Update System 2011-03-30 02:32:08 UTC
Package abrt-2.0.0-2.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing abrt-2.0.0-2.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/abrt-2.0.0-2.fc15
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2011-03-30 16:48:26 UTC
abrt-2.0.0-3.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/abrt-2.0.0-3.fc15

Comment 12 Fedora Update System 2011-04-04 14:08:08 UTC
abrt-2.0.0-4.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/abrt-2.0.0-4.fc15

Comment 13 Fedora Update System 2011-04-15 15:08:53 UTC
abrt-2.0.0-5.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/abrt-2.0.0-5.fc15

Comment 14 Fedora Update System 2011-04-20 13:29:42 UTC
abrt-2.0.1-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/abrt-2.0.1-1.fc15

Comment 15 Fedora Update System 2011-04-21 16:44:23 UTC
abrt-2.0.1-2.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/abrt-2.0.1-2.fc15

Comment 16 Fedora Update System 2011-04-26 16:11:49 UTC
abrt-2.0.1-2.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.