Bug 471433

Thank you for taking the time to report this bug. This bug report isn't very
useful because it doesn't describe the bug well. If you have time and can
still reproduce the bug, please read
http://fedoraproject.org/wiki/BugsAndFeatureRequests and add a more useful
description to this bug.

You'll also need to add a stack trace; please make sure you have debuginfo
packages installed and see http://fedoraproject.org/wiki/StackTraces
for more information about getting a useful stack trace.
Thank you.

Jon,
I posted exact version 

evince-2.22.2-1.fc9.x86_64
poppler-0.8.7-1.fc9.x86_64

and exact point where it fails.

evince[12498]: segfault at 8 ip 0000003337b1dae8 sp 00007fff24f0f5d0 error 4 in
libpoppler.so.3.0.0[3337a00000+1ab000]

It crashes randomly, the situation when a bug always reproducible is rather seldom.

What I would expect from a package mantainer who you are,
is either:
1. take the trace, apply debug procedure procedure you put a link to. I do not have more detailed error message. I copied everything I got.

2. Because this crash is evidently a critical security hole
(a special crafted .pdf can crash the evince program and  execute arbitrary code)
provide a modified package, which would print complete trace instead of a small one I copied and pasted.

Don't expect users to do your package maintainer job.

And by the way - I would expect package maintainer to take actions much faster.
This is a critical security hole, allow arbitrary code to be executed.

The original bug report was posted 2008-11-13, now it is 2009-02-13. What did you do all this time? 

Do not expect much help from the people, when you take so lazy approach.

I would consider to dig more into this bug, but taking into account that even having a detailed trace you would very likely to do nothing with it in months or even years - I consider this a waste of my time.

Thank you for reporting this bug and responding to the needinfo request. I am not a package maintainer. I am a volunteer doing bug triage work.

In order to properly diagnose this crasher bug we need a full stack trace. To properly get this stack trace see http://fedoraproject.org/wiki/StackTraces for more information. Without this stack trace nor a consistent way to reproduce the bug we do not have enough information. I was unable to reproduce this bug with the information you gave that is why I am resending this request.

Jon, I do not remember what pdf I was viewing 3 months ago to reproduce the problem.
Now I just do not have have more information.
You came too late. If it were 2008-11-13 now, when I posted this critical security hole in evince - I would provide .pdf and everything. 

What do you expect from me now - to do something with this:
evince[12498]: segfault at 8 ip 0000003337b1dae8 sp 00007fff24f0f5d0 error 4 in
libpoppler.so.3.0.0[3337a00000+1ab000]

I am in no better position than you to look at this crash.
Install -debuginfo for 
evince-2.22.2-1.fc9.x86_64
poppler-0.8.7-1.fc9.x86_64
standard packages from fedora 9.

run a trace. You probably do this regulary. If you have a problem -
contact your collegues, they would help if you have a difficulty.

Jon, just understand, because you came WAY TOO LATE I am in no better position than you to do stack trace.

(In reply to comment #5)
> Jon, I do not remember what pdf I was viewing 3 months ago to reproduce the
> problem.

According to your original post it is this pdf: http://java.sun.com/developer/Books/javaprogramming/corejavavolumeone/cjch6.pdf

I tried reproducing your bug with this pdf under Fedora 10 and evince-2.24.2-1.fc10.i386 but was unable to make evince crash using your description.

> Now I just do not have have more information.

Without the requested information then we will be unable to diagnose this issue further. Setting status to "CLOSED: INSUFFICIENT_DATA".  If you still experience this problem after updating to our latest Fedora release and can provide the information previously requested, please feel free to reopen the bug report.

Thank you in advance.

Mal, as Jon mentioned, he's a volunteer, doing this to try and help out Fedora.
Using such a confrontational tone isn't helpful and just causes people hurt;
Jon came on IRC asking if people thought he'd done anything wrong to get such a
vitriolic response.

Yes, he came to this bug late. We're sorry about that. The Bugzappers team is
quite small and trying to catch up with a backlog. Ideally, there should have
always been enough people to cover triaging all bugs as they're filed, but as
there weren't, the best the team can do is try to catch up with these bugs now
so they aren't lost forever. This is all Jon was trying to do, and he doesn't
deserve abuse for it.

Your last comment neatly illustrates the reason Jon is asking for more
information in the first place: if you, the reporter, can't now reproduce the
bug based on the information you first provided, how do you expect anyone else
to do so? We recognize that it's important to fix crasher bugs like this, but
on a practical level, they can't be fixed if they can't be reproduced, unless
the required level of detailed information is provided. You did provide all the
information the app gives you by default, and we thank you for that, but in
this case, it's not enough information.

Ideally the Bugzappers would be able to reproduce all bugs and add the
necessary extra information for maintainers, but practically speaking that is
not possible. Quite often, bugs happen only in a specific set of circumstances
- certain hardware, certain desktop settings - that it's impossible for the
Bugzappers to reproduce, so they cannot actually trigger the bugs. Even if they
could, this would take too much time for the current Bugzappers team to do
successfully on all bugs triaged. So at present we have to rely on reporters to
provide this additional information.

I did, as a matter of interest, try to reproduce this bug on my system - I only
have Rawhide installed, not 9 or 10, but I can't trigger this bug using the PDF
you linked in the initial report. I loaded it and messed with Evince for ten
minutes, and could not cause it to crash.

Thanks for the report, and we're sorry it may not be possible to fix it. But
please understand that the Bugzappers are a volunteer group doing the best they
can to help improve Fedora in their spare time. Unload on me, if you like - Red
Hat is paying me for it. But please don't be unnecessarily aggressive towards
guys who are just giving their time to try and help the project.

Jon has closed the bug, because we just can't fix it without the information he asked for. It can, of course, be re-opened in future if the required information becomes available.

Adam Williamson
Fedora QA Community Monkey

Adam,
I agree I was too offensive. Sorry, Jon for this. I really did not want to offence you.

In the past I was very polite and was trying to help Fedora package maintainers as much as I can. But in the last few years the response became extremely slow, even for critical bugs. Then, even if detailed information is posted - the bug often go nowhere anyway.

So I mostly stopped posting bug reports which are either hard to reproduce or not critical for me personally. Because most of them go nowhere anyway.

And this specific evince bug #471433 fitted exactly the pattern. Critical security bug posted - then in 3-4 months first initial response. 

So I overreacted and responded rudely. 
It was just a response pattern, on which I responded offencively. It is not you Jon. Sorry for this.
Typically I (and I think many other people) just ignore bugzilla notifications, when they come too late.

Vladislav

P.S. Adam, I do not know whether this is feasible currently at RedHat,
but it is critically important to respond on bug reports very fast.
Even in 1-2 days people typically forget the details, find some workaround or start using other software. Even 2 days delay make the reporters much less cooperative.

Thanks, Mal.

I agree that it's definitely frustrating when reports don't get responded to by the developers - it's probably in the nature of things that this will happen for some, but of course it should be minimized as much as possible. That's what the Bugzappers project is trying to help do, by providing front-line response to most bugs as they arrive. But it's a big task and there probably still aren't enough people to do them all :(. That's partly what I've been brought in to try and improve. So, yes, it's a definite goal to have more and more active Bugzappers with the necessary skills and tools to get all bugs properly handled.

There are a few important issues raised by this post. To be honest it's not clear to me that it actually is a security issue, though that may just be my own inexperience with code. As I understand it, not all segfaults are exploitable - only buffer overflows are. Is there an indication that this is definitely an exploitable crash that I'm just too dumb to understand? :)

In any case, it does raise the question of what can you do to flag a bug report as a security issue to make sure the security team looks at it as a priority. I'm not sure there's a good process for that, it's something I'm going to look into.

I'm also suggesting the Bugzappers group adopt a uniform signature to be used whenever doing stuff on Bugzilla, so it's clear that they're volunteer triagers and not actually the maintainers or paid RH staff. That might help avoid misunderstandings like this one.

Adam,
I agree with your points regarding improvement with response speed and signatures.

evince is of critically security importance because it is attached to firefox file types, thus any evince bug can be easily exploited by the attacker, all he need is to create a web site with special content.

Historically there were many bugs in evince/gnostscript, many of of them fixed,
some of them were reported by me personally with a testcases.

On 2008-11-13 I found one more evince crash, was not sure whether it was exploitable and posted as regular bug. It might be exploitable, but not sure.

Then within a month or two I started getting regular crashed of evince/poppler on various .pdf documents. Some of them were exactly at the memory address reported on 2008-11-13 bug report, other happened randomly, at various addresses, in different circumstances, some required no clicking at all.

Then I started to think there are a number of issues in poppler/evince and some of them have to be exploitable. This is why I think this is a security issue. Critical issue, because any computer with web browser expose evince to the world.

The reason I did not post an update to the report - it is hard to reproduce the crashes. Back then I was reading hundreds .pdf documents a day, some crashed, some documents were corrupted, and very hard to figure out which one crashed evince and after what of my actions.

The only thing I can easily get - a line in dmesg about crash at the end of the day.
Those where I was able to provide easy test case - were posted separately.


What does this problem indicate - currently fedora misses a number 
of bug reports because they are hard to report and reproduce.
 
Historically there was exactly this problem in linux kernel, when ksymoops were hard to interpret and were useful to the developers only when the reporter perform some analysis on them. So most of the kernel panic reports were actually useless.

The problem was solved around 2005 by integrating to the kernel 
trace analysis, and now any panic message is sufficient by itself for the developer to be useful. All the reported need - copy & paste.

Similar approach should be applied to userspace programs.
Critical part of infrastructure (apache, ssh) and programs known to be buggy and crash regulary (firefox, evince, bluetooth, php, etc) must be installed with -debuginfo package AND integrated with some userspace stack trace analyzis which would print trace to dmesg instead of  this:

evince[12498]: segfault at 8 ip 0000003337b1dae8 sp 00007fff24f0f5d0 error 4 in
libpoppler.so.3.0.0[3337a00000+1ab000]

a complete stack trace. 
I am not sure this is necessary for all program,
but this has to be done to all infrastructure and all programs related to web browser integration. 

You will get the same result as happened in kernel: from most of the reports to be useless to most of the reports to be very useful.
And you already have all necessary for this, just slight adjustment in traces printed and installation -debuginfo by default for all the programs known to be buggy.

Bugzilla isn't meant for extended discussions, so I'll continue this by email.

Created attachment 332249 [details]
A script for coredumps analysis

(In reply to comment #10)
> Then within a month or two I started getting regular crashed of evince/poppler
> on various .pdf documents. Some of them were exactly at the memory address
> reported on 2008-11-13 bug report, other happened randomly, at various
> addresses, in different circumstances, some required no clicking at all.

Actually, let me chime in as well -- I am a bug triager as well, but also an employee of Red Hat. If you have these random crashes of an application, it would be useful to allow your system to generate coredumps and then when the system crashes send us FULL backtrace from gdb. Let me explain.

1) First of all, Fedora (as many other Linux distributions) have switched off per default generation of core dump files (most users have no clue what to do with them and for them they are just waste of space on disk).

You can switch the generation of core dumps per gnome-terminal session by running

ulimit -c unlimited

but it is incovenient to do it for every gnome-terminal and it doesn't work for applications run from the start menu or a toolbar.

So, if you go (as root) to /etc/security and in the file limits.conf add these two lines (probably somewhere in the middle, but it doesn't matter really that much where):

*               hard    core            unlimited
*               soft    core            unlimited

Then in the file /etc/profile comment out line saying

ulimit -S -c 0 > /dev/null 2>&1

so that it looks this way

#ulimit -S -c 0 > /dev/null 2>&1

Now, you have to reboot computer (and it might be wise to run as root from to time

find / -regex '.*\/core\.[0-9]*$'

to find out whether you don't have droppings on your disk).

2) install -debuginfo packages. Using debuginfo-install (from yum-utils package) run (as root):

debuginfo-install evince

to get all debuginfo packages.

3) When you get an evince crashing, find the coredump file on your disk (most likely it is in your $HOME directory) and then run the attached script gbt against it like

gbt evince coredump-file-name

Attach the generated file evince-backtrace-<pid-number>.txt to this bug and reopen it.

Thank you.

Thanks. done.
It would also be very convenient to put this info (how to turn on core dumps) to some FAQ or, better, add configuration option of the system.
 
I also upgraded evince to evince-2.25.90-1.fc9.x86_64 and poppler to poppler-0.10.3-2.fc9.x86_64
(the version from fedora debug).

I will check whether the core dumps would show up.