Bug 448586

Summary: Firefox 3 crashes Xorg at picture.c:1600
Product: Red Hat Enterprise Linux 5 Reporter: Richard Ryder <rryder>
Component: xorg-x11-serverAssignee: Adam Jackson <ajax>
Status: CLOSED ERRATA QA Contact: desktop-bugs <desktop-bugs>
Severity: high Docs Contact:
Priority: urgent    
Version: 5.2CC: bugs+fedora, cmeadors, john, mgordon, peak, shillman, xgl-maint
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 11:42:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Xorg core generated by gdb gcore command
none
dmesg from sosreport
none
xorg.conf from sosreport
none
Xorg.0.log from sosreport
none
test case none

Description Richard Ryder 2008-05-27 18:16:36 UTC
Description of problem:

Visiting http://developer.pidgin.im/ticket/4986 with Firefox 3.0b5 will cause
Xorg to crash.


Version-Release number of selected component (if applicable):
RHEL5.2 i386 and x86_64
kernel-2.6.18-92.el5
xorg-x11-server-Xorg-1.1.1-48.41.el5
firefox-3.0-0.beta5.6.el5

How reproducible:
100%.  Tested on i386 and x86_64

Steps to Reproduce:
1.Install RHEL5.2
2.Open Firefox and vist http://developer.pidgin.im/ticket/4986
3.As the page is loading Xorg will crash
  
Actual results:
Xorg crashes

Expected results:
Xorg doesn't crash. 

Additional info:
Core and sosreport attached.  No NVidia driver, firefox plugins, or extensions
are installed.  Video driver is i810, also duplicated on nv.  Does not cause
crash on F9.

Program terminated with signal 11, Segmentation fault.
#0  0x08147845 in ValidateOnePicture (pPicture=0x963a730) at picture.c:1600
1600            (*ps->ValidatePicture) (pPicture, pPicture->stateChanges);
(gdb) thread apply all bt

Thread 1 (process 2887):
#0  0x08147845 in ValidateOnePicture (pPicture=0x963a730) at picture.c:1600
#1  0x08147881 in ValidatePicture (pPicture=0x963a730) at picture.c:1609
#2  0x08147941 in CompositePicture (op=12 '\f', pSrc=0x96437b8, pMask=0x963a730,
pDst=0x963a588, xSrc=0, ySrc=0, xMask=0, yMask=0, xDst=0, yDst=0, width=1,
height=1) at picture.c:1782
#3  0x08146d68 in miTrapezoids (op=12 '\f', pSrc=0x96437b8, pDst=0x963a588,
maskFormat=<value optimized out>, xSrc=1, ySrc=0, ntrap=0, traps=0x9711294) at
mitrap.c:175
#4  0x0815e1d4 in cwTrapezoids (op=12 '\f', pSrcPicture=0x96437b8,
pDstPicture=0x963a588, maskFormat=0x953b448, xSrc=1, ySrc=0, ntrap=3,
traps=0x971121c) at cw_render.c:365
#5  0x08147cb3 in CompositeTrapezoids (op=12 '\f', pSrc=0x96437b8,
pDst=0x963a588, maskFormat=0x953b448, xSrc=1, ySrc=0, ntrap=3, traps=0x971121c)
at picture.c:1848
#6  0x0814e934 in ProcRenderTrapezoids (client=0x9643668) at render.c:820
#7  0x0814abb5 in ProcRenderDispatch (client=0x7) at render.c:2001
#8  0x0808815a in Dispatch () at dispatch.c:459
#9  0x0806fab5 in main (argc=10, argv=0xbfce6f44, envp=0xff53f0) at main.c:447
(gdb)

Comment 1 Richard Ryder 2008-05-27 18:16:36 UTC
Created attachment 306809 [details]
Xorg core generated by gdb gcore command

Comment 4 Matěj Cepl 2008-06-06 13:54:25 UTC
Created attachment 308531 [details]
dmesg from sosreport

Comment 5 Matěj Cepl 2008-06-06 13:54:34 UTC
Created attachment 308532 [details]
xorg.conf from sosreport

Comment 6 Matěj Cepl 2008-06-06 13:54:45 UTC
Created attachment 308533 [details]
Xorg.0.log from sosreport

Comment 7 Juliano F. Ravasi 2008-07-13 01:28:16 UTC
Please, could you confirm if this URL causes the same crash?
http://en.wikipedia.org/wiki/Special:Allmessages

I'm having similar crashes in Fedora 8 with the above URL, with intel
(open-source) and nvidia (closed-source) drivers, but with a different backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x0044817f in fbRasterizeEdges (buf=0x8695098, bpp=8, width=436, stride=109,
l=0xbffaff54,
    r=0xbffaff2c, t=-2147481464, b=28571510) at fbedge.c:171
171                     ap[lxi] = clip255 (ap[lxi] + N_X_FRAC(8) - lxs);
(gdb) backtrace
#0  0x0044817f in fbRasterizeEdges (buf=0x8695098, bpp=8, width=436, stride=109,
l=0xbffaff54,
    r=0xbffaff2c, t=-2147481464, b=28571510) at fbedge.c:171
#1  0x00440a0f in fbRasterizeTrapezoid (pPicture=0x85fe368, trap=0x864e978,
    x_off=<value optimized out>, y_off=0) at fbtrap.c:143
#2  0x0035534a in _nv000753X () from /usr/lib/xorg/modules/drivers//nvidia_drv.so

Also bug 453607 seems related.

Comment 8 Matěj Cepl 2008-07-15 16:47:05 UTC
This seems to be a duplicate of bug 455209 (which is in Fedora 8).

Comment 9 Matěj Cepl 2008-07-15 16:47:43 UTC
*** Bug 453607 has been marked as a duplicate of this bug. ***

Comment 10 John Perkins 2008-08-07 14:48:23 UTC
We are experiencing this particular error at our site as well.  The wikipedia link above causes the same crash for us.

backtrace informatino varies slightly based on whether teh Composite X extension is enabled.   With Composite enabled, the backgrace looks as such:

Program received signal SIGSEGV, Segmentation fault.
0x08147898 in PictureMatchFormat ()
(gdb) where
#0  0x08147898 in PictureMatchFormat ()
#1  0x081478e1 in ValidatePicture ()
#2  0x081479a1 in CompositePicture ()
#3  0x08146dc8 in miTrapezoids ()
#4  0x00a61837 in _nv002361X ()
   from /usr/lib/xorg/modules/drivers/nvidia_drv.so
#5  0x0000000c in ?? ()
#6  0x088b67a8 in ?? ()
#7  0x08904908 in ?? ()
#8  0x08824798 in ?? ()
#9  0x00000000 in ?? ()
(gdb)

Disabling Composite changes the backtrace slightly:

Program received signal SIGSEGV, Segmentation fault.
0x08147978 in CompositePicture ()
(gdb) where
#0  0x08147978 in CompositePicture ()
#1  0x08146dc8 in miTrapezoids ()
#2  0x00f82837 in _nv002361X ()
   from /usr/lib/xorg/modules/drivers/nvidia_drv.so
#3  0x0000000c in ?? ()
#4  0x09f2aca8 in ?? ()
#5  0x09efdbd0 in ?? ()
#6  0x09e8f2f0 in ?? ()
#7  0x00000000 in ?? ()
(gdb)

I tried using an X11R7.3 X-server, with Composite extensions enabled...this bug did not occur when running with the X11R7.3 X-server.

All our tests were done using the "nvidia" proprietary video device driver.

Comment 11 Pavel Kankovsky 2009-01-30 01:04:01 UTC
The problem lies in the negative (and very large) value of parameter "t" when fbRasterizeEdges() is called. The function draws outside the allocated buffer ("buf") and causes a lot of collateral damage.

The sources of negative values of "t" is RenderSampleCeilY() called from fbRasterizeTrapezoid(). Something overflows and yields a negative result when it is called with a large positive value of its first parameter "y" (afaik >= 2147481463) and this happens when the client asks the server to draw a strange trapezoid very close the edge of the coordinate space:

(gdb) bt 1
#0  fbRasterizeTrapezoid (pPicture=0x94d5dc8, trap=0x9509b1c, x_off=0, y_off=0)
    at fbtrap.c:137
(gdb) p *trap
$36 = {top = 2147483647, bottom = 2147483647, left = {p1 = {x = 0, y = 0},
      p2 = {x = 0, y = 2147483647}}, right = {p1 = {x = 65536, y = 2147483647},
      p2 = {x = 0, y = 2147483647}}}
(gdb) p t
$37 = -2147481464
(gdb) print RenderSampleCeilY(2147483647, 8)
$38 = -2147481464

(And no, this not a duplicate of bug 455209.)

Comment 12 Pavel Kankovsky 2009-02-02 18:46:10 UTC
Created attachment 330662 [details]
test case

$ gcc trapezoid_of_death.c -lX11 -lXext -lXrender
$ DISPLAY=[the display you want to kill] ./a.out

(It is necessary to send two trapezoids, one with saner top/bottom, to get past a check in miTrapezoids().)

Comment 13 RHEL Program Management 2009-03-11 15:39:47 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 15 Adam Jackson 2009-04-22 18:15:02 UTC
Fixing component.

Comment 16 Adam Jackson 2009-04-22 18:44:45 UTC
Built as xorg-x11-server-1.1.1-48.53.el5

MODIFIED

Comment 18 Mark Gordon 2009-06-22 19:34:26 UTC
Fix verified using the 20090608.2 tree.

Comment 20 errata-xmlrpc 2009-09-02 11:42:21 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1373.html