Bug 282661 - Reproducible crash in libvnc.so [crashes Xorg]
Reproducible crash in libvnc.so [crashes Xorg]
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: vnc (Show other bugs)
5.1
All Linux
medium Severity high
: beta
: ---
Assigned To: Adam Tkac
:
: 240379 251165 (view as bug list)
Depends On:
Blocks: 391501
  Show dependency treegraph
 
Reported: 2007-09-07 12:21 EDT by Aleksey Nogin
Modified: 2013-04-30 19:37 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-20 16:15:58 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch to fix crash in vnc.so module (1.07 KB, patch)
2007-12-06 22:12 EST, Tom Sightler
no flags Details | Diff
Patch which validates framebuffer bounds before framebuffer dump (2.11 KB, patch)
2008-03-26 07:56 EDT, Adam Tkac
no flags Details | Diff

  None (edit)
Description Aleksey Nogin 2007-09-07 12:21:03 EDT
Description of problem:

I have the vnc module enabled in my xorg.conf. If I use VNC from the beginning
of an X session, everything works fine. If I use X on the console for a while
and then connect to that same session via VNC, the session crashes...

Version-Release number of selected component (if applicable):

vnc-server-4.1.2-9.el5
xorg-x11-server-Xorg-1.1.1-48.13.0.1.el5

How reproducible:
Close to 100%. I have been doing this on 3 machines - a work machine with RHEL5
and two home machines with CentOS 5 and see this on all 3 (even though xorg.conf
are all somewhat different).

Steps to Reproduce:
1. Include the vnc module in xorg.conf
2. Restart X and use in on the console for a while
3. Make sure the right VNC parameters are enabled to allow VNC logins (use
vncconfig, if necessary)
4. Open a VNC connection to the X session.
  
Actual results:
X.org crashes. Here is the crash report in Xorg.0.log on the RHEL5 machine with
all updates:

Backtrace:
0: /usr/bin/Xorg(xf86SigHandler+0x81) [0x80bc1b1]
1: [0x302420]
2: /lib/libc.so.6(memcpy+0x1c) [0x4af9ca6c]
3: /usr/lib/xorg/modules/libfb.so(fbBlt+0x90e) [0x3fb39e]
4: /usr/lib/xorg/modules/libfb.so(fbBltStip+0x73) [0x3fb8c3]
5: /usr/lib/xorg/modules/libfb.so(fbGetImage+0x23a) [0x40b11a]
6: /usr/lib/xorg/modules/libxaa.so [0x1be2a2]
7: /usr/bin/Xorg [0x81088e8]
8: /usr/bin/Xorg [0x815b2a3]
9: /usr/bin/Xorg [0x811d4fd]
10:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN14XserverDesktop10grabRegionERKN3rfb6RegionE+0x12a)
[0x291c9a]
11:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN3rfb11VNCServerST11checkUpdateEv+0x1b7)
[0x29d457]
12:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN3rfb16VNCSConnectionST22writeFramebufferUpdateEv+0x5c)
[0x2ac8ec]
13:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN3rfb16VNCSConnectionST24framebufferUpdateRequestERKNS_4RectEb+0xbf)
[0x2ad1ef]
14:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN3rfb10SMsgReader28readFramebufferUpdateRequestEv+0x14f)
[0x2b3bcf]
15:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN3rfb12SMsgReaderV37readMsgEv+0x14a)
[0x2a83da]
16:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN3rfb11SConnection10processMsgEv+0x14f)
[0x2a7d7f]
17:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN3rfb16VNCSConnectionST15processMessagesEv+0x48)
[0x2abf18]
18:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN3rfb11VNCServerST18processSocketEventEPN7network6SocketE+0x3f)
[0x29d20f]
19:
/usr/lib/xorg/modules/extensions/libvnc.so(_ZN14XserverDesktop13wakeupHandlerEP6fd_seti+0xd4)
[0x291154]
20: /usr/lib/xorg/modules/extensions/libvnc.so [0x2893ec]
21: /usr/bin/Xorg(WakeupHandler+0x59) [0x808c169]
22: /usr/bin/Xorg(WaitForSomething+0x1b9) [0x81a0849]
23: /usr/bin/Xorg(Dispatch+0x8d) [0x8087f9d]
24: /usr/bin/Xorg(main+0x485) [0x806fa35]
25: /lib/libc.so.6(__libc_start_main+0xdc) [0x4af44dec]
26: /usr/bin/Xorg(FontFileCompleteXLFD+0x1e9) [0x806ed71]

Fatal server error:
Caught signal 11.  Server aborting

Expected results:
VNC session starts

Additional info:
My xorg.conf:

Section "ServerLayout"
   Identifier     "Default Layout"
   Screen      0  "Screen0" 0 0
   InputDevice    "Keyboard0" "CoreKeyboard"
EndSection

Section "InputDevice"
   Identifier  "Keyboard0"
   Driver      "kbd"
   Option       "XkbModel" "pc105"
   Option       "XkbLayout" "us"
EndSection

Section "Monitor"
   Identifier "Mitsubishi"
   ModeLine "1800x1350"   299.67   1800 1968 2168 2480   1350 1351 1354 1397
-hsync -vsync
   Option "DPMS" "true"
EndSection

Section "Device"
   Identifier  "Videocard0"
   Driver      "nv"
EndSection

Section "Screen"
   Identifier "Screen0"
   Device     "Videocard0"
   Monitor    "Mitsubishi"
   DefaultDepth     24
   SubSection "Display"
                Virtual 1800 1350
      Viewport   0 0
      Depth     24
   EndSubSection
EndSection

Section "Module"
   SubSection  "vnc"
   # XXX: Do not know why, but these options do not get picked up
      Option       "SecurityTypes" "VncAuth"
      Option       "PasswordFile" "/root/.vnc/passwd"
      Option       "Localhost" "true"
      Option       "AlwaysShared" "true"
   EndSubSection
   Load "extmod"
   Load "dbe"
   Load "glx"
   Load "freetype"
   Load "type1"
   Load "record"
   Load "dri"
   Load "drm"
EndSection
Comment 1 Adam Tkac 2007-09-12 06:21:37 EDT
I'm not able to reproduce this one. Could you tell me if crash is reproducible
without "dri" module, please? Also better should be specify vnc module options
in Screen section.

Section "Screen"
        Identifier "..."
        Device     "..."
        DefaultDepth     ...
        Option "SecurityTypes" "VncAuth"
        Option "PasswordFile" "/root/.vnc/passwd"
        SubSection "..."
        EndSubSection
EndSection

Also attach your complete /var/log/Xorg.<display>.log and specify your
architecture, please

Thanks, Adam
Comment 2 Tom Sightler 2007-11-20 11:45:22 EST
I am seeing this issue as well, very reproducible on my Dell 2950 server with
the stock RHEL5.1 ati driver and vnc-server packages.

Backtrace:
0: /usr/bin/Xorg(xf86SigHandler+0x71) [0x4a0921]
1: /lib64/libc.so.6 [0x315ac300b0]
2: /usr/bin/Xorg [0x4c28aa]
3: /usr/lib64/xorg/modules/libxaa.so [0x2aaaada10da7]
4: /usr/bin/Xorg [0x510449]
5: /usr/bin/Xorg [0x4d5ebc]
6:
/usr/lib64/xorg/modules/extensions/libvnc.so(_ZN14XserverDesktop10grabRegionERKN3rfb6RegionE+0x106)
[0x2aaaac26e346]
7:
/usr/lib64/xorg/modules/extensions/libvnc.so(_ZN3rfb11VNCServerST11checkUpdateEv+0x182)
[0x2aaaac2782a2]
8:
/usr/lib64/xorg/modules/extensions/libvnc.so(_ZN3rfb16VNCSConnectionST22writeFramebufferUpdateEv+0x96)
[0x2aaaac285ee6]
9:
/usr/lib64/xorg/modules/extensions/libvnc.so(_ZN3rfb16VNCSConnectionST24framebufferUpdateRequestERKNS_4RectEb+0xa6)
[0x2aaaac286606]
10:
/usr/lib64/xorg/modules/extensions/libvnc.so(_ZN3rfb10SMsgReader28readFramebufferUpdateRequestEv+0x180)
[0x2aaaac28bb80]
11:
/usr/lib64/xorg/modules/extensions/libvnc.so(_ZN3rfb16VNCSConnectionST15processMessagesEv+0x30)
[0x2aaaac285520]
12:
/usr/lib64/xorg/modules/extensions/libvnc.so(_ZN14XserverDesktop13wakeupHandlerEP6fd_seti+0xf0)
[0x2aaaac26dcc0]
13: /usr/lib64/xorg/modules/extensions/libvnc.so [0x2aaaac265ae9]
14: /usr/bin/Xorg(WakeupHandler+0x51) [0x44daa1]
15: /usr/bin/Xorg(WaitForSomething+0x1d4) [0x5509c4]
16: /usr/bin/Xorg(Dispatch+0x9a) [0x449b4a]
17: /usr/bin/Xorg(main+0x455) [0x4325b5]
18: /lib64/libc.so.6(__libc_start_main+0xf4) [0x315ac1d8a4]
19: /usr/bin/Xorg(FontFileCompleteXLFD+0x231) [0x4318a9]

xorg.conf (relevant parts):
Section "Module"
        Load  "dbe"
        Load  "extmod"
        Load  "fbdevhw"
        Load  "glx"
        Load  "record"
        Load  "freetype"
        Load  "type1"
        Load  "dri"
        Load  "vnc"
EndSection

Section "Device"
        Identifier  "Videocard0"
        Driver      "ati"
        VendorName  "Videocard vendor"
        BoardName   "ATI ES1000"
EndSection

Section "Screen"
        Identifier "Screen0"
        Device     "Videocard0"
        Monitor    "Monitor0"
        DefaultDepth     16
        SubSection "Display"
                Viewport   0 0
                Depth     16
                Modes    "1024x768" "800x600" "640x480"
        EndSubSection
        Option "SecurityTypes" "VncAuth"
        Option "passwordFile"  "/etc/vnc/passwd"
EndSection

Pretty much makes VNC unusable.
Comment 3 Aleksey Nogin 2007-11-20 12:01:35 EST
Tom, could you try responding to Adam's questions (dri, etc) in comment #1? I
did not have a chance to test that (I ended up switching to x11vnc instead of
using the vnc module). Thanks!

P.S. Marking "Version: 5.1" per Tom's comment.
Comment 4 Tom Sightler 2007-11-20 14:54:13 EST
I disable the dri and glx modules in my xorg.conf and so far it hasn't crashed.
 I guess I'll need to test with each one disabled individually and spend a few
hours without it crashing before I know for sure.

Thanks,
Tom
Comment 5 Tom Sightler 2007-12-06 22:12:41 EST
Created attachment 280481 [details]
Patch to fix crash in vnc.so module

I lifted the patch directly from the vnc-server-4.1.2-19.fc7.src.rpm.  This
single patch applied to the tree seems to resolve the issue for me.
Comment 6 Tom Sightler 2007-12-06 22:27:42 EST
(In reply to comment #4)
> I disable the dri and glx modules in my xorg.conf and so far it hasn't crashed.
>  I guess I'll need to test with each one disabled individually and spend a few
> hours without it crashing before I know for sure.

Unfortunately disable DRI didn't completely fix the issue.  However, I've got a 
nearly perfectly reproducible test case with one of our applications.  The 
application is nothing special, just a commercial backup program (Netvault from 
a company called Bakbone).  Opening the job window in this application, 
maximizing it, and then closing the window causes the crash every time.

I've attached a patch which I pulled from the most recent FC7 version of the 
package which seems to fix the issue for me.  After rebuilding the RHEL5 4.1.2-9 
version of the vnc package with only this single patch added I've found the 
server to run stable.

I've got a patched src.rpm and binaries for x86 and x86_64 if anyone else wants 
to test.
 
Comment 7 Adam Tkac 2007-12-11 10:13:12 EST
Thank you for your report. I think best should be rebase RHEL5 Xvnc to F7 vnc
(bug #385241) because it is more stable. I'm going to try put attached patch to
next vnc update.

Adam
Comment 8 Tom Sightler 2007-12-11 13:38:18 EST
(In reply to comment #7)
> Thank you for your report. I think best should be rebase RHEL5 Xvnc to F7 vnc
> (bug #385241) because it is more stable. I'm going to try put attached patch to
> next vnc update.

I'm sure it's probably true that the better option is to rebase, however, you
also state in that bug that this is not likely to happen before RHEL5U3.  From a
user perspective that's likely to be a year away and potentially longer (U1 took
9 months).

I would hope that a bugfix can get in faster as the vnc.so module as currently
shipped is basically unusable.  It's unrealistic to expect customers to wait for
two years for a working VNC, especially when it worked fine with previous
releases of RHEL.

Comment 9 Adam Tkac 2008-03-19 08:38:31 EDT
*** Bug 251165 has been marked as a duplicate of this bug. ***
Comment 11 Alan Hagge 2008-03-25 16:08:46 EDT
I'd like to add my voice to those requesting a workaround for this issue.  I can
reproduce this issue pretty much at will, and have tried recompiling the EL5.1
vnc-server package with the above patch from Tom S.  In my case, it did not
resolve the issue that I'm having.

I don't have the DRI module in my xorg.conf, and I've tried with and without the
GLX module loading.  No difference.  In my case, I can crash the X server on the
VNC server machine within about 10 seconds of logging in.

I concur that although a rebase is the best solution, the fact that it won't
come before U3 and U2 isn't even out yet means that we need some other workaround.
Comment 12 Adam Tkac 2008-03-26 07:51:07 EDT
Workaround for this bug doesn't exist. I have experience that disable composite
and render extension sometimes help but sometimes no. I'm going to attach patch
for this issue which lives in Fedora and solves this problem
Comment 13 Adam Tkac 2008-03-26 07:56:03 EDT
Created attachment 299134 [details]
Patch which validates framebuffer bounds before framebuffer dump

Anyone can apply this patch with above patch to solve this problem. Final patch
might differ (some optimizations etc)
Comment 14 Alan Hagge 2008-04-03 14:47:12 EDT
I recompiled from the vnc-4.1.2-9.el5.src.rpm incorporating the above patch
(id=299134 from comment #13) and I'm happy to report that it seems to have
resolved my issue.  I tried the resulting RPM on two machines which had been
crashing within 10-15 seconds of login and both stayed up for the 5 minutes that
I tested them.

Thanks for your diligence!  Hopefully this patch can make it into a release
version soon, and the mentioned rebase will happen in the none-too-distant future.
Comment 15 Adam Tkac 2008-04-04 05:07:30 EDT
Thanks for your positive feedback. But this update is not scheduled to 5.2 . I
think it will be in 5.3
Comment 17 RHEL Product and Program Management 2008-06-02 16:32:12 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 20 Adam Tkac 2008-07-04 04:29:02 EDT
*** Bug 240379 has been marked as a duplicate of this bug. ***
Comment 24 errata-xmlrpc 2009-01-20 16:15:58 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0119.html

Note You need to log in before you can comment on or make changes to this bug.