Bug 522369 - Assertion when changing colour depth
Assertion when changing colour depth
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: tigervnc (Show other bugs)
11
All Linux
low Severity medium
: ---
: ---
Assigned To: Adam Tkac
Fedora Extras Quality Assurance
:
: 541373 547370 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-09-10 04:34 EDT by Tom Hughes
Modified: 2013-04-30 19:44 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-06-28 10:34:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Tom Hughes 2009-09-10 04:34:46 EDT
Description of problem:

There is an assertion in vncviewer when I try and change the colour depth to 256 colours.

Version-Release number of selected component (if applicable):

tigervnc-0.0.91-0.11.fc11.x86_64

How reproducible:

Oftem

Steps to Reproduce:
1. Start vncviewer and connect to server
2. Hit F8 and use options to change colour depth to 256 colours.
3. Watch vncviewer crash
  
Actual results:

Assertion in vncviewer.

Expected results:

No assertion.

Additional info:

Server is vino 2.26.2 on a Fedora 11 system.

Backtrace from gdb when the assertion fires:

#0  0x0000003e0a6332f5 in *__GI_raise (sig=<value optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0000003e0a634b20 in *__GI_abort () at abort.c:88
#2  0x0000003e0a62c2fa in *__GI___assert_fail (
    assertion=0x46a4f3 "len <= end - ptr", file=<value optimized out>, 
    line=196, 
    function=0x46a740 "void rfb::zrleDecode8(const rfb::Rect&, rdr::InStream*, rdr::ZlibInStream*, rdr::U8*, rfb::CMsgHandler*)") at assert.c:78
#3  0x000000000043ded8 in rfb::zrleDecode8 (r=<value optimized out>, 
    is=<value optimized out>, zis=0x8b1520, buf=<value optimized out>, 
    handler=<value optimized out>) at ../rfb/zrleDecode.h:196
#4  0x000000000045085c in rfb::CMsgReader::readRect (this=0x8afc80, 
    r=@0x7fffffff9d60, encoding=16) at CMsgReader.cxx:115
#5  0x000000000041df18 in rfb::CMsgReaderV3::readMsg (this=0x8afc80)
    at CMsgReaderV3.cxx:94
#6  0x00000000004160b3 in main (argc=<value optimized out>, 
    argv=0x7fffffffda58) at vncviewer.cxx:400
Comment 1 Tomáš Bžatek 2009-10-27 09:11:36 EDT
Just happened to me twice, using vncviewer tunneled through ssh (the -via argument).

Steps to Reproduce:
1. Start vncviewer and connect to server
2. Hit F8, select Options
3. Uncheck the Auto select checkbox, select 64 colours. Tight encoding is active.
4. The key to crash is to press the OK button when redraw is in progress.

If you wait until the screen is redrawed, it won't crash. I have very slow connection and complex screen, redrawing with full colour depth takes me up to ten seconds. So within this interval I'm able to reconfigure settings and will get crash once the redraw is finished.


tigervnc-1.0.0-1.fc12.x86_64
Rawhide (F12 beta)


TigerVNC Viewer for X version 1.0.0 - built Aug 28 2009 07:36:22
Copyright (C) 2002-2005 RealVNC Ltd.
Copyright (C) 2000-2006 TightVNC Group
Copyright (C) 2004-2009 Peter Astrand for Cendio AB
See http://www.tigervnc.org for information on TigerVNC.

Tue Oct 27 14:06:05 2009
 CConn:       connected to host localhost port 5599
 CConnection: Server supports RFB protocol version 3.8
 CConnection: Using RFB protocol version 3.8

Tue Oct 27 14:06:07 2009
 TXImage:     Using default colormap and visual, TrueColor, depth 24.
 CConn:       Using pixel format depth 24 (32bpp) little-endian rgb888
 CConn:       Using Tight encoding

Tue Oct 27 14:06:12 2009
 CConn:       Using pixel format depth 6 (8bpp) rgb222
vncviewer: ../rfb/zrleDecode.h:196: void rfb::zrleDecode8(const rfb::Rect&, rdr::InStream*, rdr::ZlibInStream*, rdr::U8*, rfb::CMsgHandler*): Assertion `len <= end - ptr' failed.
Aborted



#0  0x00007ffff675b575 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff675cd55 in abort () at abort.c:92
#2  0x00007ffff6754655 in __assert_fail (assertion=0x46a393 "len <= end - ptr", file=<value optimized out>, line=196, 
    function=0x46a5e0 "void rfb::zrleDecode8(const rfb::Rect&, rdr::InStream*, rdr::ZlibInStream*, rdr::U8*, rfb::CMsgHandler*)")
    at assert.c:81
#3  0x000000000043dd98 in rfb::zrleDecode8 (r=<value optimized out>, is=<value optimized out>, zis=0x6cd400, buf=<value optimized out>, 
    handler=<value optimized out>) at ../rfb/zrleDecode.h:196
#4  0x000000000045062c in rfb::CMsgReader::readRect (this=0x6ce650, r=..., encoding=16) at CMsgReader.cxx:115
#5  0x000000000041dec2 in rfb::CMsgReaderV3::readMsg (this=0x6ce650) at CMsgReaderV3.cxx:94
#6  0x000000000041609b in main (argc=<value optimized out>, argv=0x7fffffffe198) at vncviewer.cxx:400
Comment 2 Tomáš Bžatek 2009-10-27 09:16:01 EDT
Forgot to add, the server is running on Debian vnc4server-4.1.1+X4.3.0-31 (i.e. RealVNC).
Comment 3 Adam Tkac 2009-11-27 09:24:43 EST
*** Bug 541373 has been marked as a duplicate of this bug. ***
Comment 4 Jan Görig 2010-03-03 02:42:47 EST
*** Bug 547370 has been marked as a duplicate of this bug. ***
Comment 5 Charlie Brej 2010-04-07 16:23:01 EDT

How to reproduce
-----
1.Connect to a remote VNC desktop over a slow connection
2.As the page is being redrawn slowly (due to the default 24bpp), open the F8 menu and decrease the colors to 256 or 64
3.get a crash


Comment
-----
The following is the terminal output:
[brejc8@kitt ~]$ vncviewer grovel.cs.man.ac.uk:20

TigerVNC Viewer for X version 1.0.0 - built Oct 26 2009 10:57:15
Copyright (C) 2002-2005 RealVNC Ltd.
Copyright (C) 2000-2006 TightVNC Group
Copyright (C) 2004-2009 Peter Astrand for Cendio AB
See http://www.tigervnc.org for information on TigerVNC.

Wed Apr  7 19:39:37 2010
 CConn:       connected to host grovel.cs.man.ac.uk port 5920
 CConnection: Server supports RFB protocol version 3.7
 CConnection: Using RFB protocol version 3.7

Wed Apr  7 19:39:40 2010
 TXImage:     Using default colormap and visual, TrueColor, depth 24.
 CConn:       Using pixel format depth 24 (32bpp) little-endian rgb888
 CConn:       Using Tight encoding

Wed Apr  7 19:39:43 2010
 CConn:       Throughput 3218 kbit/s - changing to quality 6 
 CConn:       Using Tight encoding

Wed Apr  7 19:39:48 2010
 CConn:       Using pixel format depth 6 (8bpp) rgb222
vncviewer: ../rfb/zrleDecode.h:196: void rfb::zrleDecode8(const rfb::Rect&, rdr::InStream*, rdr::ZlibInStream*, rdr::U8*, rfb::CMsgHandler*): Assertion `len <= end - ptr' failed.
Aborted (core dumped)
Comment 6 Bug Zapper 2010-04-28 06:16:27 EDT
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 7 Nils Philippsen 2010-05-04 09:28:20 EDT
I can't reproduce this from updated F12 to updated F12.
Comment 8 Bug Zapper 2010-06-28 10:34:26 EDT
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 9 Nils Philippsen 2010-07-01 05:46:15 EDT
Seems fixed in F-13 with tigervnc-1.0.90-0.12.20100420svn4030.fc13.x86_64
Comment 10 Michael Torrie 2011-01-29 12:12:40 EST
Bug is still present in Fedora 14.  I can reproduce it by connecting to a OS X 10.5 machine (with vnc screen sharing enabled) over a slow connection.  It starts out trying 24-bit mode, then drops to 8-bit before the screen has even drawn, and core dumps.

Note You need to log in before you can comment on or make changes to this bug.