Description of problem: VNC server crashes when connected by any VNC clienct otther than TigerVNC 1.0.1 (win) Version-Release number of selected component (if applicable): TigerVNC 1.0.90 How reproducible: always Steps to Reproduce: 1. run VNC server on x64 F13 box 2. connect it with UVNC client 3. enter password Actual results: Crashing the entire VNC server session. Expected results: Connect and work without problems (was working @F12 , installed F13 and having this problem) , or at least give and error for an incompatible client. Additional info: filing log ...
Created attachment 421053 [details] vnc log
Using UltraVNC to connect.
Created attachment 421728 [details] vncserver log with backtrace I can confirm the crash with krdc (KDE VNC viewer) too. If I use the tigervnc client everything works fine.
This seems to be a 64-bit issue in the server code: 32-bit krdc connected to 32-bit tigervnc server: OK 64-bit krdc connected to 32-bit tigervnc server: OK 32-bit krdc connected to 64-bit tigervnc server: CRASH 64-bit krdc connected to 64-bit tigervnc server: CRASH Mehmet: can you confirm that?
yes, I can confirm . I can connect to 32bit server without problems.
I installed tigervnc-debuginfo and ran Xvnc inside gdb to get a backtrace: (gdb) bt #0 0x0000000000515578 in rfb::Encoder::supported (encoding=-131069) at Encoder.cxx:35 #1 0x000000000053d579 in rfb::ConnParams::setEncodings (this=0xa4f388, nEncodings=<value optimized out>, encodings=<value optimized out>) at ConnParams.cxx:132 #2 0x000000000051d995 in rfb::SMsgHandler::setEncodings (this=0xa4f380, nEncodings=<value optimized out>, encodings=<value optimized out>) at SMsgHandler.cxx:44 #3 0x000000000053dea8 in rfb::SMsgReader::readSetEncodings (this=0xa49230) at SMsgReader.cxx:57 #4 0x00000000005276d8 in rfb::VNCSConnectionST::processMessages ( this=0xa4f380) at VNCSConnectionST.cxx:117 #5 0x000000000050485c in XserverDesktop::wakeupHandler (this=0x875a90, fds=0x85a720, nfds=<value optimized out>) at XserverDesktop.cc:572 #6 0x00000000004fd464 in vncWakeupHandler (data=<value optimized out>, nfds=1, readmask=0x85a720) at vncExtInit.cc:313 #7 0x00000000005661c3 in WakeupHandler (result=1, pReadmask=0x85a720) at dixutils.c:403 #8 0x00000000005aa68f in WaitForSomething (pClientsReady=0xa48d70) at WaitFor.c:232 #9 0x0000000000560c22 in Dispatch () at dispatch.c:375 #10 0x00000000004fc8cb in main (argc=6, argv=0x7fffffffdad8, envp=<value optimized out>) at main.c:286 (gdb) l 30 { 31 } 32 33 EncoderCreateFnType Encoder::createFns[encodingMax+1] = { 0 }; 34 35 bool Encoder::supported(int encoding) 36 { 37 return encoding <= encodingMax && createFns[encoding]; 38 } encoding is -131069 => 0xFFFFFFFFFFFE0003, so I assume the original value was "3" but got incorrectly converted somehow. 2 levels up I see: 42 void SMsgHandler::setEncodings(int nEncodings, rdr::S32* encodings) 43 { 44 cp.setEncodings(nEncodings, encodings); 45 supportsLocalCursor(); 46 } So I assume "rdr::S32" conversion to "int" breaks on 64-bit? One more level up: (gdb) up #3 0x000000000053dea8 in rfb::SMsgReader::readSetEncodings (this=0xa49230) at SMsgReader.cxx:57 57 handler->setEncodings(nEncodings, encodings.buf); (gdb) l 52 is->skip(1); 53 int nEncodings = is->readU16(); 54 rdr::S32Array encodings(nEncodings); 55 for (int i = 0; i < nEncodings; i++) 56 encodings.buf[i] = is->readU32(); 57 handler->setEncodings(nEncodings, encodings.buf); 58 } (gdb) print nEncodings $6 = 18 Hmm, I guess "rdr:S32" means "signed 32-bit". Reading unsigned 32-bit values into such a variable is a little bit dangerous... I wonder if the problem goes away if "bool Encoder::supported(int encoding)" would be changed to rdr::S32?
(In reply to comment #6) > Hmm, I guess "rdr:S32" means "signed 32-bit". Reading unsigned 32-bit values > into such a variable is a little bit dangerous... > > I wonder if the problem goes away if "bool Encoder::supported(int encoding)" > would be changed to rdr::S32? Your theory looks valid but this code is pretty old and worked fine for a long time. I wonder why it got broken, conversion between "int" and "unsigned int" should be no problem, both values are 4 bytes long. I'm going to test your proposed solution.
After inspection problem is not the unsigned->signed cast. Would it be possible to test proposed update, please? Download appropriate packages from http://koji.fedoraproject.org/koji/taskinfo?taskID=2236284. Thank you in advance.
The new package works like a charm for me. So what was the actual correction? Maybe you can attach the patch?
Created attachment 422093 [details] vncserver log
Created attachment 422094 [details] xsession log
I can't confim the correction after installing those packages . (I might've done something wrong, I can provide any info you might need ) . As second , if any way i stop VNCserver service once , or server chrashes , server is not able start normally again. After restarting the service , I get the logs in the attchment and the result in viewer here : http://img41.imageshack.us/img41/3220/desktoperror.jpg
Mehmet: are you sure you installed the correct *server* package on the machine where you run vncserver on? E.g. do you see in the shell: $ rpm -q tigervnc-server tigervnc-server-1.0.90-0.11.20100420svn4030.fc13.1.x86_64 $ sha1sum /usr/bin/Xvnc 9cb300241e99837722741ac2aad14c4ae2d78cc1 /usr/bin/Xvnc I have now tested the package on both of my 64-bit machines and it works now.
BTW: the crashed server can leave locks behind, e.g. if VNC was running on server:5901 then please check on the server that /tmp/.X1-lock and /tmp/.X11-unix/X1 are removed before trying again. If vncserver detects those locks it will assign the next free port, e.g. server:5902.
(In reply to comment #14) > BTW: the crashed server can leave locks behind, e.g. if VNC was running on > server:5901 then please check on the server that /tmp/.X1-lock and > /tmp/.X11-unix/X1 are removed before trying again. If vncserver detects those > locks it will assign the next free port, e.g. server:5902. It is not a lock issue . It still starts on 5901 , but dbus keeps giving errors. And it happens even if I stop vncserver service gracefully . The other issue is : [root@local hdd]# rpm -iv tigervnc-* Preparing packages for installation... tigervnc-server-module-1.0.90-0.11.20100420svn4030.fc13.1 tigervnc-1.0.90-0.11.20100420svn4030.fc13.1 [root@local hdd]# sha1sum /usr/bin/Xvnc 207ef36c3d32d5a717e775967a7126446fcea60e /usr/bin/Xvnc Stopped services , rebooted , removed the packages and reinstalled. But sha1sum does not match. are you sure it is right?
The sha1sum changes if prelink is activated on the machine, so it is useless for comparison.
Created attachment 422470 [details] VNClog [root@local ~]# rpm -qva | grep tigervnc tigervnc-server-1.0.90-0.11.20100420svn4030.fc13.x86_64 tigervnc-server-module-1.0.90-0.11.20100420svn4030.fc13.1.x86_64 tigervnc-1.0.90-0.11.20100420svn4030.fc13.1.x86_64 If sha1sum match is not neccessary , then I still can't confim it works. Ps: I have got some warnings about protocol version mismatch in my log. Please check the last one I posted.
(In reply to comment #17) > Created an attachment (id=422470) [details] > VNClog > > [root@local ~]# rpm -qva | grep tigervnc > tigervnc-server-1.0.90-0.11.20100420svn4030.fc13.x86_64 ^^^^^^^ should be .fc13.1.x86_64 > tigervnc-server-module-1.0.90-0.11.20100420svn4030.fc13.1.x86_64 > tigervnc-1.0.90-0.11.20100420svn4030.fc13.1.x86_64 You have updated only "tigervnc" and "tigervnc-server-module" packages but you have to update the "tigervnc-server" package. Fix is in the Xvnc binary which is part of the tigervnc-server.
(In reply to comment #9) > The new package works like a charm for me. > > So what was the actual correction? Maybe you can attach the patch? The actual correction was expansion of the encoding range checks. Per RFB protocol only signed 32bit values from interval <0;255> are used as encodings but some VNC implementations use negative values for their private purposes, like UltraVNC. I will attach the patch.
Created attachment 422509 [details] Patch This patch extends encoding range checks.
(In reply to comment #12) > As second , if any way i stop VNCserver service once , or server chrashes , > server is not able start normally again. After restarting the service , I get > the logs in the attchment and the result in viewer here : > > http://img41.imageshack.us/img41/3220/desktoperror.jpg If you are able to continuously reproduce this issue please open a separate bug for it. This bug is about Xvnc crashes when different clients than vncviewer/vinagre are used.
It seems that this bug has one really irritating impact. The server is also used for vnc option in anacanoda. And if you do not know about this problem you just wonder why instalation crashes everytime you try to connect. For now it seems that using tigervnc solved the problem however is there any easy workaround how to add the corrected VNC server to stage2 file? I did install 32bit version without any problems few days ago so this really seems to be only problem of 64bit version.
> The actual correction was expansion of the encoding range checks. Per RFB > protocol only signed 32bit values from interval <0;255> are used as encodings > but some VNC implementations use negative values for their private purposes, > like UltraVNC. I will attach the patch. I don't think is a entirely correct reading of the RFB protocol. The latest PDF of the spec doesn't appear to mention the 0;255 interval at all. I think that simply by convention they are allocating positive numbers to real framebuffer encodings, and negative numbers to pseudo encodings. There are many psuedo encodings listed in the RFB spec with negative numbers already.
tigervnc-1.0.90-0.12.20100420svn4030.fc13 has been submitted as an update for Fedora 13. http://admin.fedoraproject.org/updates/tigervnc-1.0.90-0.12.20100420svn4030.fc13
tigervnc-1.0.90-0.12.20100420svn4030.fc13 has been pushed to the Fedora 13 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update tigervnc'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/tigervnc-1.0.90-0.12.20100420svn4030.fc13
tigervnc-1.0.90-0.12.20100420svn4030.fc13 has been pushed to the Fedora 13 stable repository. If problems still persist, please make note of it in this bug report.