Bug 1004093

Summary: Xorg hangs on logout when using xdm & the VNC module
Product: Red Hat Enterprise Linux 6 Reporter: Michael Kolbas <mkolbas>
Component: tigervncAssignee: Tim Waugh <twaugh>
Status: CLOSED ERRATA QA Contact: qe-baseos-daemons
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: psklenar
Target Milestone: rcKeywords: Patch
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: tigervnc-1.1.0-9.el6 Doc Type: Bug Fix
Doc Text:
The vnc Xorg extension could be initialized twice, leading to a busy loop on exit. This has been fixed by backporting a patch to only allow initialization once.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-14 04:39:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
tigervnc-extension-init-twice.patch none

Description Michael Kolbas 2013-09-03 22:47:20 UTC
Description of problem:

 When customer sets DISPLAYMANAGER=xdm in /etc/sysconfig/desktop and
 also has the native VNC extension for X loaded, logout from gnome
 hangs when using 32bit RHEL, but customer does not see this issue
 with x86_64.  Though, GDM behaves OK with the VNC extension loaded.


Version-Release number of selected component (if applicable):

 xorg-x11-server-Xorg-1.13.0-11.el6.i686
 tigervnc-server-module-1.1.0-5.el6_4.1.i686

 (though it may apply to other i686 xorg versions)
 

How reproducible:
 
 Always with 32bit RHEL with Xorg and tigervnc-server-module.
 Hardware can be i686 or x86_64, but issue is seen with 32bit OS.

Steps to Reproduce:

 1. Install a minimal RHEL 6 32-bit server.
 
 2. yum install tigervnc-server-module xorg-x11-fonts-misc \
    xscreensaver xorg-x11-drivers openmotif xorg-x11-server-Xorg \
    xorg-x11-xauth xorg-x11-xinit xorg-x11-server-utils xorg-x11-utils \
    xdm

 3. Edit/create /etc/sysconfig/desktop: 
     DISPLAYMANAGER=xdm

 4. Create /etc/X11/xorg.conf.d/vnc.conf:
     Section "Module"
             Load    "vnc"
     EndSection

 5. Create a test user.

 6. Login at the the test user and start runlevel 5 if not there
    already.

 7. Login locally as the test user at the xdm greeter prompt.

 8. Logout and observe the hang.  


Actual results:

 - xdm / Xorg hangs after logging out.
 - X server may eventually consume 100% cpu usage until killed.
 - Never returns to the xdm screen.


Expected results:
 
 - Should return to the xdm login prompt after logging out.


Additional info:

 - This does not happen with 32bit RHEL and gdm with the native
   tigervnc-server-module loaded.

Comment 4 Tim Waugh 2013-12-05 15:58:20 UTC
The rfb::Configuration linked list has an element whose next element is itself.

Not entirely sure how this happens yet.

(gdb) p *rfb::Configuration::global()
$54 = {name = {buf = 0x1ba58c0 "Global"}, head = 0x7f61c4e9d660, 
  _next = 0x1ba1080, static global_ = 0x1ba58a0, static server_ = 0x1ba1080, 
          ^^^^^^^^^
  static viewer_ = 0x1ba5820}
(gdb) p *rfb::Configuration::global()->_next
$55 = {name = {buf = 0x1ba5210 "Server"}, head = 0x7f61c4e9f3e0, 
  _next = 0x1ba1080, static global_ = 0x1ba58a0, static server_ = 0x1ba1080, 
          ^^^^^^^^^
  static viewer_ = 0x1ba5820}

Comment 5 Tim Waugh 2013-12-05 16:37:29 UTC
This is the point at which the list gets looped:

Old value = (rfb::Configuration *) 0x0
New value = (rfb::Configuration *) 0x291c080
0x00007f2700a011e0 in appendConfiguration ()
    at ../../../../common/rfb/Configuration.h:152
152	      conf->_next = _next; _next = conf;
(gdb) bt
#0  0x00007f2700a011e0 in appendConfiguration ()
    at ../../../../common/rfb/Configuration.h:152
#1  enableServerParams () at ../../../../common/rfb/Configuration.h:104
#2  vncExtensionInitWithParams () at xf86vncModule.cc:86
#3  0x00000000004c1e79 in InitExtensions (argc=<value optimized out>, 
    argv=<value optimized out>) at ../../../mi/miinitext.c:334
#4  0x000000000047c9db in main (argc=4, argv=<value optimized out>, 
    envp=<value optimized out>) at main.c:208

Comment 6 Tim Waugh 2013-12-05 17:18:56 UTC
Created attachment 833286 [details]
tigervnc-extension-init-twice.patch

This seems to stop the problem from happening.

Comment 7 Tim Waugh 2013-12-05 17:20:14 UTC
I saw this on x86_64, so removing "32-bit" from the description.

Comment 8 Tim Waugh 2013-12-06 17:19:40 UTC
Better to use the upstream fix (r5065).

Comment 11 Tim Waugh 2014-05-23 10:32:49 UTC
*** Bug 983089 has been marked as a duplicate of this bug. ***

Comment 25 errata-xmlrpc 2014-10-14 04:39:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1412.html