Bug 1004093 - Xorg hangs on logout when using xdm & the VNC module
Xorg hangs on logout when using xdm & the VNC module
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: tigervnc (Show other bugs)
All Linux
medium Severity medium
: rc
: ---
Assigned To: Tim Waugh
: Patch
: 983089 (view as bug list)
Depends On:
  Show dependency treegraph
Reported: 2013-09-03 18:47 EDT by Michael Kolbas
Modified: 2014-10-14 00:39 EDT (History)
1 user (show)

See Also:
Fixed In Version: tigervnc-1.1.0-9.el6
Doc Type: Bug Fix
Doc Text:
The vnc Xorg extension could be initialized twice, leading to a busy loop on exit. This has been fixed by backporting a patch to only allow initialization once.
Story Points: ---
Clone Of:
Last Closed: 2014-10-14 00:39:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
tigervnc-extension-init-twice.patch (715 bytes, patch)
2013-12-05 12:18 EST, Tim Waugh
no flags Details | Diff

  None (edit)
Description Michael Kolbas 2013-09-03 18:47:20 EDT
Description of problem:

 When customer sets DISPLAYMANAGER=xdm in /etc/sysconfig/desktop and
 also has the native VNC extension for X loaded, logout from gnome
 hangs when using 32bit RHEL, but customer does not see this issue
 with x86_64.  Though, GDM behaves OK with the VNC extension loaded.

Version-Release number of selected component (if applicable):


 (though it may apply to other i686 xorg versions)

How reproducible:
 Always with 32bit RHEL with Xorg and tigervnc-server-module.
 Hardware can be i686 or x86_64, but issue is seen with 32bit OS.

Steps to Reproduce:

 1. Install a minimal RHEL 6 32-bit server.
 2. yum install tigervnc-server-module xorg-x11-fonts-misc \
    xscreensaver xorg-x11-drivers openmotif xorg-x11-server-Xorg \
    xorg-x11-xauth xorg-x11-xinit xorg-x11-server-utils xorg-x11-utils \

 3. Edit/create /etc/sysconfig/desktop: 

 4. Create /etc/X11/xorg.conf.d/vnc.conf:
     Section "Module"
             Load    "vnc"

 5. Create a test user.

 6. Login at the the test user and start runlevel 5 if not there

 7. Login locally as the test user at the xdm greeter prompt.

 8. Logout and observe the hang.  

Actual results:

 - xdm / Xorg hangs after logging out.
 - X server may eventually consume 100% cpu usage until killed.
 - Never returns to the xdm screen.

Expected results:
 - Should return to the xdm login prompt after logging out.

Additional info:

 - This does not happen with 32bit RHEL and gdm with the native
   tigervnc-server-module loaded.
Comment 4 Tim Waugh 2013-12-05 10:58:20 EST
The rfb::Configuration linked list has an element whose next element is itself.

Not entirely sure how this happens yet.

(gdb) p *rfb::Configuration::global()
$54 = {name = {buf = 0x1ba58c0 "Global"}, head = 0x7f61c4e9d660, 
  _next = 0x1ba1080, static global_ = 0x1ba58a0, static server_ = 0x1ba1080, 
  static viewer_ = 0x1ba5820}
(gdb) p *rfb::Configuration::global()->_next
$55 = {name = {buf = 0x1ba5210 "Server"}, head = 0x7f61c4e9f3e0, 
  _next = 0x1ba1080, static global_ = 0x1ba58a0, static server_ = 0x1ba1080, 
  static viewer_ = 0x1ba5820}
Comment 5 Tim Waugh 2013-12-05 11:37:29 EST
This is the point at which the list gets looped:

Old value = (rfb::Configuration *) 0x0
New value = (rfb::Configuration *) 0x291c080
0x00007f2700a011e0 in appendConfiguration ()
    at ../../../../common/rfb/Configuration.h:152
152	      conf->_next = _next; _next = conf;
(gdb) bt
#0  0x00007f2700a011e0 in appendConfiguration ()
    at ../../../../common/rfb/Configuration.h:152
#1  enableServerParams () at ../../../../common/rfb/Configuration.h:104
#2  vncExtensionInitWithParams () at xf86vncModule.cc:86
#3  0x00000000004c1e79 in InitExtensions (argc=<value optimized out>, 
    argv=<value optimized out>) at ../../../mi/miinitext.c:334
#4  0x000000000047c9db in main (argc=4, argv=<value optimized out>, 
    envp=<value optimized out>) at main.c:208
Comment 6 Tim Waugh 2013-12-05 12:18:56 EST
Created attachment 833286 [details]

This seems to stop the problem from happening.
Comment 7 Tim Waugh 2013-12-05 12:20:14 EST
I saw this on x86_64, so removing "32-bit" from the description.
Comment 8 Tim Waugh 2013-12-06 12:19:40 EST
Better to use the upstream fix (r5065).
Comment 11 Tim Waugh 2014-05-23 06:32:49 EDT
*** Bug 983089 has been marked as a duplicate of this bug. ***
Comment 25 errata-xmlrpc 2014-10-14 00:39:17 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.