Bug 1673793

Summary:	When WaylandEnable=false, starting X11 session from GDM clobbers existing Xvnc display
Product:	Red Hat Enterprise Linux 8	Reporter:	DRC <dcommander>
Component:	gdm	Assignee:	Ray Strode [halfline] <rstrode>
Status:	CLOSED WONTFIX	QA Contact:	Desktop QE <desktop-qa-list>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.0	CC:	hdegoede, tpelka
Target Milestone:	rc
Target Release:	8.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-01 07:32:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description DRC 2019-02-08 06:15:19 UTC

Description of problem:
The TurboVNC Server is an Xvnc solution based on Xorg 1.19.6, similar in concept to but with different focus than the TigerVNC Server.  Whenever a TurboVNC Server session is running on Display :1 in RHEL 8 beta, an X11 session started from GDM with WaylandEnable=false will also use Display :1, which prevents the TurboVNC session from working.  For reasons I can't fathom, this doesn't happen with the RHEL 8 build of TigerVNC, even though it appears to be registering its use of Display :1 in exactly the same way (/tmp/.X11-unix/X1 and /tmp/.X1-lock).  I've tried various combinations of enabling and disabling X11 over TCP in the TurboVNC Server, and that didn't seem to make a difference.  I posit that GDM's X display selection logic may not be robust enough, but if there's something we need to be doing to keep GDM from stomping on our display number, then I'm happy to do that.

This may also affect other 3rd-party "X proxy" solutions other than TurboVNC.  I haven't tested that.

Version-Release number of selected component (if applicable):
gdm-3.28.3-9

How reproducible:
Always

Steps to Reproduce:
- Uncomment WaylandEnable=false in /etc/gdm/custom.conf
- Restart GDM
- Leave it sitting at the login prompt
- SSH into the system (or switch to an unused virtual console)
- sudo dnf install https://sourceforge.net/projects/turbovnc/files/2.2.1/turbovnc-2.2.1.x86_64.rpm
- /opt/TurboVNC/bin/vncserver
  (double check that it is listening on Display :1)
- Log in locally using "Standard (X11 display server)" session

Actual results:
The local session uses Display :1

Expected results:
The local session uses Display :2 or another unused X display

Additional info:

Comment 3 RHEL Program Management 2021-02-01 07:32:37 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 4 DRC 2024-01-04 23:39:00 UTC

In case anyone stumbles upon this issue via Google, the workaround is to pass '-listen local' to /opt/TurboVNC/bin/vncserver.  Apparently GDM will choose Display :1 for the X11 session regardless of whether something is listening on the pathname Unix domain socket associated with Display :1 (/tmp/.X11-unix/X1).  However, GDM will choose Display :2 for the X11 session if it detects that something is listening on the abstract Unix domain socket associated with Display :1.  Passing '-listen local' to /opt/TurboVNC/bin/vncserver causes TurboVNC to listen on both the pathname and abstract UDS associated with its X display number.  Just be aware that abstract UDSs have some security concerns, so this workaround may not be a good idea on production TurboVNC hosts.