Bug 2075132 - "Xorg -listen tcp" fails to start if ipv6 is disabled on the system [NEEDINFO]
Summary: "Xorg -listen tcp" fails to start if ipv6 is disabled on the system
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: xorg-x11-xtrans-devel
Version: 8.5
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Adam Jackson
QA Contact: Peter Kopec
Marek Suchánek
URL:
Whiteboard:
Depends On:
Blocks: 2090323
TreeView+ depends on / blocked
 
Reported: 2022-04-13 17:01 UTC by Ray Strode [halfline]
Modified: 2023-03-01 15:00 UTC (History)
18 users (show)

Fixed In Version: xorg-x11-server-1.20.11-8.el8 xorg-x11-server-Xwayland-21.1.3-5.el8 xorg-x11-xtrans-devel-1.4.0-4.el8
Doc Type: Bug Fix
Doc Text:
.The desktop no longer fails to start with disabled IPv6 and DisallowTCP=false Previously, the X11 desktop session failed to start after login under the following circumstances: * IPv6 networking was disabled on your system. * The `DisallowTCP=false` option was enabled in GDM configuration. With this update, the problem has been fixed, and you can log into the X11 session as expected with the described configuration.
Clone Of: 2029202
: 2090323 (view as bug list)
Environment:
Last Closed: 2022-11-08 09:44:33 UTC
Type: ---
Target Upstream Version:
msuchane: needinfo? (ajax)


Attachments (Terms of Use)
proposed fix (3.40 KB, patch)
2022-05-06 18:57 UTC, Ray Strode [halfline]
no flags Details | Diff
proposed fix (updated) (7.24 KB, application/mbox)
2022-05-06 19:01 UTC, Ray Strode [halfline]
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-118870 0 None None None 2022-04-13 17:06:32 UTC
Red Hat Product Errata RHSA-2022:7583 0 None None None 2022-11-08 09:44:54 UTC
freedesktop.org Gitlab xorg/lib libxtrans merge_requests 10 0 None opened Automatically disable inet6 transport if ipv6 is disabled on machine 2022-05-10 13:38:45 UTC

Description Ray Strode [halfline] 2022-04-13 17:01:21 UTC
+++ This bug was initially created as a clone of Bug #2029202 +++
--- Additional comment from Andrew Mike on 2022-04-13 11:55:11 EDT ---

The customer in the support case 03183764 has reported that the patch provided for RHEL 8.5 causes the GUI to fail displaying when IPv6 is disabled. Their report follows:

==============================================================
We spent yesterday afternoon and this morning testing a vanilla 8.5 ISO install on a box with and without the updated gdm rpm.  Details of the system:

-  HP Z-series
-  Xeon W-1270 processor
- 16GB ECC memory
- nVidia Quadro P2200 with a single monitor connected (Dell 30" 4k)

The results are as follows:

-  8.5 vanilla with vanilla gdm install - DisallowTCP=false setting does nothing, as expected.  gdm login screen on display 0, sessions are on other vtys (usually 1)
-  8.5 vanilla with updated gdm - Without using DisallowTCP=false, gdm login screen on display 0, sessions load fine
-  8.5 vanilla with updated gdm - DisallowTCP=false does a "-listen tcp" on the command line, gdm login screen on display 0, sessions open on other vtys and work properly, with a listen on both ipv4 and ipv6 on port 600{vty} as expected
-  8.5 vanilla with updated gdm and ipv6 disabled (our baseline) - Without using DisallowTCP=false, gdm login screen on display 0, sessions load fine
-  8.5 vanilla with updated gdm and ipv6 disabled (our baseline) - DisallowTCP=false does a "-listen tcp" on the command line, gdm login screen on display 0, sessions -  attempt to open on other vtys, but the session never completes and never presents the desktop.  I noticed that in this case the ipv4 listen doesn't happen either (via netstat -natp), so I'm assuming it tries to do the AF_INET6 session first.

In the Xorg log file for the display, there are thousands of attempts to created an AF_INET6 socket, which obviously will never work.  So it's trying to open a socket for AF_INET6 and when it can't open the socket, it iterates and continues trying until the entire session load times out.  It's pretty much what I was trying to convey in the previous comment of the ticket.

In the Xorg documentation for the latest builds, it looks like the -listen argument can provide multiple transports.  It's been a while, but I think it was the EINVAL error that comes back from the socket call in cases where the protocol or family isn't supported.
==============================================================

Let me know what else you need from us here, or if we should open a new bug against this issue.

--- Additional comment from Ray Strode [halfline] on 2022-04-13 12:59:01 EDT ---

Normally, AF_INET6 can handle both ipv6 and ipv4 address families. This ipv4 backward compatibility will obviously fail if ipv6 is disabled entirely.

I do see this in the code:

static Sockettrans2dev Sockettrans2devtab[] = {•
...
#else /* IPv6 */•
    {"tcp",AF_INET6,SOCK_STREAM,SOCK_DGRAM,0},•
    {"tcp",AF_INET,SOCK_STREAM,SOCK_DGRAM,0}, /* fallback */•
...
};•

So it looks like the intention is for it automatically fallback to AF_INET if AF_INET6 fails. I guess it's not working though.

I'll clone the bug against libXtrans.

Comment 2 Dwayne 2022-05-06 16:05:38 UTC
Raising the Customer Escalation flag, per RME EN-50824 and the business impact provided by the customer,

Business impact: 
We have 2000+ Red Hat desktops in our private environment.  Several mission critical applications are unable to launch for our environment, making it impossible for our users to do their jobs.

We have custom X applications that rely on being able to present remotely to the desktop over the X ports (60xx).  We received a bug fix for GDM that actually fixes the ability to *turn on* the port, but now have a bug in the Xorg server that breaks the desktop completely (Xorg will not load) when IPv6 is disabled (which it is in our environment.)  We use nVidia video cards on all physical desktops and therefore cannot use Wayland.  We currently cannot proceed in our 8.5 deployment because of the mission critical application block.

2 bugs have been filed on this:  2029202 against GDM which has been fixed, which then led to bug 2075132 against libXTrans, which has not been fixed.  Until we get a build that allows TCP ports to listen appropriately on the Xorg server, we can't proceed with our 8.x rollout.  We need a build that actually works properly with IPv6 disabled.  Bug 2075132 has been marked as medium priority and medium severity, which seems completely off since it simply does not work at all with IPv6 disabled.

Comment 3 Ray Strode [halfline] 2022-05-06 17:59:54 UTC
So the problematic code is here in the xserver:

```
void•
CreateWellKnownSockets(void)•
{•
 ...
    else { /* -displayfd and no explicit display number */•
        Bool found = 0;•
        for (i = 0; i < 65536 - X_TCP_PORT; i++) {•
            if (TryCreateSocket(i, &partial) && !partial) {•
                found = 1;•
                break;•
            }•
            else•
                CloseWellKnownConnections();•
        }•
...
    }•
...
}•
```

The `TryCreateSocket()` function not only returns whether or not it is able to create any of its sockets, it also returns if it is able to create all of its sockets, through the `partial` out variable.

This is an important distinction, because you could imagine a situation where e.g. a rogue X server is listening on a tcp port, but not on the local socket in /tmp. In that situation, it's important for the `-displayfd` code to disregard that potential display number, so that e.g., DISPLAY=:1 and DISPLAY=localhost:1 don't go to different X servers, defying user expectations.

But, unfortunately, `TryCreateSocket` also returns `partial=TRUE` in the case where it can bind to `AF_INET` but not `AF_INET6`.

I think the xtrans code needs to set the `TRANS_DISABLED` flag on the transport if it fails with `EAFNOSUPPORT` so it will get ignored when doing the `partial` handling.

Comment 4 Ray Strode [halfline] 2022-05-06 18:57:38 UTC Comment hidden (obsolete)
Comment 5 Ray Strode [halfline] 2022-05-06 19:01:44 UTC Comment hidden (obsolete)
Comment 6 Ray Strode [halfline] 2022-05-06 19:09:42 UTC
Created attachment 1877647 [details]
proposed fix (updated)

This patch does what I proposed in the comment 3 with one minor change. I used the NOLISTEN flag instead of the DISABLED flag, because the DISABLED flag doesn't restrict it from the listener count when checking for how to set partial.

This attachment obsoletes attachment 1877644 [details] which was missing the actual patch in the patch.

Comment 36 errata-xmlrpc 2022-11-08 09:44:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: xorg-x11-server and xorg-x11-server-Xwayland security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7583


Note You need to log in before you can comment on or make changes to this bug.