Bug 496882 - XDMCP connection fails in recent gdm versions
XDMCP connection fails in recent gdm versions
Product: Fedora
Classification: Fedora
Component: gdm (Show other bugs)
All Linux
low Severity medium
: ---
: ---
Assigned To: jmccann
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2009-04-21 10:32 EDT by Michael Young
Modified: 2015-01-14 18:22 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2010-06-28 08:08:51 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
patch which fixes the problem for me (1005 bytes, patch)
2009-04-23 07:16 EDT, Michael Young
no flags Details | Diff
patch workarounding the problem: BadValue ... integer parameter out of range for operation (2.06 KB, patch)
2009-10-25 16:25 EDT, Albert Flügel
no flags Details | Diff
Spec file for including patch gdm-2.26.1-ign-xaddhost-fail.patch into gdm-2.26.1-13.1 (78.94 KB, text/plain)
2009-10-25 16:28 EDT, Albert Flügel
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
GNOME Bugzilla 599683 None None None Never

  None (edit)
Description Michael Young 2009-04-21 10:32:44 EDT
When trying to connect to a server using xdmcp, I get repeated errors like the following in the gdm server log files,
Invalid MIT-MAGIC-COOKIE-1 keygdm-simple-slave[10451]: WARNING: Unable to connect to display
and the X window remains blank (this is via Xvnc, but the same thing happens if I connect via Xnest). If I downgrade gdm to gdm-2.25.2-19.fc11 or earlier then I can connect, but not with gdm-2.25.2-20.fc11 to anything I have tried after that.
Comment 1 Michael Young 2009-04-21 11:49:34 EDT
I have now looked at the traffic between the Xnest and gdm, and the first 4 bytes of the MIT-MAGIC-COOKIE-1 data in the XDMCP ACCEPT call from gdm are null bytes. However in the initial X connection from the gdm end these 4 bytes are no longer null so Xnest rejects the connection. The final 12 bytes of data are the same. So it looks like the ACCEPT call has lost a few bytes.
Comment 2 Michael Young 2009-04-23 07:16:40 EDT
Created attachment 340927 [details]
patch which fixes the problem for me

And the bug is that you are freeing the cookie GArray too soon. In daemon/gdm-xdmcp-display-factory.c from line 2309 onwards you have the section
                                authorization_data.data     = (CARD8 *) cookie->data;
                                authorization_data.length   = cookie->len;

                                g_array_free (cookie, TRUE);

                                /* the addrs are NOT copied */
                                gdm_xdmcp_send_accept (factory,

and the bug is that you are freeing cookie, but then using string associated to it in gdm_xdmcp_send_accept and thus you lose the first four bytes of the cookie data presumably as part of the freeing process. Moving the g_array_free line after the gdm_xdmcp_send_accept call fixes the problem for me (as in the attached patch).
Comment 3 bardicrune 2009-04-27 18:23:36 EDT
I am experiencing the same issue in the latest rawhide update.  gdm-2.26.1-3.fc11
Comment 4 bardicrune 2009-04-28 13:52:32 EDT
I recompiled gdm with the patch supplied by Michael Young and this corrected this problem for me.
Comment 5 Michael Young 2009-04-29 06:21:31 EDT
gdm-2.26.1-6.fc11 fixes the problem for me.
Comment 6 Gerd Pokorra 2009-04-29 11:39:27 EDT
Even with gdm-2.26.1-6.fc11 I can not connect to a server from my IGEL using xdmcp. The problem is still the same: The gdm daemon stops after a short while.

[root@maritim ~]#  ps ax |grep gdm
 5865 ?        Ss     0:00 /usr/sbin/gdm-binary -nodaemon
 5937 ?        S      0:00 /usr/libexec/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display6
 5938 tty7     Rs+    0:00 /usr/bin/Xorg :5 -br -verbose -auth /var/run/gdm/auth-for-gdm-RS0fRl/database -nolisten tcp
 5943 pts/2    S+     0:00 grep gdm
[root@maritim ~]#  ps ax |grep gdm
 5947 ?        Ss     0:00 /usr/sbin/gdm-binary -nodaemon
 6019 ?        S      0:00 /usr/libexec/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display6
 6020 tty7     Rs+    0:00 /usr/bin/Xorg :5 -br -verbose -auth /var/run/gdm/auth-for-gdm-FXd1CS/database -nolisten tcp
 6025 pts/2    S+     0:00 grep gdm
[root@maritim ~]#  ps ax |grep gdm
 6029 ?        Ss     0:00 /usr/sbin/gdm-binary -nodaemon
 6077 ?        S      0:00 /usr/libexec/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display3
 6078 tty7     Rs+    0:00 /usr/bin/Xorg :2 -br -verbose -auth /var/run/gdm/auth-for-gdm-8CieF5/database -nolisten tcp
 6083 pts/2    S+     0:00 grep gdm
[root@maritim ~]#  ps ax |grep gdm
 6607 pts/2    R+     0:00 grep gdm
[root@maritim ~]#

Would it be useful if I add debugging to send more information? Or would it be the best if I post the problem to the mailing-list of gdm?
Comment 7 Gerd Pokorra 2009-05-29 05:19:36 EDT
With gdm-2.26.1-10, kernel- and a monitor connected I see the computer listed by the chooser. After choosing the computer the IGEL try to connect but after a short time the chooser is shown again. The Fedora 11 computer will still listed and the gdm daemon is still running but the connecting fails.
Comment 8 Bug Zapper 2009-06-09 10:18:47 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
Comment 9 Michael Ploujnikov 2009-06-17 11:08:15 EDT
Do you see any errors in the xorg logs?
Comment 10 Gerd Pokorra 2009-06-29 02:59:40 EDT
(In reply to comment #9)
> Do you see any errors in the xorg logs?  

I am sorry for the late answer, but I was on a journey. I do not see any errors in the xorg and gdm logs.
Comment 11 Gerd Pokorra 2009-06-29 03:29:52 EDT
(In reply to comment #9)
> Do you see any errors in the xorg logs?  

I am sorry again. There is a file /var/log/gdm/ is the IP number of my IGEL. The contents is:

X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  109 (X_ChangeHosts)
  Value in failed request:  0xd
  Serial number of failed request:  8
  Current serial number in output stream:  10
Comment 12 Gerd Pokorra 2009-06-29 06:30:43 EDT
X -query to the X-Client (Login-Server) works.

IGEL                    <->  login-server        # do not work
X -query login-server   <->  login-server        # works
Comment 13 Chen Jun 2009-08-06 05:09:37 EDT
How can I let xdmcp work on fc11? What's the steps? Thanks.
Comment 14 Chen Jun 2009-08-06 05:14:17 EDT
P.S. I run X-win32 9.1.1026s on windows xp to connect to xdmcp server (FC11). The x-window flash out black window, then disappear.
Comment 15 David Pelletier 2009-08-19 11:21:16 EDT
gdm log shows

X Error of failed request:  BadAccess (attempt to access private resource denied
  Major opcode of failed request:  109 (X_ChangeHosts)
  Serial number of failed request:  6
  Current serial number in output stream:  9
Comment 16 Albert Flügel 2009-10-25 16:21:29 EDT
Regarding this last error, it seems to me a different issue than the problem
above. In gdm-slave.c XAddHosts is called, what might fail with X11 Servers
of older version (? not absolutely sure). This failure seems to be no real
problem, so this error can be ignored. Ignoring X11 errors is a bit tricky.
The attached patch gdm-2.26.1-ign-xaddhost-fail.patch does this job.

To apply this patch, put it to the SOURCES directory and add two appropriate
lines to the gdm.spec file e.g.:

Patch30: gdm-2.26.1-ign-xaddhost-fail.patch
%patch30 -p1 -b .ign-xaddhost-fail

Or for fedora 11 use the attached spec file and rebuild.
For fedora 11 and i586 the RPM can be downloaded here, if you trust me:

Comment 17 Albert Flügel 2009-10-25 16:25:14 EDT
Created attachment 366040 [details]
patch workarounding the problem: BadValue ... integer parameter out of range for operation
Comment 18 Albert Flügel 2009-10-25 16:28:22 EDT
Created attachment 366041 [details]
Spec file for including patch gdm-2.26.1-ign-xaddhost-fail.patch into gdm-2.26.1-13.1
Comment 19 Albert Flügel 2009-10-26 10:18:25 EDT
In the meantime i'm pretty sure, that my patch regarding
"X Error of failed request:  BadValue" is valid. The value
FamilyServerInterpreted for the family member in the
XHostAddress argument of XAddHosts seems to have never
existed in XFree86 (what is quite likly running on an IGEL)
and made it's way into Xorg some time before X11R6.7 (cannot

Anyway hardcoding this without catching the case, that the
call might fail will lead to problems with older X server
implementations including all XFree86 programs.

Will try to post this to the GNOME maintainers, too.
Any hint how to do this ?
Comment 20 Ray Strode [halfline] 2009-10-26 11:13:54 EDT

You're patch can be simplified quite a bit by using gdk_error_trap_push and gdk_error_trap_pop.

Both Jon and I are upstream GNOME maintainers for GDM (Brian Cameron from Sun is the other maintainer).

The best place to post your patch is bugs.gnome.org.

If you do post it there, please mention the gnome bug number here.
Comment 21 Albert Flügel 2009-10-26 14:47:57 EDT
Regarding the question, how to make xdmcp work on Fedora 11: Howtos are probably not apppropriate in this place, nonetheless i don't mind showing, how i did it:

Edit /etc/gdm/custom.conf to read sth. like this:
# GDM configuration storage

# the default of MaxSessions seems to be 1 !?!
# increase this ^ as necessary



#---------------------------end of file------------------------

gdm must be restarted (kill gdm-binary, this logs out all users logged in via gnome).

One might want to use the X11 fontserver on this host, too. So: in /etc/X11/fs/config comment out (if active) the no-listen parameter:
#no-listen = tcp

and restart the fontserver:
/etc/init.d/xfs restart
Comment 22 Bug Zapper 2010-04-27 09:51:13 EDT
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
Comment 23 Bug Zapper 2010-06-28 08:08:51 EDT
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.