I've built/installed XFree86-4.0.1-36 on my RHL 6.2, running kernel 2.2.16-3. I made a few minor changes in the xfs init-script to get xfs working, and had to slightly modify /etc/X11/xdm/Xsession (added /usr/X11R6/bin to the path). The X-server seems to generally work well when using either startx or xdm. The problem is that pam_console does not work when I log in using xdm: no /var/lock/console.lock generated and no change in /dev file ownerships. I have pam-0.72-20 installed, and pam_console works correctly when I login to the console (X not running). In the xdm-errors log I get the message "xdm error (pid #): SetPerClientControls failed", which may be related to the problem, but I'm not sure. Also, when I login using xdm, no "PAM_pwdb[#]: (xdm) session ..." messages are logged in /var/log/messages, as would be expected (for now I'm using an older /etc/pam.d/xdm file, so /etc/pam.d/system-auth isn't being used). The link to the server in /etc/X11 is X -> ../../usr/X11R6/bin/XFree86, and in /usr/X11R6/bin, X -> Xwrapper. All of the related configuration files (that I know of, especially those in /etc/X11/xdm) are either the same or similar to those on my other RHL 6.2 system running XFree86-3.3.6-20 where xdm/pam_console works fine.
I'm now running xdm using the executable (/usr/X11R6/bin/xdm) taken from XFree86-3.3.6-20. This eliminates the problem. From this I would infer that there must be a bug in XFree86-xdm-4.0.1. I played around a bit with the xdm source code, but couldn't find much -- my C-programming skills are too weak. However, it looks to me like pam gets started alright (in the function "Verify"), but the pointer value of "pamh" somehow doesn't make it into the functions in session.c, particularly the function "StartClient" where pamh is a null pointer. Also, it looks to me like the xdm-error message "xdm error (pid #): SetPerClientControls failed" is due to a bug in the xdm source code: greet.c, line 183. I could be wrong, but shouldn't the line be if (! XkbSetPerClientControls (dpy, XkbPCF_GrabsUseXKBStateMask, &value)) instead of if (XkbSetPerClientControls (dpy, XkbPCF_GrabsUseXKBStateMask, &value)) ? I hope this helps.
Certainly looks like it should be that way. And yes, the XDM pam code was extremely broken. Fixed in XFree86-4.0.1-0.44.
Is XFree86-4.0.1-0.44 accessible? I haven't been able to find it. The latest version on rawhide is 0.43 which has been there for the last month.