When I run xfs, it doesn't accept connections from another machine. After stracing it, it seems that it doesn't call bind() on the AF_INET socket it creates, it only binds to the local AF_UNIX socket in /tmp.
Grab the latest errata XFree86 RPMs. These have xfs changed so that you can specify if you actually want to listen to a network port or not.