XFree86 4.0.1 font server neatly installed itself to listen on a TCP port - totally ignoring the current setting in my XF86Config. Besides this annoying behaviour it opens up a possibility for remote exploitation. I find it, on my single user machine, a lot safer not to have another open port. XFS do indeed support a none-networked port mode - why not use that instead?
This should be fixed in the rawhide XFree86 packages.