If you have the X-server running for long periods of time it slowly grows in memory-consumption. The growth rate is low and only noticable if you run X for several weeks. I have noticed this behavior in XFree86 3.3.5.
this is most likely due actually to programs that you are using. Some programs allocate pixmaps in the the server and then never free them up. This causes the server to grow in size. XScreenSaver is a frequent culprit. There is no real fix for this on the horizon.