The Xnest nested X server dumps core. This is the case with the lab's RH5.2 machine and for my own 6.0 machine. On my machine Xnest is in XFree86-Xnest-3.3.3.1-49.i386.rpm
OOPS I hadn't noticed that it doesn't dump core when the actual server in which Xnest is running is another XFree86 server. It dumps core with the display being an SGI Indigo running the X server from Irix 6.3.
I was unable to get Xnest to dump core when running it withing another running XFree86 session on a local machine. We do not have an Indigo machine to test this on so you may get a faster responce by posting a request to one of the usenet newsgroups such as comp.os.linux.x.