Celeron 366, 128MB RAM, ATI@work Video-card (4MB) using Mach64-server. Graphic install worked OK with Redhat 6.1 - With "Piglet" i got an error message - and the installer returned to text mode. Later X was configured (and the X-server detected and probed) without any problems.
What kind of mouse do you have on the system? There are a couple of conditions under which the GUI installer will not start. 1) can't bring up the Xserver 2) can't determine type of mouse 3) not enough RAM on the system We have been having known bugs probing for mouse type and I am betting this is what you are seeing. Let me know the type of mouse and I will confirm here in the lab.
This mouse would fall into the category of the mice that we have been seeing problems with during the installation, so that is a good thing. We have fixed up these problems and will include the fixes in the next rev of the beta.