I am currently having to firewall any robot that gets into /webx as they all seem to be getting stuck and then the app servers go into load averages of 15-20. I need some help or suggestions here thanks.
updated robots.txt to keep the robots going into infinite loops.