Fetching groups via REST does not scale. This is due to the code always fetching all implicit and explicit resources for the fetched groups. AbstractRestBean.fillGroup() is the problem, it pulls in a bunch of lazy-load data, most notably, all of the group's resources. Note that fillGroup is used in some other rest services as well. To recreate just rhqhost:7080/rest/group.html, as the group membership grows so will the resources consumed, until it falls over.
PR sent: https://github.com/rhq-project/rhq/pull/255