setroubleshoot uses the rpm python bindings and is a long running service, Each time an AVC arrives it checks on the version of selinux policy, the kernel, and potentially the version of the source program and the target program to identify which version of the package was being used. If we are hit with a storm of AVC's we are seeing the memory skyrocket. We diagnosed the problem to rpm python bindings leaking. For now we are removing the bindings and going to executing rpm -qf PATH. Not an ideal solution, but we need this fix in RHEL5 and RHEL6. As well as Fedora.
Dave Malcolm, believes he has a fix for this problem. http://lists.rpm.org/pipermail/rpm-maint/2011-December/003138.html
rpm-4.9.1.2-14.fc17 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/rpm-4.9.1.2-14.fc17
rpm-4.9.1.2-6.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/rpm-4.9.1.2-6.fc16
Package rpm-4.9.1.2-6.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing rpm-4.9.1.2-6.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-3336/rpm-4.9.1.2-6.fc16 then log in and leave karma (feedback).
rpm-4.9.1.2-6.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.
rpm-4.9.1.2-14.fc17 has been pushed to the Fedora 17 stable repository. If problems still persist, please make note of it in this bug report.