dup of bug 1026100?
Probably a duplicate of bug 1028966. To verify this we would need to see at least the content of /var/log/messages file of the engine machine, to see the amount of RSS that it was actually consuming when it was killed by the OOM killer. I strongly recommend to run all the non trivial tests/jobs in RHEL 6.5 with the PKCS#11 provider disabled, as we positively know that it generates a serious leak of off-heap memory. See here for details: https://bugzilla.redhat.com/show_bug.cgi?id=1028966#c27
Please attach /var/log/messages
Created attachment 831467 [details] /var/log/messages
According to the messages file in comment 5 the engine was consuming 2.69 GiB of RSS when it was killed. This is the same situation we saw in bug 1028966, so I'm closing it as a duplicate of that bug. *** This bug has been marked as a duplicate of bug 1028966 ***