It was found that when receiving a response from the server protocol data is not validated sufficiently. The memory for filter names is reserved right after receiving the reply. After that, filters are iterated and each individual filter name is stored in that reserved memory. The individual name lengths are not checked for validity, which means that a malicious server can reserve less memory than it will write to during each iteration.
Created libXrender tracking bugs for this issue:
Affects: fedora-all [bug 1381930]