HornetQ present in SOA-P 5.3.ER2 has been built with netty 3.2.5.Final. The version of Netty present in the installer is 3.2.3. This makes some API incompatibilities and unit test failures.
Based on comments in JBPAPP-8396, netty 3.2.3 and netty 3.2.5 are compatible as far as HornetQ are concerned.