Bug 1198556

William Burns <wburns> updated the status of jira ISPN-5264 to Coding In Progress

William Burns <wburns> updated the status of jira ISPN-5265 to Coding In Progress

William Burns <wburns> updated the status of jira ISPN-5265 to Open

William Burns <wburns> updated the status of jira ISPN-5266 to Coding In Progress

Tristan Tarrant <ttarrant> updated the status of jira ISPN-5263 to Closed

William Burns <wburns> updated the status of jira ISPN-5264 to Reopened

PR for putAll https://github.com/infinispan/jdg/pull/613

This issue is to handle the putAll improvements for both embedded and remote caches.

William Burns <wburns> updated the status of jira ISPN-5265 to Coding In Progress

Here are some measurements:
Performance comparison of putAll() operation with ER1, ER2, it looks it is fast when data are loaded on single node. I have tried following configurations:
  a) HR client - 500MBs of data single node, ER2 is 3-4x faster then ER1, 1 entry is 1024Bs, 20bytes for key and rest for value.
  b) HR client - 200MBs of data cluster of 2 nodes, NO PERFORMANCE IMPROVEMENT FOUND
    ON ER1 [Data loading took 0 hours-0 minutes-58 seconds-325 milliseconds]
    ON ER2 [Data loading took 0 hours-0 minutes-58 seconds-197 milliseconds] 
  c) HR client- 500MBs, increased heap size to 3Gbs, with ER2 test fails with OOM so NO PERFORMANCE IMPROVEMENT
    ON ER1 [Data loading took 0 hours-1 minutes-55 seconds-678 milliseconds]
    ON ER2 OOM occurs
         ERROR [org.jgroups.protocols.UNICAST3] (OOB-21,shared=udp) JGRP000039: JDG2/clustered: failed to deliver OOB message [dst: JDG2/clustered, src: JDG1/clustered (3 headers), size=60000 bytes, flags=OOB|DONT_BUNDLE|NO_TOTAL_ORDER]: java.lang.OutOfMemoryError: Java heap space
         ERROR [org.jgroups.protocols.UNICAST3] (OOB-16,shared=udp) JGRP000039: JDG1/clustered: failed to deliver OOB message [dst: JDG1/clustered, src: JDG2/clustered (3 headers), size=60000 bytes, flags=OOB|DONT_BUNDLE|NO_TOTAL_ORDER]: java.lang.OutOfMemoryError: Java heap space
  d) Library mode with EAP, 200MBs, NO PERFORMANCE IMPROVEMENT
    ON ER1 [Data loading took 0 hours-0 minutes-0 seconds-869 milliseconds] 
    ON ER2 [Data loading took 0 hours-0 minutes-0 seconds-897 milliseconds]
  e) Library mode with EAP, 500MBs, NO PERFORMANCE IMPROVEMENT   
    ON ER1 [Data loading took 0 hours-0 minutes-2 seconds-930 milliseconds]
    ON ER2 [Data loading took 0 hours-0 minutes-2 seconds-853 milliseconds]

Response from wburns:

If the library mode is with only 2 nodes, that is to be expected.  The changes for embedded should only improve performance for DIST only when you have more nodes than numOwners.

The remote is odd, it should be substantially faster if you have a good amount of entries > 10 in pretty much all cases.

The extra memory usage is to be expected, unfortunately.  Compared to ER1 it sequentially sent each entry so there was a smaller message overhead, where as now it has to hold the entire map in a message.  So 3 GB for both servers and client with 500MB being inserted is not going to be enough.

I will dig a little to see why it wasn't performing properly in the 200MBs range though.

Martin:
We'll probably need to run a few more tests with the number of nodes > numOwners.

Hi Will,
in order to proceed with verification of this BZ, could you please specify more precisely what should be outcome from this issue, ideally in terms under which circumstances the performance should increase and rough estimate of improvement (i.e. which improvement you want to achieve)?
Thanks
Vojta

(In reply to Martin Gencur from comment #10)
> Here are some measurements:
> Performance comparison of putAll() operation with ER1, ER2, it looks it is
> fast when data are loaded on single node. I have tried following
> configurations:
>   a) HR client - 500MBs of data single node, ER2 is 3-4x faster then ER1, 1
> entry is 1024Bs, 20bytes for key and rest for value.
>   b) HR client - 200MBs of data cluster of 2 nodes, NO PERFORMANCE
> IMPROVEMENT FOUND
>     ON ER1 [Data loading took 0 hours-0 minutes-58 seconds-325 milliseconds]
>     ON ER2 [Data loading took 0 hours-0 minutes-58 seconds-197 milliseconds] 
>   c) HR client- 500MBs, increased heap size to 3Gbs, with ER2 test fails
> with OOM so NO PERFORMANCE IMPROVEMENT
>     ON ER1 [Data loading took 0 hours-1 minutes-55 seconds-678 milliseconds]
>     ON ER2 OOM occurs
>          ERROR [org.jgroups.protocols.UNICAST3] (OOB-21,shared=udp)
> JGRP000039: JDG2/clustered: failed to deliver OOB message [dst:
> JDG2/clustered, src: JDG1/clustered (3 headers), size=60000 bytes,
> flags=OOB|DONT_BUNDLE|NO_TOTAL_ORDER]: java.lang.OutOfMemoryError: Java heap
> space
>          ERROR [org.jgroups.protocols.UNICAST3] (OOB-16,shared=udp)
> JGRP000039: JDG1/clustered: failed to deliver OOB message [dst:
> JDG1/clustered, src: JDG2/clustered (3 headers), size=60000 bytes,
> flags=OOB|DONT_BUNDLE|NO_TOTAL_ORDER]: java.lang.OutOfMemoryError: Java heap
> space
>   d) Library mode with EAP, 200MBs, NO PERFORMANCE IMPROVEMENT
>     ON ER1 [Data loading took 0 hours-0 minutes-0 seconds-869 milliseconds] 
>     ON ER2 [Data loading took 0 hours-0 minutes-0 seconds-897 milliseconds]
>   e) Library mode with EAP, 500MBs, NO PERFORMANCE IMPROVEMENT   
>     ON ER1 [Data loading took 0 hours-0 minutes-2 seconds-930 milliseconds]
>     ON ER2 [Data loading took 0 hours-0 minutes-2 seconds-853 milliseconds]
> 
> Response from wburns:
> 
> If the library mode is with only 2 nodes, that is to be expected.  The
> changes for embedded should only improve performance for DIST only when you
> have more nodes than numOwners.
> 
> The remote is odd, it should be substantially faster if you have a good
> amount of entries > 10 in pretty much all cases.
> 
> The extra memory usage is to be expected, unfortunately.  Compared to ER1 it
> sequentially sent each entry so there was a smaller message overhead, where
> as now it has to hold the entire map in a message.  So 3 GB for both servers
> and client with 500MB being inserted is not going to be enough.
> 
> I will dig a little to see why it wasn't performing properly in the 200MBs
> range though.
> 
> Martin:
> We'll probably need to run a few more tests with the number of nodes >
> numOwners.

To further clarify the 3 GB note I put here.  The issue was that both servers and the client were ran on the same JVM.  So in that case 3 GB for all of these combined is not enough.  Normally these would be on separate JVMs and a much smaller heap could be utilized.  Just a bit more than double the target putAll map size should be sufficient (this would allow for the map itself and the message containing the map being fully serialized).