I was running indexing performance test with asynchronous indexing (using 1 or 8 indexing threads) with JDG 6.4. On one of the nodes, the performance is substantially lower than on the others: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-perf-query-indexing/26/artifact/results/html/test_test.html (click in the table on the configuration name - in first column - to show the statistics for each node separately rather than average on the whole node) Note that the histograms are not displayed correctly (bug in reporting) but it seems that some of the requests (about 5%) take 5-10 seconds and another (again, about 5%) about 1 second - this raises the average (rest of the operations has expected latency distribution).
I've done some investigation, and this is likely caused by a contention due to the combination of: * a bounded queue to store async jobs * the default configuration of the IndexWriter, that triggers frequent expensive merges of the index So in theory under certain load the queue gets filled because merge(s) blocks any index subsequent work. As the rejection policy is 'block', it can cause slowdown in the writing thread I'm saying *likely* because in my tests I did observe some slowdown in the indexing node, but far from being as drastic as yours. So could you please re-run the test again with a 'tuned' configuration to validate or deny this assumption? default.worker.execution "async" default.max_queue_length "10000" default.indexwriter.merge_factor "30" default.index_flush_interval "2000" default.indexwriter.merge_max_size "1024" default.indexwriter.ram_buffer_size "512"
Did not help, at least not much. https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-perf-query-indexing/lastSuccessfulBuild/artifact/results/html/test_test.html
This bug is fixed in JDG 6.4.0