Created attachment 314341 [details]
modified cpg.c to make mcast asynchrnous.

Description of problem:

cpg_mcast_joined is a synchronous call, i.e. the caller has to wait for the IPC round-trip to the aisexec daemon before proceeding. An asynchronous call from the client library would give much better throughput. This is to support the MRG project in order to use CPG as its replication protocol. Further work needed to determine the extent of the benefit, but inital experiment below showed a threefold speedup.

I hacked lib/cpg.c (attached) to ignore the response in mcast_joined and
instead retrieve and discard them in the dispatch poll. This almost
triples throughput of the cpgbench test on my laptop.

I'm not proposing this hack as an implementation, it was just to get an
idea of possible performance impact. At least the following things are
wrong with it:
 - async_mcast should be a new API, existing sync mcast should remain.
 - wrong locking around response_fd
 - polling 2 fd's in dispatch so cpg_fd_get() is no longer sufficient. 
 - cpgbench goes into flow control quickly (not surprising) but then
never gets out, I think I missed something about resetting flow control.

A neater solution might be to drop the responses at the daemon for
successful mcasts and send an error response to the dispatch_fd for
failed mcasts so there's only 1 fd that needs to be polled. Flow control changes as a result of async mcasts would also need to be sent to dispatch_fd.

Version-Release number of selected component (if applicable):

openais-0.80.3-17.fc9.i386

Steps to Reproduce:

Run cpgbench against normal AIS and with the attached changes to see the throughput difference. I saw almost a factor of 3 improvement.


Additional info: