Bug 764511 (GLUSTER-2779)

In a slightly different but related world, this feature is called "hinted handoff" based on an Amazon paper about their Dynamo system.  The key questions are:

* How do clients find the "hinted" location?

* How do the writes propagate to the "correct" server when it comes back up?

* What happens if that server *never* comes back up?

These are significant challenges even within the Dynamo world of atomic key/value pairs and weak consistency.  They're likely to be even more challenging in the context of a POSIX filesytem like GlusterFS, and the solution to the write-propagation issue in particular is likely to require a kind of server-to-server communication that GlusterFS has mostly eschewed until now.  Lastly, I don't think the current elastic hashing implementation in DHT lends itself very well to this.  For CloudFS I have written a translator that takes a more Dynamo-like approach in which additional or alternative bricks will automatically be found and used if the primary brick for a file is down.  This might support hinted handoff a little better, in addition to addressing issues of directory-operation scalability, replication performance, etc.  I'd be glad to collaborate with others on moving this forward, as it's currently stalled behind other CloudFS priorities.

For now and for the immediate future, I think the solution to this problem will be to run distribute over replicate, so that single-node failures are never visible to DHT.