Bug 1214644

Description of problem:

As part of upcall support on the server-side, we maintain certain state to notify clients of the cache-invalidation and recall-leaselk events.

We have certain known limitations with Rebalance and Self-Heal. Details in the below link -
http://www.gluster.org/community/documentation/index.php/Features/Upcall-infrastructure#Limitations

In case of Cache-invalidation,
upcall state is not migrated and once the rebalance is finished, the file is deleted and we may falsely notify the client that file is deleted when in reality it isn't.

In case of Lease-locks,
The current understanding is that rebalance daemon seem to be opening the file in read-write mode before starting the migration. This will trigger brick-process to send recall-lease notification to the client. Though there is no data corruption, we will end up breaking lease-lk instead of migrating it.

Here rebalance is an admin driven job, but that is not the case with respect to Data tiering.


One of the proposed solutions in gluster-devel - 

http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10316

>>>>>>>>>>>>>
Migration of a file is a multi-state process. I/O is accepted at the same time migration is underway. I believe the upcall manager and the migration manager (for lack of better words) would have to coordinate. The former subsystem understands locks, and the later how to move files.

With that "coordination" in mind, a basic strategy might be something like this:

On the source, when a file is ready to be moved, the migration manager informs the upcall manager.

The upcall manager packages relevant lock information and returns it to the migrator.  The information reflects the state of posix or lease locks.

The migration manager moves the file.

The migration manager then sends the lock information as a virtual extended attribute. 

On the destination server, the upcall manager is invoked. It is passed the contents of the virtual attributes. The upcall manager rebuilds the lock state and puts the file into proper order. 

Only at that point, does the setxattr RPC return, and only then, shall the file be declared "migrated".

We would have to handle any changes to the lock state that occur when the file is in the middle of being migrated. Probably, the upcall manager would change the contents of the "package".

It is desirable to invent something that would work with both DHT and tiering (i.e. be implemented at the core dht-rebalance.c layer). And in fact, the mechanism I describe could be useful for other meta-data transfer applications. 

This is just a high level sketch, to provoke discussion and checkpoint if this is the right direction. It would take much time to sort through the details. Other ideas are welcome.
<<<<<<<<<<<<<<<<<<<<