Hide Forgot
I've been implementing experimental support for GlusterFS for Disco. See http://discoproject.org/doc/start/dfs.html I've been playing with a replicating distributed storage, similar to the setup described here http://www.gluster.org/docs/index.php/Mixing_DHT_and_AFR A script is included in Disco that generates a suitable volfile given a simple list of available nodes and disks. I attached here the volfile I've been experimenting with. The volfile works correctly in most cases with 2.0.2. Specifically, the following works: dx2$ cd /gluster-mountpoint dx2$ mkdir test1 dx2$ cd test1 dx2$ dd if=/dev/urandom of=testfile bs=1M count=10 so that on all nodes dx1$ ls /gluster-mountpoint/test1/* /gluster-mountpoint/test1/testfile However, in v2.0.3 and v2.0.4 the same sequence produces the following output dx1$ ls /gluster-mountpoint/test1/* /gluster-mountpoint/test1/testfile /gluster-mountpoint/test1/testfile i.e. it shows incorrectly a directory entry for each replica.
the problem is with the volume file that is being used on the client side. notice the two replicate volume definitions taken from your volume file: volume dx03-vol1-repl type cluster/replicate option read-subvolume `echo "$(hostname)-vol1"` subvolumes dx03-vol1 dx01-vol1 end-volume volume dx01-vol1-repl type cluster/replicate option read-subvolume `echo "$(hostname)-vol1"` subvolumes dx01-vol1 dx02-vol1 end-volume both of the volumes have dx01-vol1 as one of the subvolumes. it is a wrong way defining replicate volume. please ensure that you do not use same volume as subvolume of more than one replicate volume. can you specify which version of Disco are you using? that will help us find the exact issue with the json configuration file or gluster_config.py the same volume file might have worked as you desired, in glusterfs version 2.0.2 but the problem existed there also and was not seen. after 2.0.2 we have made enhancements to distribute translator which result in exposure of the problem.
Thanks for a quick reply. It is somewhat unfortunate that overlapping volumes are not allowed in replication. This makes it difficult to implement k-way replication with n nodes if n is not divisible by k, like original gluster_config.py used to do. Anyways, I've now fixed gluster_config.py in Disco to produce non-overlapping replication groups. It indeed seems to fix this issue.