Bug 761892 (GLUSTER-160)

Summary: Distribute+replicate regression 2.0.2 -> 2.0.3
Product: [Community] GlusterFS Reporter: Ville Tuulos <tuulos>
Component: replicateAssignee: Vikas Gorur <vikas>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 2.0.3CC: gluster-bugs, gowda, lakshmipathi, tuulos
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTNR Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
volume file none

Description Ville Tuulos 2009-07-23 09:27:05 UTC 
I've been implementing experimental support for GlusterFS for Disco. See 

http://discoproject.org/doc/start/dfs.html

I've been playing with a replicating distributed storage, similar to the setup described here

http://www.gluster.org/docs/index.php/Mixing_DHT_and_AFR

A script is included in Disco that generates a suitable volfile given a simple list of available nodes and disks. I attached here the volfile I've been experimenting with.

The volfile works correctly in most cases with 2.0.2. Specifically, the following works:

dx2$ cd /gluster-mountpoint
dx2$ mkdir test1
dx2$ cd test1
dx2$ dd if=/dev/urandom of=testfile bs=1M count=10

so that on all nodes

dx1$ ls /gluster-mountpoint/test1/*
/gluster-mountpoint/test1/testfile

However, in v2.0.3 and v2.0.4 the same sequence produces the following output

dx1$ ls /gluster-mountpoint/test1/*
/gluster-mountpoint/test1/testfile
/gluster-mountpoint/test1/testfile

i.e. it shows incorrectly a directory entry for each replica.
Comment 1 Basavanagowda Kanur 2009-07-23 09:40:06 UTC 
the problem is with the volume file that is being used on the client side.

notice the two replicate volume definitions taken from your volume file:

volume dx03-vol1-repl
    type cluster/replicate
    option read-subvolume `echo "$(hostname)-vol1"`
    subvolumes dx03-vol1 dx01-vol1
end-volume

volume dx01-vol1-repl
    type cluster/replicate
    option read-subvolume `echo "$(hostname)-vol1"`
    subvolumes dx01-vol1 dx02-vol1
end-volume

both of the volumes have dx01-vol1 as one of the subvolumes. it is a wrong way defining replicate volume.

please ensure that you do not use same volume as subvolume of more than one replicate volume.

can you specify which version of Disco are you using? that will help us find the exact issue with the json configuration file or gluster_config.py

the same volume file might have worked as you desired, in glusterfs version 2.0.2 but the problem existed there also and was not seen. after 2.0.2 we have made enhancements to distribute translator which result in exposure of the problem.
Comment 2 Ville Tuulos 2009-07-24 00:14:18 UTC 
Thanks for a quick reply.

It is somewhat unfortunate that overlapping volumes are not allowed in replication. This makes it difficult to implement k-way replication with n nodes if n is not divisible by k, like original gluster_config.py used to do.

Anyways, I've now fixed gluster_config.py in Disco to produce non-overlapping replication groups. It indeed seems to fix this issue.