Bug 764263 (GLUSTER-2531)

Problem:  I have 15 bricks in my gluster config.  Occasionally when I create a directory in my fuse gluster mount a directory on one of the  bricks will be owned by root instead of me.  This causes a permission denied 1 out of 15 times when I try and put files in that directory.   I've only seen this happen under heavy loads (cluster pushing/pulling  lots of files)

Current setup:

  - 5x data servers, each with 3 data partitions (6 TB each) running RHEL6 and gluster 3.1.2.  So that gives me a total of 15 bricks (5x3).  Each server has 3 copper gigabit connections channel bonded together.

- Gluster is shared to my cluster using FUSE.  Cluster is ROCKS 5.2 with approximately 24 computers and 232 cpus.

- The data files we get and put are usually between 10MB and 150MB.  With full out processing the cluster sustains data transfer rates of around 6.5 gbits.  I'd like to see that more around 10 gbits, but 6.5 is pretty good.  The cpu load on the data server is around 65% due to the software raid. So overall I'm happy with the performance of gluster.

------------------
Now on to the problem
------------------

The jobs my cluster are running involving downloading a couple files, processing them for a minute or two and then writing results back to gluster.  The data files get put into a directory that gets created if it doesn't already exist.  288 products will go into one directory from different compute nodes.  The problem is sometimes when the directory gets created that it will not get the proper permissions in the gluster private directory on one of the bricks.  So if I look in the gluster private directory on all 15 bricks, sometimes I will see one directory owned by root where as on the other 14 bricks they are  
all owned by my user.  Because it is owned by root my jobs that attempt to push files that would fall on that brick return with a permission denied.

I should mention this doesn't happen a lot.  In my processing job which is working on one month of data this usually only happens for 1 day out of the month.  Each day is its own directory and there are 4   different product directories.  So I'm basically creating 120 directories and storing 288 files per directory.  Out of those 120 directories only one will fail to get the proper permissions on one of the directories.  This makes it a really tough problem to debug.

My guess is that 2 cluster machines try and make the same directory at the same time and there is a race condition that causes one directory on one brick to get the failed permissions.  Any ideas?  If I can provide any other info that may help you debug this, please don't hesitate to ask.

I did just notice that 3.1.3 came out yesterday, I will test this on 3.1.3 but I see nothing in the changelogs to suggest that it was fixed.