Description of problem:

If a client has cd-ed into a directory and is creating files inside, no lookup is sent on the directory. If in the meantime, a rebalance changes the directory layout, it is not updated in the client hence causing files to be created on the wrong subvols.

Version-Release number of selected component (if applicable):


How reproducible:
Consistently.

Steps to Reproduce:

1. Create a volume
[root@rhgs313-6 ~]# gluster volume create vol1 192.168.122.6:/bricks/brick1/vol1-1
volume create: vol1: success: please start the volume to access data
[root@rhgs313-6 ~]# gluster v start vol1
volume start: vol1: success

2. Fuse mount the volume
[root@rhgs313-6 ~]# mount -t glusterfs -s 192.168.122.6:/vol1 /mnt/fuse1
[root@rhgs313-6 ~]# cd /mnt/fuse1
[root@rhgs313-6 fuse1]# ll
total 0

3. Create a directory, cd into it and create some files.

[root@rhgs313-6 fuse1]# mkdir dir1
[root@rhgs313-6 fuse1]# cd dir1/
[root@rhgs313-6 dir1]# ll
total 0
[root@rhgs313-6 dir1]# touch file-{1..10}

4. Add a brick. Check the files on the bricks. They will all be present on the old brick.

[root@rhgs313-6 dir1]# gluster v add-brick vol1 192.168.122.6:/bricks/brick1/vol1-2 force
volume add-brick: success
[root@rhgs313-6 dir1]# ll
total 0
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-1
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-10
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-2
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-3
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-4
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-5
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-6
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-7
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-8
-rw-r--r--. 1 root root 0 Jun  4 15:34 file-9



[root@rhgs313-6 dir1]# ll /bricks/brick1/vol1-*/dir1
/bricks/brick1/vol1-1/dir1:
total 0
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-1
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-10
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-2
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-3
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-4
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-5
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-6
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-7
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-8
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-9

/bricks/brick1/vol1-2/dir1:
total 0
[root@rhgs313-6 dir1]# 


6. From a different terminal, run a rebalance on the volume

[root@rhgs313-6 ~]# gluster v rebalance vol1 start
volume rebalance: vol1: success: Rebalance on vol1 has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: 268595b3-bf1e-44b8-80af-de2a90d853a8
[root@rhgs313-6 ~]# gluster v rebalance vol1 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                3        0Bytes            10             0             0            completed        0:00:00
volume rebalance: vol1: success


[root@rhgs313-6 dir1]# 
[root@rhgs313-6 dir1]# 
[root@rhgs313-6 dir1]# ll /bricks/brick1/vol1-*/dir1
/bricks/brick1/vol1-1/dir1:
total 0
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-10
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-2
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-4
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-5
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-7
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-8
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-9

/bricks/brick1/vol1-2/dir1:
total 0
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-1
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-3
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-6
[root@rhgs313-6 dir1]# 


7. From the first terminal, create more files and check the backend bricks. All the new files will still be created only on the old brick.


[root@rhgs313-6 dir1]# touch after-file-{1..20}
[root@rhgs313-6 dir1]# ll /bricks/brick1/vol1-*/dir1
/bricks/brick1/vol1-1/dir1:
total 0
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-1
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-10
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-11
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-12
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-13
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-14
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-15
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-16
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-17
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-18
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-19
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-2
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-20
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-3
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-4
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-5
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-6
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-7
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-8
-rw-r--r--. 2 root root 0 Jun  4 15:47 after-file-9
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-10
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-2
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-4
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-5
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-7
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-8
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-9

/bricks/brick1/vol1-2/dir1:
total 0
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-1
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-3
-rw-r--r--. 2 root root 0 Jun  4 15:34 file-6
[root@rhgs313-6 dir1]# 



Using GDB and setting a breakpoint in dht_lookup, on creating a file, it can be seen that no lookup is sent on dir1. The in-memory layout for  dir1 is still the old one and has not been updated to the layout set by the rebalance.

Actual results:

Files are created on the wrong subvolume and cannot be accessed by a different client as lookup-optimize is enabled by default on the volume.

[root@rhgs313-6 ~]# mount -t glusterfs -s 192.168.122.6:/vol1 /mnt/fuse4
[root@rhgs313-6 ~]# cd /mnt/fuse4
[root@rhgs313-6 fuse4]# cd dir1/
[root@rhgs313-6 dir1]# for i in {1..20}; do stat after-file-$i >/dev/null; done
stat: cannot stat ‘after-file-2’: No such file or directory
stat: cannot stat ‘after-file-3’: No such file or directory
stat: cannot stat ‘after-file-6’: No such file or directory
stat: cannot stat ‘after-file-7’: No such file or directory
stat: cannot stat ‘after-file-8’: No such file or directory
stat: cannot stat ‘after-file-11’: No such file or directory
stat: cannot stat ‘after-file-13’: No such file or directory
stat: cannot stat ‘after-file-15’: No such file or directory
stat: cannot stat ‘after-file-17’: No such file or directory
stat: cannot stat ‘after-file-20’: No such file or directory
[root@rhgs313-6 dir1]# 




Expected results:
Files should be placed in the correct subvol.

Additional info: