Bug 1716874
| Summary: | DHT: Once client does a cd into a directory, a lookup not sent on it | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nithya Balachandran <nbalacha> |
| Component: | distribute | Assignee: | Susant Kumar Palai <spalai> |
| Status: | CLOSED DUPLICATE | QA Contact: | Sayalee <saraut> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | rhgs-3.5 | CC: | amukherj, moagrawa, rhs-bugs, saraut, sheggodu, storage-qa-internal, vdas |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-16 07:35:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** Bug 1724094 has been marked as a duplicate of this bug. *** |
Description of problem: If a client has cd-ed into a directory and is creating files inside, no lookup is sent on the directory. If in the meantime, a rebalance changes the directory layout, it is not updated in the client hence causing files to be created on the wrong subvols. Version-Release number of selected component (if applicable): How reproducible: Consistently. Steps to Reproduce: 1. Create a volume [root@rhgs313-6 ~]# gluster volume create vol1 192.168.122.6:/bricks/brick1/vol1-1 volume create: vol1: success: please start the volume to access data [root@rhgs313-6 ~]# gluster v start vol1 volume start: vol1: success 2. Fuse mount the volume [root@rhgs313-6 ~]# mount -t glusterfs -s 192.168.122.6:/vol1 /mnt/fuse1 [root@rhgs313-6 ~]# cd /mnt/fuse1 [root@rhgs313-6 fuse1]# ll total 0 3. Create a directory, cd into it and create some files. [root@rhgs313-6 fuse1]# mkdir dir1 [root@rhgs313-6 fuse1]# cd dir1/ [root@rhgs313-6 dir1]# ll total 0 [root@rhgs313-6 dir1]# touch file-{1..10} 4. Add a brick. Check the files on the bricks. They will all be present on the old brick. [root@rhgs313-6 dir1]# gluster v add-brick vol1 192.168.122.6:/bricks/brick1/vol1-2 force volume add-brick: success [root@rhgs313-6 dir1]# ll total 0 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-1 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-10 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-2 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-3 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-4 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-5 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-6 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-7 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-8 -rw-r--r--. 1 root root 0 Jun 4 15:34 file-9 [root@rhgs313-6 dir1]# ll /bricks/brick1/vol1-*/dir1 /bricks/brick1/vol1-1/dir1: total 0 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-1 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-10 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-2 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-3 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-4 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-5 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-6 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-7 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-8 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-9 /bricks/brick1/vol1-2/dir1: total 0 [root@rhgs313-6 dir1]# 6. From a different terminal, run a rebalance on the volume [root@rhgs313-6 ~]# gluster v rebalance vol1 start volume rebalance: vol1: success: Rebalance on vol1 has been started successfully. Use rebalance status command to check status of the rebalance process. ID: 268595b3-bf1e-44b8-80af-de2a90d853a8 [root@rhgs313-6 ~]# gluster v rebalance vol1 status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 3 0Bytes 10 0 0 completed 0:00:00 volume rebalance: vol1: success [root@rhgs313-6 dir1]# [root@rhgs313-6 dir1]# [root@rhgs313-6 dir1]# ll /bricks/brick1/vol1-*/dir1 /bricks/brick1/vol1-1/dir1: total 0 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-10 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-2 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-4 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-5 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-7 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-8 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-9 /bricks/brick1/vol1-2/dir1: total 0 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-1 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-3 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-6 [root@rhgs313-6 dir1]# 7. From the first terminal, create more files and check the backend bricks. All the new files will still be created only on the old brick. [root@rhgs313-6 dir1]# touch after-file-{1..20} [root@rhgs313-6 dir1]# ll /bricks/brick1/vol1-*/dir1 /bricks/brick1/vol1-1/dir1: total 0 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-1 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-10 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-11 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-12 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-13 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-14 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-15 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-16 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-17 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-18 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-19 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-2 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-20 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-3 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-4 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-5 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-6 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-7 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-8 -rw-r--r--. 2 root root 0 Jun 4 15:47 after-file-9 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-10 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-2 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-4 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-5 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-7 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-8 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-9 /bricks/brick1/vol1-2/dir1: total 0 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-1 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-3 -rw-r--r--. 2 root root 0 Jun 4 15:34 file-6 [root@rhgs313-6 dir1]# Using GDB and setting a breakpoint in dht_lookup, on creating a file, it can be seen that no lookup is sent on dir1. The in-memory layout for dir1 is still the old one and has not been updated to the layout set by the rebalance. Actual results: Files are created on the wrong subvolume and cannot be accessed by a different client as lookup-optimize is enabled by default on the volume. [root@rhgs313-6 ~]# mount -t glusterfs -s 192.168.122.6:/vol1 /mnt/fuse4 [root@rhgs313-6 ~]# cd /mnt/fuse4 [root@rhgs313-6 fuse4]# cd dir1/ [root@rhgs313-6 dir1]# for i in {1..20}; do stat after-file-$i >/dev/null; done stat: cannot stat ‘after-file-2’: No such file or directory stat: cannot stat ‘after-file-3’: No such file or directory stat: cannot stat ‘after-file-6’: No such file or directory stat: cannot stat ‘after-file-7’: No such file or directory stat: cannot stat ‘after-file-8’: No such file or directory stat: cannot stat ‘after-file-11’: No such file or directory stat: cannot stat ‘after-file-13’: No such file or directory stat: cannot stat ‘after-file-15’: No such file or directory stat: cannot stat ‘after-file-17’: No such file or directory stat: cannot stat ‘after-file-20’: No such file or directory [root@rhgs313-6 dir1]# Expected results: Files should be placed in the correct subvol. Additional info: