Bug 1064283
| Summary: | DHT + rebalance : rename of files fails with an error 'No such file or directory' even though files are present. | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Rachana Patel <racpatel> | |
| Component: | distribute | Assignee: | Nithya Balachandran <nbalacha> | |
| Status: | CLOSED DEFERRED | QA Contact: | storage-qa-internal <storage-qa-internal> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 2.1 | CC: | mzywusko, nlevinki, nsathyan, spalai, vbellur | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1286127 (view as bug list) | Environment: | ||
| Last Closed: | 2015-11-27 11:38:57 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1286127, 1395217 | |||
Volume was mounted on multiple mountpoint using FUSE/nfs and rename fails on all mountpoint for same files. On creating new mount or unmount volume and mount again then rename operation is not failing for the same file. Cloning this to 3.1. To be fixed in future. |
Description of problem: On Distributed volume, add new bricks and start rebalance process. whle rebealance is in progress tried to rename files and for few files rename operation failed with error 'No such file or directory'. on lookup from mount point we can find those files. log has error:- [2014-02-12 08:30:36.603608] W [dht-rename.c:556:dht_rename_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf1: rename on dht1-client-3 failed (No such file or directory) Version-Release number of selected component (if applicable): 3.4.0.59rhs-1.el6rhs.x86_64 How reproducible: 2/2 Steps to Reproduce: 1. create DHT volume and FUSE mount volume. 2. create Directories and files from mount point. 3. add bricks to volume and start rebalance process. 4. while rebalance is in progress rename files. for few files it fails with 'No such file or directory' 5. On mount point verify those files are present Actual results: rename for few files fails with error Expected results: rename should not fail with 'No such file or directory' Additional info: gluster v info [root@OVM1 tmp]# gluster volume info gluster Volume Name: dht1 Type: Distribute Volume ID: c0abd5ee-2f93-4de8-a287-178fde6e2283 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 10.70.35.187:/rhs/brick1/d1 Brick2: 10.70.35.187:/rhs/brick1/d2 Brick3: 10.70.35.228:/rhs/brick1/d1 ######################### after add brick:- [root@OVM1 tmp]# gluster volume info Volume Name: dht1 Type: Distribute Volume ID: c0abd5ee-2f93-4de8-a287-178fde6e2283 Status: Started Number of Bricks: 5 Transport-type: tcp Bricks: Brick1: 10.70.35.187:/rhs/brick1/d1 Brick2: 10.70.35.187:/rhs/brick1/d2 Brick3: 10.70.35.228:/rhs/brick1/d1 Brick4: 10.70.35.228:/rhs/brick1/212 Brick5: 10.70.35.212:/rhs/brick1/d1 ############################ rename files while rebalance in progress [u1@rhs-client22 flat1]$ for j in {1..20}; do cd $j ;for i in `ls` ; do mv $i mv$i ; done ; cd .. ; done mv: cannot move `ddf1' to `mvddf1': No such file or directory mv: cannot move `ddf90' to `mvddf90': No such file or directory mv: cannot move `ddf98' to `mvddf98': No such file or directory ^C [u1@rhs-client22 1]$ ls ddf1 ddf1 [u1@rhs-client22 1]$ ls -l ddf1 -rw-rw-r--. 1 u1 u1 10485760 Feb 6 04:04 ddf1 [u1@rhs-client22 1]$ ls -l ddf90 -rw-rw-r--. 1 u1 u1 10485760 Feb 6 04:14 ddf90 [u1@rhs-client22 1]$ ls -l ddf98 -rw-rw-r--. 1 u1 u1 10485760 Feb 6 04:15 ddf98 <log snippet> [2014-02-12 08:30:36.603608] W [dht-rename.c:556:dht_rename_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf1: rename on dht1-client-3 failed (No such file or directory) [2014-02-12 08:30:36.605151] W [fuse-bridge.c:2171:fuse_rename_cbk] 0-glusterfs-fuse: 6649: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b> /1/ddf1 -> <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/mvddf1 => -1 (No such file or directory) [2014-02-12 08:30:36.619568] W [dht-rename.c:381:dht_rename_unlink_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf10 : unlink on dht1-client-3 failed (No such file or directory) [2014-02-12 08:30:41.984739] W [dht-rename.c:381:dht_rename_unlink_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf2: unlink on dht1-client-4 failed (No such file or directory) [2014-02-12 08:30:41.997854] W [dht-rename.c:381:dht_rename_unlink_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf21 : unlink on dht1-client-4 failed (No such file or directory) [2014-02-12 08:30:42.013382] W [dht-rename.c:381:dht_rename_unlink_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf22 : unlink on dht1-client-4 failed (No such file or directory) [2014-02-12 08:30:45.111688] W [dht-rename.c:381:dht_rename_unlink_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf26 : unlink on dht1-client-4 failed (No such file or directory) ... [2014-02-12 08:31:08.090439] W [dht-rename.c:556:dht_rename_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf90: rename on dht1-client-4 failed (No such file or directory) [2014-02-12 08:31:08.091176] W [fuse-bridge.c:2171:fuse_rename_cbk] 0-glusterfs-fuse: 7030: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf90 -> <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/mvddf90 => -1 (No such file or directory) [2014-02-12 08:31:09.059524] W [client-rpc-fops.c:2385:client3_3_rename_cbk] 1-dht1-client-3: remote operation failed: No such file or directory [2014-02-12 08:31:09.059588] W [dht-rename.c:556:dht_rename_cbk] 1-dht1-dht: <gfid:b4dae3fe-c5a7-4f5b-bac7-5e00b61a6a9b>/1/ddf98: rename on dht1-client-3 failed (No such file or directory)