1128747 – rebalance :- while rebalance was in progress, recursive copy of Directory gave error 'No such file or directory' and data was not copied

Bug 1128747 - rebalance :- while rebalance was in progress, recursive copy of Directory gave error 'No such file or directory' and data was not copied

Summary: rebalance :- while rebalance was in progress, recursive copy of Directory gav...

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	distribute
Sub Component:
Version:	2.1
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Bug Updates Notification Mailing List
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1286109
TreeView+	depends on / blocked

Reported:	2014-08-11 13:11 UTC by Rachana Patel
Modified:	2015-11-27 10:52 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1286109 (view as bug list)
Environment:
Last Closed:	2015-11-27 10:52:24 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Rachana Patel 2014-08-11 13:11:01 UTC

Description of problem:
=======================
while rebalance wa sin progress, started copying data using cp -R where source and destination both are on same mount point.
for one Directory it gave error :-

[root@localhost nfs]# cp -Ra fuse cpnfs
cp: cannot access `fuse/linux-2.6.32.63/Documentation/video4linux/bttv 	: No such file or directory


when copy was done found that source has :-

[root@localhost mio]# ls -l fuse/linux-2.6.32.63/Documentation/video4linux/bttv
total 63
-rw-rw-r-- 1 root root 26708 Jun 18 18:26 Cards
-rw-rw-r-- 1 root root   495 Jun 18 18:26 CONTRIBUTORS
-rw-rw-r-- 1 root root   881 Jun 18 18:26 ICs
-rw-rw-r-- 1 root root  6326 Jun 18 18:26 Insmod-options
-rw-rw-r-- 1 root root   543 Jun 18 18:26 MAKEDEV
-rw-rw-r-- 1 root root   212 Jun 18 18:26 Modprobe.conf
-rw-rw-r-- 1 root root   354 Jun 18 18:26 Modules.conf
-rw-rw-r-- 1 root root  2659 Jun 18 18:26 PROBLEMS
-rw-rw-r-- 1 root root  3156 Jun 18 18:26 README
-rw-rw-r-- 1 root root  2943 Jun 18 18:26 README.freeze
-rw-rw-r-- 1 root root  3765 Jun 18 18:26 README.quirks
-rw-rw-r-- 1 root root  1493 Jun 18 18:26 README.WINVIEW
-rw-rw-r-- 1 root root  5885 Jun 18 18:26 Sound-FAQ
-rw-rw-r-- 1 root root   169 Jun 18 18:26 Specs
-rw-rw-r-- 1 root root   832 Jun 18 18:26 THANKS
-rw-rw-r-- 1 root root  3164 Jun 18 18:26 Tuners

but destination does not have any files :-

[root@localhost mio]# ls -l cpnfs/linux-2.6.32.63/Documentation/video4linux/bttv
total 0



Version-Release number of selected component (if applicable):
=============================================================
3.4.0.59rhs-1.2.toyota.hotfix.el6rhs.x86_64


How reproducible:
=================
intermittent

Steps to Reproduce:
===================
1. create and mount Dist-rep(18x2) volume 
2. create files and Directories on it.
3. add 12 new bricks in that volume.
4. start rebalance with start force option
5. keep sending lookup on all files and Directory from another mount point
6. start copying existing data to same mount point

[root@localhost nfs]# ls
fuse  in  linux-2.6.32.63
[root@localhost nfs]# cp -Ra fuse cpnfs
cp: cannot access `fuse/linux-2.6.32.63/Documentation/video4linux/bttv 	: No such file or directory


Actual results:
===============
copy operation give error 'No such file or Directory' and did not copy data from that Directory,


Expected results:
================
copy operation should not fail with error 'No such file or directory' when source Directory is present

Additional info:
================

log snippet

less /var/log/glusterfs/nfs.log | grep 'video4linux/bttv' -B 3 -A 3| less

[2014-08-11 07:19:47.324758] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/bttv/README.WINVIEW missing on subvol mio-replicate-12
[2014-08-11 07:19:47.337965] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-12cached_subvol null
[2014-08-11 07:19:47.449534] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/cafe_ccic missing on subvol mio-replicate-15
[2014-08-11 07:19:47.454784] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-15cached_subvol null
[root@localhost ~]# less /var/log/glusterfs/nfs.log | grep 'video4linux/bttv' -B 3 -A 3| less
[2014-08-11 04:51:21.633198] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-10cached_subvol null
[2014-08-11 04:51:21.687314] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/linux-2.6.32.63/Documentation/video4linux/README.ivtv missing on subvol mio-replicate-10
[2014-08-11 04:51:21.694999] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-10cached_subvol null
[2014-08-11 04:51:21.713011] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/linux-2.6.32.63/Documentation/video4linux/bttv missing on subvol mio-replicate-0
[2014-08-11 04:51:21.719103] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-0cached_subvol null
[2014-08-11 04:51:21.804145] I [dht-layout.c:646:dht_layout_normalize] 0-mio-dht: found anomalies in <gfid:4bef9d3b-4db3-4114-906f-dd16c62f4a6e>. holes=2 overlaps=1 missing=6 down=0 misc=0
[2014-08-11 04:51:21.805752] W [client-rpc-fops.c:1369:client3_3_access_cbk] 0-mio-client-44: remote operation failed: No such file or directory
--
[2014-08-11 06:05:15.959483] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-17cached_subvol null
[2014-08-11 06:05:16.030582] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/linux-2.6.32.63/Documentation/video4linux/Zoran missing on subvol mio-replicate-18
[2014-08-11 06:05:16.037981] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-18cached_subvol null
[2014-08-11 06:05:16.342160] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/linux-2.6.32.63/Documentation/video4linux/bttv missing on subvol mio-replicate-3
[2014-08-11 06:05:16.361110] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-3cached_subvol null
[2014-08-11 06:05:16.738256] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/linux-2.6.32.63/Documentation/video4linux/bttv/README missing on subvol mio-replicate-8
[2014-08-11 06:05:16.746995] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-8cached_subvol null
[2014-08-11 06:05:16.802614] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/linux-2.6.32.63/Documentation/video4linux/bttv/Cards missing on subvol mio-replicate-1
[2014-08-11 06:05:16.816226] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-1cached_subvol null
[2014-08-11 06:05:16.905031] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/linux-2.6.32.63/Documentation/video4linux...skipping...
[2014-08-11 07:19:46.669083] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-17cached_subvol null
[2014-08-11 07:19:46.867137] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/bttv/Modprobe.conf missing on subvol mio-replicate-21
[2014-08-11 07:19:46.878172] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-21cached_subvol null
[2014-08-11 07:19:47.020435] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/bttv/Modules.conf missing on subvol mio-replicate-5
[2014-08-11 07:19:47.032055] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-5cached_subvol null
[2014-08-11 07:19:47.074980] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/bttv/CONTRIBUTORS missing on subvol mio-replicate-6
[2014-08-11 07:19:47.084459] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-6cached_subvol null
[2014-08-11 07:19:47.115305] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/bttv/README.freeze missing on subvol mio-replicate-6
[2014-08-11 07:19:47.124665] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-6cached_subvol null
[2014-08-11 07:19:47.157068] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/bttv/Insmod-options missing on subvol mio-replicate-7
[2014-08-11 07:19:47.168632] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-7cached_subvol null
[2014-08-11 07:19:47.221350] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/bttv/ICs missing on subvol mio-replicate-12
[2014-08-11 07:19:47.273445] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-12cached_subvol null
[2014-08-11 07:19:47.324758] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/bttv/README.WINVIEW missing on subvol mio-replicate-12
[2014-08-11 07:19:47.337965] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-12cached_subvol null
[2014-08-11 07:19:47.449534] I [dht-common.c:1738:dht_lookup_cbk] 0-mio-dht: Entry /cpnfs/in/anotherin/linux-2.6.32.63/Documentation/video4linux/cafe_ccic missing on subvol mio-replicate-15
[2014-08-11 07:19:47.454784] I [dht-common.c:1010:dht_lookup_everywhere_done] 0-mio-dht: _RR_ STATUS: hashed_subvol mio-replicate-15cached_subvol null

================

Comment 1 Rachana Patel 2014-08-11 13:11:41 UTC

volume info :-
 
Volume Name: mio
Type: Distributed-Replicate
Volume ID: 3aea4fc9-b128-4af7-8692-d2ac268bbea7
Status: Started
Number of Bricks: 24 x 2 = 48
Transport-type: tcp
Bricks:
Brick1: 192.168.12.13:/brick/mio0
Brick2: 192.168.12.14:/brick/mio1
Brick3: 192.168.12.15:/brick/mio2
Brick4: 192.168.12.16:/brick/mio3
Brick5: 192.168.12.17:/brick/mio4
Brick6: 192.168.12.18:/brick/mio5
Brick7: 192.168.12.19:/brick/mio6
Brick8: 192.168.12.22:/brick/mio7
Brick9: 192.168.12.23:/brick/mio8
Brick10: 192.168.12.24:/brick/mio9
Brick11: 192.168.12.25:/brick/mio10
Brick12: 192.168.12.26:/brick/mio11
Brick13: 192.168.12.27:/brick/mio12
Brick14: 192.168.12.28:/brick/mio13
Brick15: 192.168.12.29:/brick/mio14
Brick16: 192.168.12.32:/brick/mio15
Brick17: 192.168.12.33:/brick/mio16
Brick18: 192.168.12.34:/brick/mio17
Brick19: 192.168.12.35:/brick/mio18
Brick20: 192.168.12.36:/brick/mio19
Brick21: 192.168.12.37:/brick/mio20
Brick22: 192.168.12.38:/brick/mio21
Brick23: 192.168.12.39:/brick/mio22
Brick24: 192.168.12.42:/brick/mio23
Brick25: 192.168.12.43:/brick/mio24
Brick26: 192.168.12.44:/brick/mio25
Brick27: 192.168.12.45:/brick/mio26
Brick28: 192.168.12.46:/brick/mio27
Brick29: 192.168.12.47:/brick/mio28
Brick30: 192.168.12.48:/brick/mio29
Brick31: 192.168.12.49:/brick/mio30
Brick32: 192.168.12.62:/brick/mio31
Brick33: 192.168.12.63:/brick/mio32
Brick34: 192.168.12.64:/brick/mio33
Brick35: 192.168.12.65:/brick/mio34
Brick36: 192.168.12.66:/brick/mio35
Brick37: 192.168.12.13:/brick/mionew0
Brick38: 192.168.12.14:/brick/mionew1
Brick39: 192.168.12.15:/brick/mionew2
Brick40: 192.168.12.16:/brick/mionew3
Brick41: 192.168.12.17:/brick/mionew4
Brick42: 192.168.12.18:/brick/mionew5
Brick43: 192.168.12.19:/brick/mionew6
Brick44: 192.168.12.22:/brick/mionew7
Brick45: 192.168.12.23:/brick/mionew8
Brick46: 192.168.12.24:/brick/mionew9
Brick47: 192.168.12.25:/brick/mionew10
Brick48: 192.168.12.26:/brick/mionew11
Options Reconfigured:
server.allow-insecure: on

brick 37 to 48 are added later.

mount info :-
[root@localhost in]# mount -t nfs 192.168.12.13:mio /mnt/nfs

Comment 3 Nithya Balachandran 2014-08-12 08:45:11 UTC

As there was no data loss in the scenario, this will not be fixed for Denali. Moving this to RHS 3.0.z

Comment 6 Susant Kumar Palai 2015-11-27 10:52:24 UTC

Cloning to 3.1. To be fixed in future release.

Note You need to log in before you can comment on or make changes to this bug.