763892 – (GLUSTER-2160) readdir function can't get all files ,when the directory has huge files

Bug 763892 (GLUSTER-2160) - readdir function can't get all files ,when the directory has huge files

Summary: readdir function can't get all files ,when the directory has huge files

Keywords:
Status:	CLOSED WONTFIX
Alias:	GLUSTER-2160
Product:	GlusterFS
Classification:	Community
Component:	distribute
Sub Component:
Version:	3.1.0
Hardware:	i386
OS:	Linux
Priority:	low
Severity:	medium
Target Milestone:	---
Assignee:	shishir gowda
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2010-11-29 01:39 UTC by rick
Modified:	2013-12-09 01:22 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2012-04-27 05:33:37 UTC
Regression:	---
Mount Type:	fuse
Documentation:	---
CRM:
Verified Versions:
Dependent Products:

Attachments	(Terms of Use)
test process (292.20 KB, application/zip) 2010-12-02 06:05 UTC, rick	no flags	Details
Screenshot. (125.08 KB, image/png) 2011-01-17 09:40 UTC, Gaurav	no flags	Details
have a log file and a screenshot (541.40 KB, application/x-compressed-tar) 2011-01-18 07:46 UTC, rick	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Description rick 2010-11-29 01:39:04 UTC

My mount directory have 10000 files,but use readdir() i just get 3114. on local disk ,readdir() is normal ,could get all files in directory.

Comment 1 Amar Tumballi 2010-11-30 23:55:20 UTC

Hi Rick,

Can you please give more info on how to reproduce the issue? The description of the bug doesn't clearly define it. you were using 'readdir()' call? or /bin/ls command? also, it doesn't tell, where are you running 'readdir()'..

Comment 2 rick 2010-12-02 06:05:14 UTC

Created attachment 393



  The attachment have my test code ,makefile,test process graph and volume file.

Comment 3 rick 2010-12-02 06:06:08 UTC

(In reply to comment #1)
> Hi Rick,
> 
> Can you please give more info on how to reproduce the issue? The description of
> the bug doesn't clearly define it. you were using 'readdir()' call? or /bin/ls
> command? also, it doesn't tell, where are you running 'readdir()'..

Hi Amar Tumballi

   Oh yes,the attachment have my test code ,makefile,test process graph and volume file.I execute "glusterfs -f t-volume-fuse.vol /glusterfs" ,the volume was mounted on /glusterfs.then, then run "./fsopt creat /glusterfs 2 4094 8192 10000" to create file.it's mean startup 2 threads to create files in /glusterfs directory ,per thread create 10000 files and the file size between 4096 and 8192 bytes. run "ls /glusterfs/thread0/group0/ | wc -l" ,the output is 10000 ,this is right.but i run "/root/fsopt/readfiles /glusterfs/thread0/group0",just get 649 files. the program readfiles  call readdir api to get the files in directory.

Comment 4 Anand Avati 2010-12-02 06:30:39 UTC

Are you using unify? Can you please give us your t-volume-fuse.vol file?

Avati

(In reply to comment #3)
> (In reply to comment #1)
> > Hi Rick,
> > 
> > Can you please give more info on how to reproduce the issue? The description of
> > the bug doesn't clearly define it. you were using 'readdir()' call? or /bin/ls
> > command? also, it doesn't tell, where are you running 'readdir()'..
> 
> Hi Amar Tumballi
> 
>    Oh yes,the attachment have my test code ,makefile,test process graph and
> volume file.I execute "glusterfs -f t-volume-fuse.vol /glusterfs" ,the volume
> was mounted on /glusterfs.then, then run "./fsopt creat /glusterfs 2 4094 8192
> 10000" to create file.it's mean startup 2 threads to create files in /glusterfs
> directory ,per thread create 10000 files and the file size between 4096 and
> 8192 bytes. run "ls /glusterfs/thread0/group0/ | wc -l" ,the output is 10000
> ,this is right.but i run "/root/fsopt/readfiles /glusterfs/thread0/group0",just
> get 649 files. the program readfiles  call readdir api to get the files in
> directory.

Comment 5 rick 2010-12-02 06:37:33 UTC

vol file
volume t-volume-client-0
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdb
    option transport-type tcp
end-volume

volume t-volume-client-1
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdc
    option transport-type tcp
end-volume

volume t-volume-client-2
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdd
    option transport-type tcp
end-volume

volume t-volume-client-3
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sde
    option transport-type tcp
end-volume

volume t-volume-client-4
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdf
    option transport-type tcp
end-volume

volume t-volume-client-5
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdg
    option transport-type tcp
end-volume

volume t-volume-client-6
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdh
    option transport-type tcp
end-volume

volume t-volume-client-7
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdi
    option transport-type tcp
end-volume

volume t-volume-client-8
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdj
    option transport-type tcp
end-volume

volume t-volume-client-9
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdk
    option transport-type tcp
end-volume

volume t-volume-client-10
    type protocol/client
    option remote-host 192.168.13.146
    option remote-subvolume /mnt/sdl
    option transport-type tcp
end-volume

volume t-volume-client-11
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdb
    option transport-type tcp
end-volume

volume t-volume-client-12
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdc
    option transport-type tcp
end-volume

volume t-volume-client-13
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdd
    option transport-type tcp
end-volume

volume t-volume-client-14
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sde
    option transport-type tcp
end-volume

volume t-volume-client-15
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdf
    option transport-type tcp
end-volume

volume t-volume-client-16
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdg
    option transport-type tcp
end-volume

volume t-volume-client-17
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdh
    option transport-type tcp
end-volume

volume t-volume-client-18
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdi
    option transport-type tcp
end-volume

volume t-volume-client-19
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdj
    option transport-type tcp
end-volume

volume t-volume-client-20
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdk
    option transport-type tcp
end-volume

volume t-volume-client-21
    type protocol/client
    option remote-host 192.168.13.147
    option remote-subvolume /mnt/sdl
    option transport-type tcp
end-volume

volume t-volume-client-22
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdb
    option transport-type tcp
end-volume

volume t-volume-client-23
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdc
    option transport-type tcp
end-volume

volume t-volume-client-24
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdd
    option transport-type tcp
end-volume

volume t-volume-client-25
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sde
    option transport-type tcp
end-volume

volume t-volume-client-26
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdf
    option transport-type tcp
end-volume

volume t-volume-client-27
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdg
    option transport-type tcp
end-volume

volume t-volume-client-28
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdh
    option transport-type tcp
end-volume

volume t-volume-client-29
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdi
    option transport-type tcp
end-volume

volume t-volume-client-30
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdj
    option transport-type tcp
end-volume

volume t-volume-client-31
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdk
    option transport-type tcp
end-volume

volume t-volume-client-32
    type protocol/client
    option remote-host 192.168.13.145
    option remote-subvolume /mnt/sdl
    option transport-type tcp
end-volume

volume t-volume-client-33
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdb
    option transport-type tcp
end-volume

volume t-volume-client-34
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdc
    option transport-type tcp
end-volume

volume t-volume-client-35
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdd
    option transport-type tcp
end-volume

volume t-volume-client-36
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sde
    option transport-type tcp
end-volume

volume t-volume-client-37
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdf
    option transport-type tcp
end-volume

volume t-volume-client-38
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdg
    option transport-type tcp
end-volume

volume t-volume-client-39
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdh
    option transport-type tcp
end-volume

volume t-volume-client-40
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdi
    option transport-type tcp
end-volume

volume t-volume-client-41
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdj
    option transport-type tcp
end-volume

volume t-volume-client-42
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdk
    option transport-type tcp
end-volume

volume t-volume-client-43
    type protocol/client
    option remote-host 192.168.13.148
    option remote-subvolume /mnt/sdl
    option transport-type tcp
end-volume

volume t-volume-dht
    type cluster/distribute
    subvolumes t-volume-client-0 t-volume-client-1 t-volume-client-2 t-volume-client-3 t-volume-client-4 t-volume-client-5 t-volume-client-6 t-volume-client-7 t-volume-client-8 t-volume-client-9 t-volume-client-10 t-volume-client-11 t-volume-client-12 t-volume-client-13 t-volume-client-14 t-volume-client-15 t-volume-client-16 t-volume-client-17 t-volume-client-18 t-volume-client-19 t-volume-client-20 t-volume-client-21 t-volume-client-22 t-volume-client-23 t-volume-client-24 t-volume-client-25 t-volume-client-26 t-volume-client-27 t-volume-client-28 t-volume-client-29 t-volume-client-30 t-volume-client-31 t-volume-client-32 t-volume-client-33 t-volume-client-34 t-volume-client-35 t-volume-client-36 t-volume-client-37 t-volume-client-38 t-volume-client-39 t-volume-client-40 t-volume-client-41 t-volume-client-42 t-volume-client-43
end-volume

volume t-volume-write-behind
    type performance/write-behind
    subvolumes t-volume-dht
end-volume

volume t-volume-read-ahead
    type performance/read-ahead
    subvolumes t-volume-write-behind
end-volume

volume t-volume-io-cache
    type performance/io-cache
    subvolumes t-volume-read-ahead
end-volume

volume t-volume-quick-read
    type performance/quick-read
    subvolumes t-volume-io-cache
end-volume

volume t-volume
    type debug/io-stats
    subvolumes t-volume-quick-read
end-volume

Comment 6 Gaurav 2011-01-17 09:40:51 UTC

Created attachment 417

Comment 7 Gaurav 2011-01-17 09:45:48 UTC

Hi,
   I tried to reproduce this issue on my setup using the same fsopt tool.
Created 10000 file on distributed setup, both ls and readdir tool showing the correct results. Attached the screenshot. See volfile below.

volume vol_dht-client-0
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol1
    option transport-type tcp
end-volume

volume vol_dht-client-1
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol2
    option transport-type tcp
end-volume

volume vol_dht-client-2
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol3
    option transport-type tcp
end-volume

volume vol_dht-client-3
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol4
    option transport-type tcp
end-volume

volume vol_dht-client-4
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol5
    option transport-type tcp
end-volume

volume vol_dht-client-5
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol6
    option transport-type tcp
end-volume

volume vol_dht-client-6
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol7
    option transport-type tcp
end-volume

volume vol_dht-client-7
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol8
    option transport-type tcp
end-volume

volume vol_dht-client-8
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol9
    option transport-type tcp
end-volume

volume vol_dht-client-9
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol10
    option transport-type tcp
end-volume

volume vol_dht-client-10
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol11
    option transport-type tcp
end-volume

volume vol_dht-client-11
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol12
    option transport-type tcp
end-volume

volume vol_dht-client-12
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol13
    option transport-type tcp
end-volume

volume vol_dht-client-13
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol14
    option transport-type tcp
end-volume

volume vol_dht-client-14
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol15
    option transport-type tcp
end-volume

volume vol_dht-client-15
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol16
    option transport-type tcp
end-volume

volume vol_dht-client-16
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol17
    option transport-type tcp
end-volume

volume vol_dht-client-17
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol18
    option transport-type tcp
end-volume

volume vol_dht-client-18
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol19
    option transport-type tcp
end-volume

volume vol_dht-client-19
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol20
    option transport-type tcp
end-volume

volume vol_dht-client-20
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol21
    option transport-type tcp
end-volume

volume vol_dht-client-21
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol22
    option transport-type tcp
end-volume

volume vol_dht-client-22
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol23
    option transport-type tcp
end-volume

volume vol_dht-client-23
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol24
    option transport-type tcp
end-volume

volume vol_dht-client-24
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol25
    option transport-type tcp
end-volume

volume vol_dht-client-25
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol26
    option transport-type tcp
end-volume

volume vol_dht-client-26
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol27
    option transport-type tcp
end-volume

volume vol_dht-client-27
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol28
    option transport-type tcp
end-volume

volume vol_dht-client-28
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol29
    option transport-type tcp
end-volume

volume vol_dht-client-29
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol30
    option transport-type tcp
end-volume

volume vol_dht-client-30
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol31
    option transport-type tcp
end-volume

volume vol_dht-client-31
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol32
    option transport-type tcp
end-volume

volume vol_dht-client-32
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol33
    option transport-type tcp
end-volume

volume vol_dht-client-33
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol34
    option transport-type tcp
end-volume

volume vol_dht-client-34
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol35
    option transport-type tcp
end-volume

volume vol_dht-client-35
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol36
    option transport-type tcp
end-volume

volume vol_dht-client-36
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol37
    option transport-type tcp
end-volume

volume vol_dht-client-37
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol38
    option transport-type tcp
end-volume

volume vol_dht-client-38
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol39
    option transport-type tcp
end-volume

volume vol_dht-client-39
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol40
    option transport-type tcp
end-volume

volume vol_dht-client-40
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol41
    option transport-type tcp
end-volume

volume vol_dht-client-41
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol42
    option transport-type tcp
end-volume

volume vol_dht-client-42
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol43
    option transport-type tcp
end-volume

volume vol_dht-client-43
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol44
    option transport-type tcp
end-volume

volume vol_dht-client-44
    type protocol/client
    option remote-host gaurav-Vostro-3400
    option remote-subvolume /home/gaurav/vol45
    option transport-type tcp
end-volume

volume vol_dht-dht
    type cluster/distribute
    subvolumes vol_dht-client-0 vol_dht-client-1 vol_dht-client-2 vol_dht-client-3 vol_dht-client-4 vol_dht-client-5 vol_dht-client-6 vol_dht-client-7 vol_dht-client-8 vol_dht-client-9 vol_dht-client-10 vol_dht-client-11 vol_dht-client-12 vol_dht-client-13 vol_dht-client-14 vol_dht-client-15 vol_dht-client-16 vol_dht-client-17 vol_dht-client-18 vol_dht-client-19 vol_dht-client-20 vol_dht-client-21 vol_dht-client-22 vol_dht-client-23 vol_dht-client-24 vol_dht-client-25 vol_dht-client-26 vol_dht-client-27 vol_dht-client-28 vol_dht-client-29 vol_dht-client-30 vol_dht-client-31 vol_dht-client-32 vol_dht-client-33 vol_dht-client-34 vol_dht-client-35 vol_dht-client-36 vol_dht-client-37 vol_dht-client-38 vol_dht-client-39 vol_dht-client-40 vol_dht-client-41 vol_dht-client-42 vol_dht-client-43 vol_dht-client-44
end-volume

volume vol_dht-write-behind
    type performance/write-behind
    subvolumes vol_dht-dht
end-volume

volume vol_dht-read-ahead
    type performance/read-ahead
    subvolumes vol_dht-write-behind
end-volume

volume vol_dht-io-cache
    type performance/io-cache
    subvolumes vol_dht-read-ahead
end-volume

volume vol_dht-quick-read
    type performance/quick-read
    subvolumes vol_dht-io-cache
end-volume

volume vol_dht-stat-prefetch
    type performance/stat-prefetch
    subvolumes vol_dht-quick-read
end-volume

volume vol_dht
    type debug/io-stats
    subvolumes vol_dht-stat-prefetch
end-volume

Please check with latest build, we can close it if not reproducible.

Comment 8 Gaurav 2011-01-18 00:30:37 UTC

Also give log files if it is reproduced.

Comment 9 rick 2011-01-18 07:24:49 UTC

(In reply to comment #8)
> Also give log files if it is reproduced.

hi,
  I first find this issue is on 3.1.0 version,today i reproduce this issue on build of 3.1.1.i add subvolume of debug/trace in order to get detail log.

Comment 10 rick 2011-01-18 07:46:11 UTC

Created attachment 421

Comment 11 Gaurav 2011-01-19 00:26:18 UTC

ls is also calling readdir, but its succeding. We can clearly differentiate from client logs that, both ls and readfiles tool calling readdir, but readfiles stop after offset=94489280481.

I am doubting its a readdir problem, otherwise ls should also fail.

Comment 12 rick 2011-01-19 22:43:46 UTC

it's very strange,you can't reproduce this issue on your setup,but here ,it's very easy to reproduce. result is correct on local file system in some test way. i also find that "ls" run time is shorter than "readfiles".

Comment 13 Amar Tumballi 2012-02-27 10:43:19 UTC

priority will be re-addressed after 3.3.0 GA

Comment 14 shishir gowda 2012-04-27 05:33:37 UTC

We are not able to reproduce this issue in-house.
please check with 3.3.0beta3.
If reproduced, please re-open the bug.

Note You need to log in before you can comment on or make changes to this bug.