Description of problem: If size of the files created on volume exceeds the space available on the volume all the clients are marked fools. Version-Release number of selected component (if applicable): mainline How reproducible: Steps to Reproduce: 1.create a distribute-replicate volume. start the volume (each brick has 50G space available) 2.create gluster,nfs mounts from client1 3.perform "dd if=/dev/zero of=gfsf1 bs=1M count=102400" from mount1 4.perform "dd if=/dev/zero of=nfsf1 bs=1M count=102400" from mount2 5.perform "dd if=/dev/urandom of=gfsf2 bs=1M count=102400" from mount3 6.perform "dd if=/dev/urandom of=nfsf2 bs=1M count=102400" from mount4 7.The file sizes created should exceed the space on the device. Actual results: gluster volume info Volume Name: datastore Type: Distributed-Replicate Volume ID: bc4bb820-400f-493e-bef7-ed09b87c8c91 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 192.168.2.35:/export1 Brick2: 192.168.2.36:/export1 Brick3: 192.168.2.35:/export2 Brick4: 192.168.2.36:/export2 Options Reconfigured: diagnostics.brick-log-level: DEBUG diagnostics.client-log-level: DEBUG Brick1:- --------- [03/07/12 - 20:33:10 root@APP-SERVER1 ~]# getfattr -R -m . -d -e hex /export1/* getfattr: Removing leading '/' from absolute path names # file: export1/nfsf1 trusted.afr.datastore-client-0=0x000001d10000000000000000 trusted.afr.datastore-client-1=0x000001d10000000000000000 trusted.gfid=0x4505225ade9d470290588082a5260ccb Brick2:- -------- [03/07/12 - 20:22:43 root@APP-SERVER2 glusterfs]# getfattr -m . -d -e hex /export1/* getfattr: Removing leading '/' from absolute path names # file: export1/nfsf1 trusted.afr.datastore-client-0=0x000001d10000000000000000 trusted.afr.datastore-client-1=0x000001d10000000000000000 trusted.gfid=0x4505225ade9d470290588082a5260ccb Brick3:- ---------- [03/07/12 - 20:33:07 root@APP-SERVER1 ~]# getfattr -R -m . -d -e hex /export2/* getfattr: Removing leading '/' from absolute path names # file: export2/gfsf1 trusted.afr.datastore-client-2=0x0000000b0000000000000000 trusted.afr.datastore-client-3=0x0000000a0000000000000000 trusted.gfid=0x4ddef0724f4346d9b486a4a83ac649c6 # file: export2/gfsf2 trusted.afr.datastore-client-2=0x000000120000000000000000 trusted.afr.datastore-client-3=0x000000100000000000000000 trusted.gfid=0xfd5014406f67407f9abae7d3b97d7206 # file: export2/nfsf2 trusted.afr.datastore-client-2=0x00000d3e0000000000000000 trusted.afr.datastore-client-3=0x00000d3f0000000000000000 trusted.gfid=0x2f2a9de774d947e1830c8777ee4bbadf Brick4:- ------- [03/07/12 - 20:34:00 root@APP-SERVER2 glusterfs]# getfattr -m . -d -e hex /export2/* getfattr: Removing leading '/' from absolute path names # file: export2/gfsf1 trusted.afr.datastore-client-2=0x0000000b0000000000000000 trusted.afr.datastore-client-3=0x0000000a0000000000000000 trusted.gfid=0x4ddef0724f4346d9b486a4a83ac649c6 # file: export2/gfsf2 trusted.afr.datastore-client-2=0x000000120000000000000000 trusted.afr.datastore-client-3=0x000000100000000000000000 trusted.gfid=0xfd5014406f67407f9abae7d3b97d7206 # file: export2/nfsf2 trusted.afr.datastore-client-2=0x00000d3f0000000000000000 trusted.afr.datastore-client-3=0x00000d400000000000000000 trusted.gfid=0x2f2a9de774d947e1830c8777ee4bbadf Expected results: Additional info:
After marking the clients all-fools , lookup on files has various behaviors. 1) cat gfsf2/nfsf2 : Successful 2) ls -l gfsf1 : No such file or directory [03/08/12 - 11:45:49 root@Shwetha-Laptop nfsc1]# ls -lh ls: cannot access file10: Invalid argument ls: cannot access gfsf1: Invalid argument ls: cannot access gfsf2: Invalid argument total 46G -?????????? ? ? ? ? ? file10 -?????????? ? ? ? ? ? gfsf1 -?????????? ? ? ? ? ? gfsf2 -rw-r--r--. 1 root root 41G Mar 8 2012 nfsf1 -rw-r--r--. 1 root root 4.9G Mar 7 23:33 nfsf2 [03/08/12 - 11:46:09 root@Shwetha-Laptop nfsc1]# ls -lh gfsf1 -rw-r--r--. 1 root root 41G Mar 8 2012 gfsf1 [03/08/12 - 11:46:24 root@Shwetha-Laptop nfsc1]# ls -lh ls: cannot access file10: Invalid argument ls: cannot access gfsf2: Invalid argument total 86G -?????????? ? ? ? ? ? file10 -rw-r--r--. 1 root root 41G Mar 8 2012 gfsf1 -?????????? ? ? ? ? ? gfsf2 -rw-r--r--. 1 root root 41G Mar 8 2012 nfsf1 -rw-r--r--. 1 root root 4.9G Mar 7 23:33 nfsf2
Shwetha, I'm unable to reproduce this on current master using the steps above. Writes eventually fail with ENOSPEC (or sometimes EIO) but the changelogs are all clear and directory listings look normal.
as per comment #2, no specific fixes targetting this behavior, but please re-test initial scripts, and see if its happening.
Please feel free to re-open the bug if it appears again. Pranith.
Discard comment 5 as it applies for the bug 853685 which is clone of this bug.
because of the large number of bugs filed against mainline version\ is ambiguous and about to be removed as a choice. If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.