Bug 990909

Summary: Duplicate files seen on mount point while trying to create files which are greater than the brick size
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: senaik
Component: glusterfsAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED DEFERRED QA Contact: storage-qa-internal <storage-qa-internal>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.1CC: asmarre, rhs-bugs, spalai, surs, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1286092 (view as bug list) Environment:
Last Closed: 2015-11-27 10:45:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1286092    

Description senaik 2013-08-01 07:54:06 UTC
Description of problem:
========================== 
Duplicate files seen on mount point while trying to create files which are greater than the brick size 

Version-Release number of selected component (if applicable):
============================================================ 
3.4.0.14rhs-1.el6rhs.x86_64

How reproducible:


Steps to Reproduce:
===================== 
1.Create a distribute volume with 2 bricks (each brick is 1GB in size)

2.Fuse mount the volume and create files which are greater than the brick size 
for i in {1..500} ; do dd if=/dev/urandom of=f"$i" bs=10M count=1; done

Files are created and after some time , we get 'No Space left on device'

[root@RHEL6 sample]# du -sh 
1.9G

3. ll on the mount point 

-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f163
---------T. 1 root root        0 Aug  1 12:04 f164
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f164
---------T. 1 root root        0 Aug  1 12:04 f165
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f165
---------T. 1 root root        0 Aug  1 12:04 f166
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f166
---------T. 1 root root        0 Aug  1 12:04 f167
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f167
---------T. 1 root root        0 Aug  1 12:04 f168
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f168
---------T. 1 root root        0 Aug  1 12:04 f169
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f169
-rw-r--r--. 1 root root 10485760 Aug  1 12:01 f17
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f170
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f171
---------T. 1 root root        0 Aug  1 12:04 f172
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f172
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f173
-rw-r--r--. 1 root root 10485760 Aug  1 12:05 f174
-rw-r--r--. 1 root root 10485760 Aug  1 12:05 f175
---------T. 1 root root        0 Aug  1 09:41 f175
-rw-r--r--. 1 root root 10485760 Aug  1 12:05 f176
---------T. 1 root root        0 Aug  1 09:41 f176
---------T. 1 root root        0 Aug  1 12:05 f177
-rw-r--r--. 1 root root 10485760 Aug  1 09:41 f177


Actual results:
=================== 
Duplicate files seen on the mount point 

Expected results:
===================== 
There should be no duplicate files seen on the mount point 

Additional info:
================= 
Volume info : 
============ 

[root@jay tmp]# gluster v i sample
 
Volume Name: sample
Type: Distribute
Volume ID: 1895ec09-a451-4182-acd6-b258519c8c8e
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.70.34.86:/BRICK1/b1
Brick2: 10.70.34.87:/BRICK1/b2

sosreports : 
============ 
http://rhsqe-repo.lab.eng.blr.redhat.com/bugs_necessary_info/990897/

Comment 2 senaik 2013-08-01 11:49:25 UTC
Some more info : 
==============
[root@jay tmp]# getfattr -d -m . -e hex /BRICK1/b1/f164
getfattr: Removing leading '/' from absolute path names
# file: BRICK1/b1/f164
trusted.gfid=0x68e279ee24494eff9648a2e2b6d6d70a



[root@jay tmp]# stat /BRICK1/b1/f164
  File: `/BRICK1/b1/f164'
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: 700h/1792d	Inode: 1572993     Links: 2
Access: (1000/---------T)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-08-01 12:26:24.902225460 +0530
Modify: 2013-08-01 12:04:48.460217599 +0530
Change: 2013-08-01 12:04:48.460217599 +0530



----------------------------Part of the Log ------------------------ 
[2013-08-01 06:34:41.445602] E [posix.c:1766:posix_create] 0-sample-posix: setting xattrs on /BRICK1/b1/f159 failed (Operation not supported)
[2013-08-01 06:34:42.841664] E [posix.c:1766:posix_create] 0-sample-posix: setting xattrs on /BRICK1/b1/f160 failed (Operation not supported)
[2013-08-01 06:34:48.461813] E [posix.c:864:posix_mknod] 0-sample-posix: setting xattrs on /BRICK1/b1/f164 failed (Operation not supported)
[2013-08-01 06:34:49.863653] E [posix.c:864:posix_mknod] 0-sample-posix: setting xattrs on /BRICK1/b1/f165 failed (Operation not supported)
[2013-08-01 06:34:51.284356] E [posix.c:864:posix_mknod] 0-sample-posix: setting xattrs on /BRICK1/b1/f166 failed (Operation not supported)
[2013-08-01 06:34:52.691675] E [posix.c:864:posix_mknod] 0-sample-posix: setting xattrs on /BRICK1/b1/f167 failed (Operation not supported)
[2013-08-01 06:34:54.085016] E [posix.c:864:posix_mknod] 0-sample-posix: setting xattrs on /BRICK1/b1/f168 failed (Operation not supported)
[2013-08-01 06:34:55.485422] E [posix.c:864:posix_mknod] 0-sample-posix: setting xattrs on /BRICK1/b1/f169 failed (Operation not supported)
[2013-08-01 06:34:59.720479] E [posix.c:864:posix_mknod] 0-sample-posix: setting xattrs on /BRICK1/b1/f172 failed (Operation not supported)

.
.
.
[2013-08-01 06:36:05.666891] E [posix.c:1732:posix_create] 0-sample-posix: open on /BRICK1/b1/f247 failed: No space left on device
[2013-08-01 06:36:05.666930] I [server-rpc-fops.c:1618:server_create_cbk] 0-sample-server: 9404: CREATE /f247 (00000000-0000-0000-0000-000000000001/f2
47) ==> (No space left on device)
[2013-08-01 06:36:05.672870] E [posix.c:1732:posix_create] 0-sample-posix: open on /BRICK1/b1/f248 failed: No space left on device
[2013-08-01 06:36:05.672907] I [server-rpc-fops.c:1618:server_create_cbk] 0-sample-server: 9409: CREATE /f248 (00000000-0000-0000-0000-000000000001/f2
48) ==> (No space left on device)
[2013-08-01 06:36:05.688410] E [posix.c:1732:posix_create] 0-sample-posix: open on /BRICK1/b1/f250 failed: No space left on device
[2013-08-01 06:36:05.688446] I [server-rpc-fops.c:1618:server_create_cbk] 0-sample-server: 9416: CREATE /f250 (00000000-0000-0000-0000-000000000001/f2
50) ==> (No space left on device)