Bug 998893 - quota: zero byte file created
quota: zero byte file created
Status: CLOSED NEXTRELEASE
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: quota (Show other bugs)
2.1
x86_64 Linux
medium Severity high
: ---
: ---
Assigned To: Raghavendra G
Saurabh
: ZStream
: 989753 (view as bug list)
Depends On:
Blocks: 1012216 1020127
  Show dependency treegraph
 
Reported: 2013-08-20 06:02 EDT by Saurabh
Modified: 2016-09-19 18:06 EDT (History)
17 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.36rhs-1
Doc Type: Known Issue
Doc Text:
Zero byte sized files are created when a write operation exceeds the available quota space. Since Quota does not account for the disk blocks consumed by a directory(as per Bug 1016419), the write operation creates the directory entry but the subsequent write operation fails because of unavailable disk space.
Story Points: ---
Clone Of:
: 1012216 (view as bug list)
Environment:
Last Closed: 2015-01-12 04:19:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Saurabh 2013-08-20 06:02:15 EDT
Description of problem:
while quota is about to be reached or reached, then things are no getting handled properly.
as I am I/O erros, zero byte file, truncated file even after a "Disk quota exceeded" msg is recieved.

Version-Release number of selected component (if applicable):
glusterfs-server-3.4.0.20rhsquota1-1.el6.x86_64
glusterfs-fuse-3.4.0.20rhsquota1-1.el6.x86_64
glusterfs-3.4.0.20rhsquota1-1.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. create a volume of 6x2 type, start it
2. enable quota on the volume
3. mount the volume over nfs.
4. create a directory inside the volume
5. set limit on the directory, lets say 1GB
6. start creating data inside the dir, in a for loop using dd command
   each file of 1MB size.

Actual results:

 when quota is about to be reached,

1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.354007 s, 3.0 MB/s
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.351123 s, 3.0 MB/s
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.29281 s, 3.6 MB/s
dd: closing output file `1020.1376981546': Input/output error
dd: closing output file `1021.1376981547': Input/output error
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.285709 s, 3.7 MB/s
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.305432 s, 3.4 MB/s
dd: opening `1024.1376981548': Disk quota exceeded
dd: closing output file `1025.1376981548': Disk quota exceeded
dd: opening `1026.1376981548': Disk quota exceeded
dd: opening `1027.1376981548': Disk quota exceeded
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.357487 s, 2.9 MB/s
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.36181 s, 2.9 MB/s
dd: closing output file `1030.1376981549': Input/output error


further,
[root@rhsauto036 dir3]# ls -l 1022*
-rw-r--r--. 1 root root 1048576 Aug 20 02:52 1022.1376981547

[root@rhsauto036 dir3]# ls -l 1023*
-rw-r--r--. 1 root root 1048576 Aug 20 02:52 1023.1376981547

[root@rhsauto036 dir3]# ls -l 1024*
ls: cannot access 1024*: No such file or directory

[root@rhsauto036 dir3]# ls -l 1025*
-rw-r--r--. 1 root root 0 Aug 20 02:52 1025.1376981548

[root@rhsauto036 dir3]# ls -l 1026*
ls: cannot access 1026*: No such file or directory

[root@rhsauto036 dir3]# ls -l 1027*
ls: cannot access 1027*: No such file or directory

[root@rhsauto036 dir3]# ls -l 1028*
-rw-r--r--. 1 root root 1048576 Aug 20 02:52 1028.1376981548

[root@rhsauto036 dir3]# ls -l 1029*
-rw-r--r--. 1 root root 1048576 Aug 20 02:52 1029.1376981548

[root@rhsauto036 dir3]# ls -l 1030*
-rw-r--r--. 1 root root 458752 Aug 20 02:52 1030.1376981549

[root@rhsauto036 dir3]# 
[root@rhsauto036 dir3]# ls -l 1021*
-rw-r--r--. 1 root root 458752 Aug 20 02:52 1021.1376981547

[root@rhsauto036 dir3]# 
[root@rhsauto036 dir3]# ls -l 1020*
-rw-r--r--. 1 root root 0 Aug 20 02:52 1020.1376981546

[root@rhsauto036 dir3]# ls -l 1019*
-rw-r--r--. 1 root root 1048576 Aug 20 02:52 1019.1376981546


Expected results:
if disk quota exceeded means exceeded, it should result in zero byte files and truncated files.

Additional info:
Comment 1 Saurabh 2013-08-20 06:03:13 EDT
correcting the typo in Expected results,

if disk quota exceeded means exceeded, it should not result in zero byte files and truncated files.
Comment 6 Raghavendra G 2013-08-23 00:29:06 EDT
Saurabh,

I am not able to get sos reports. I am getting permission denied errors

You don't have permission to access /sosreports/998914/sosreport-rhsauto032-20130821013955-eca0.tar.xz on this server.

Can you please tell what is the volume configuration?

regards,
Raghavendra.
Comment 7 Raghavendra G 2013-08-23 00:31:16 EDT
Please ignore my previous comments. Those were for a different bug (bug #998914).
Comment 10 Raghavendra G 2013-09-13 02:07:31 EDT
No longer seen in v3.4.0.30rhs. Please confirm.
Comment 11 Raghavendra G 2013-09-25 23:13:30 EDT
When the size of the directory is equal to limit set on it, any further writes will fail. However for operations that create new dentries - create, symlink, mknod - we consider delta (which will be updated to the directory size) to be zero. This allows the fop to pass quota checks. In the cbk, marker updates the size of directory to account for iabuf.ia_blocks. Had ia_blocks been non-zero, subsequent create operations wouldn't pass quota limit checks in enforcer (since size of directory + delta > limit). However, because of a regression introduced in the workaround to xfs pre-allocation (to not affect quota wildly), iabuf.ia_blocks in create cbk is set to zero. This results in any number of create operations to pass quota limit checks.
Comment 12 Sayan Saha 2013-09-26 13:31:26 EDT
Is there a resolution or a patch available for this one upstream yet?
Comment 13 Pranith Kumar K 2013-09-30 04:35:47 EDT
*** Bug 989753 has been marked as a duplicate of this bug. ***
Comment 14 Raghavendra G 2013-10-04 06:05:46 EDT
Downstream patches submitted at:

rhs-2.1: https://code.engineering.redhat.com/gerrit/#/c/13654/
rhs-2.1-u1: https://code.engineering.redhat.com/gerrit/#/c/13667/

upstream patch:
http://review.gluster.org/6035

regards,
Raghavendra
Comment 15 Raghavendra G 2013-10-23 06:45:09 EDT
bug 1016419 tracks accounting space consumed for storing directory entries.
Comment 16 Saurabh 2013-10-24 05:37:20 EDT
Used the similar scenario as mentioned in the description section and found a file of zero size this time,

-rw-rw-r--. 1 qa1 qa1 1048576 Oct 24  2013 file1050
-rw-rw-r--. 1 qa1 qa1       0 Oct 24  2013 file1053
-rw-rw-r--. 1 qa1 qa1 1048576 Oct 24  2013 file106

list info after i/o finished,
[root@quota1 ~]# gluster volume quota dist-rep3 list /qa1/dir1-data
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/qa1/dir1-data                             1.0GB       80%       1.0GB  0Bytes


other info,
[qa1@rhsauto005 dir1-data]$ du -sh .
1.1G	.
[qa1@rhsauto005 dir1-data]$ pwd
/mnt/nfs-test/qa1/dir1-data
[qa1@rhsauto005 dir1-data]$ mount | grep dist-rep3
10.70.42.186:/dist-rep3 on /mnt/nfs-test type nfs (rw,addr=10.70.42.186)


Found on glusterfs.3.4.0.36rhs
Comment 18 Pavithra 2013-10-29 01:00:54 EDT
Hi Du,

In yesterday's call it was noted that this bug would not be fixed for Big Bend U1. Could you please review the doc text I have entered. Once you approve. I'll add it as a known issue in the Release Notes.
Comment 19 Vivek Agarwal 2013-11-14 06:26:35 EST
Moving the known issues to Doc team, to be documented in release notes for U1
Comment 20 Vivek Agarwal 2013-11-14 06:28:41 EST
Moving the known issues to Doc team, to be documented in release notes for U1
Comment 21 Vivek Agarwal 2013-11-14 06:29:15 EST
Moving the known issues to Doc team, to be documented in release notes for U1
Comment 24 Vijaikumar Mallikarjuna 2015-01-12 04:19:11 EST
This issue is fixed in RHS-3.0. In 2.1 this will as known issue.
Closing the bug as future fix

Note You need to log in before you can comment on or make changes to this bug.