Bug 1196026

Patch:http://review.gluster.org/#/c/13907/ fixes the problem

*** Bug 1336304 has been marked as a duplicate of this bug. ***

Good day.

We are still seeing this same issue on 3.8.1 on multiple volumes. Is this a confirmed fix? If so, what versions of gluster has it been added to?

When the quota value changes, there are no errors in the logs or any other information that indicates there is an issue.

Environment:
2 primary gluster nodes on RHEL7.2
2 secondary gluster nodes on RHEL7.2 configured for geo-replication


One volume configuration:
Volume Name: appdata
Type: Replicate
Volume ID: 3bffc422-1bf3-4593-bf2e-399e0b3e2a7f
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 192.168.66.102:/data/gluster/appdata/brick1
Brick2: 192.168.66.101:/data/gluster/appdata/brick1
Options Reconfigured:
changelog.changelog: on
geo-replication.ignore-pid-check: on
geo-replication.indexing: on
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
cluster.enable-shared-storage: enable
# gluster volume status appdata
Status of volume: appdata
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 192.168.66.102:/data/gluster/
appdata/brick1                              49192     0          Y       8502 
Brick 192.168.66.101:/data/gluster/
appdata/brick1                              49194     0          Y       6600 
Self-heal Daemon on localhost               N/A       N/A        Y       22714
Quota Daemon on localhost                   N/A       N/A        Y       18396
Self-heal Daemon on 10.0.0.102              N/A       N/A        Y       7343 
Quota Daemon on 10.0.0.102                  N/A       N/A        Y       26234


Quota status post issue:
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                          6.0GB     80%(4.8GB) 16384.0PB   6.0GB              No                   No


Could the 16384.0PB be the result of an evaluation expecting an unsigned integer but receiving a -1?

We have just updated to 3.8.2 but have yet to work on replicating the issue.

Thank you in advance for your assistance.

Regards,
Dan

Hi Dan,

Good day too.

Was there an 'rm -rf *' performed on mount point? Are you testing it on a fresh install of 3.8.1? Could you also let us know the steps to reproduce.

--
Regards,
Manikandan Selvaganesh.

We have a few gluster volumes with this issue. One of them has a series of 'find' commands that executes an 'rm' on the output. the other gluster volume has data manipulated by an application and we are waiting for the code to verify.

The 3.8.1 install had been in place for a couple weeks and the affected volumes had been in use for a week or so before the issue occurred. It is sporadic when the issue appears. We have disabled and re-enabled quotas between occurrences to get the volumes functional again.

Currently, a non production volume has its quota enabled since we patched to 3.8.2. The issue has yet to introduce itself again.

Thank you for looking into this.

Regards,
Dan

We just had the non production volume hit the bug again some time in the last 24 hours. Quota said that we were using the 16.3 XB of data whilst df showed all 100GB and du show 18GB. On the georeplicated volume, df showed 93GB used and du showed 18GB. The quota log on the primary volume stopped logging at log rotation on 8/21 so we have no information on the quota behavior. The quota has been enabled for over a week without issue until now.

We did verify that the other application is executing an 'rm' within sftp. So far all applications are executing deletes/removes.

I saw that 3.8.3 was just released. I have not read the release notes yet. Hopefully there is a fix for this. 

Please let me know if you have identified anything or if you need more data from me.

Thank you.

Regards,
Dan

Hi,

Can you paste the output of 'gluster v info', 'gluster v quota <VOLNAME> list' and the output of df -h?

It would also be great if you could attach the logs on the system where you are hitting the issue.

--
Regards,
Manikandan Selvaganesh.

We had the issue again. Here is the requested information.

# gluster volume info exports_nonprod
Volume Name: exports_nonprod
Type: Replicate
Volume ID: 6a5ac071-2f33-47b7-9630-ec644e723906
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 192.168.66.102:/data/gluster/exports_nonprod/brick1
Brick2: 192.168.66.101:/data/gluster/exports_nonprod/brick1
Options Reconfigured:
features.quota-deem-statfs: on
changelog.changelog: on
geo-replication.ignore-pid-check: on
geo-replication.indexing: on
features.inode-quota: on
features.quota: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
cluster.enable-shared-storage: enable

# gluster volume quota exports_nonprod list 
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                         50.0GB     80%(40.0GB) 16384.0PB  73.8GB              No                   No

# df -h /run/gluster/exports_nonprod
Filesystem                     Size  Used Avail Use% Mounted on
localhost:exports_nonprod   50G   50G     0 100% /run/gluster/exports_nonprod

# du -sh /run/gluster/exports_nonprod
18G	/run/gluster/exports_nonprod

# du -sh /data/gluster/exports_nonprod
19G	/data/gluster/exports_nonprod

# du -csh /data/gluster/exports_nonprod/brick1/.[!.]* /data/gluster/exports_nonprod/brick1/* | sort -h 
8.0K	/data/gluster/exports_nonprod/brick1/P
16K	/data/gluster/exports_nonprod/brick1/C
16K	/data/gluster/exports_nonprod/brick1/.trashcan
48K	/data/gluster/exports_nonprod/brick1/u
56K	/data/gluster/exports_nonprod/brick1/d
56K	/data/gluster/exports_nonprod/brick1/t
19G	/data/gluster/exports_nonprod/brick1/.glusterfs
19G	total


The other item that I have found interesting is each time we have encountered this issue, the current log has only one entry. I find that interesting. 

# cat /var/log/glusterfs/quota-mount-exports_nonprod.log
[2016-08-28 08:42:12.082238] I [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing

We are looking at gluster 3.8.3 currently as well.

Thank you again. I hope you can assist!

Regards,
Dan

Looking at the main gluster servers today, 'df' showed use of the 50GB quota at 8.5GB whilst 'du' showed 21GB. The 21GB lines up with what our geo-replication servers report. Strange that it under-reports.

Regards,
Dan

Any further news?

Regards,
Dan

Any updates?

Thank you,
Dan

Hi,

The above issue is because of negative accounting. It can happen because of a few combination of operations. We have fixed the operations that we are aware of causing this.  

Being an accounting issue, you can disable and enable quota to get the accounting done right. Or use the quota fsck script to fix the issues. 

fsck script: https://review.gluster.org/#/c/glusterfs/19179/

Closing this bug as quota is not being actively developed. If seen again the above work around can be used to fix it.

-Hari.