And is it because GFS2 kernel driver slips the limits sometimes or because setquota should call Q_SYNC. And if the second one, how can you prevent from race (set, exceed limit, sync)? I was always told that this is how GFS2 behaves because it's distributed file system.

I gave a try with 2.6.32-431.el6.x86_64 and quota-3.17-21.el6_5.x86_64 and I get "Disk quota exceeded" error from dd(1) tool while creating the file. (Although kernel breaks at 24 KB).

This or that cannot be bug in user space. Assigning to kernel component.

This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.

Created attachment 882009 [details]
Proposed fix

This patch causes the quota to be refreshed to the latest values so quotas are enforced correctly.

Posted patch above to rhkernel-list for inclusion in RHEL6

Patch(es) available on kernel-2.6.32-465.el6

Verified in kernel-2.6.32-500.el6:

[root@dash-01 ~]# uname -r
2.6.32-500.el6.x86_64
[root@dash-01 ~]# dd if=/dev/zero of=/dev/mapper/mpathap1 bs=4G count=27
0+27 records in
0+27 records out
57981947904 bytes (58 GB) copied, 3816.36 s, 15.2 MB/s
[root@dash-01 ~]# mkfs.gfs2 -j1 -p lock_nolock /dev/mapper/mpathap1 -O
Device:                    /dev/mapper/mpathap1
Blocksize:                 4096
Device Size                100.00 GB (26214055 blocks)
Filesystem Size:           100.00 GB (26214053 blocks)
Journals:                  1
Resource Groups:           400
Locking Protocol:          "lock_nolock"
Lock Table:                ""
UUID:                      b1b7e47d-e211-1883-7b9c-a0852838eb3f

[root@dash-01 ~]# mount -t gfs2 -o rw,relatime,nodiratime,localflocks,acl,quota=on /dev/mapper/mpathap1 /mnt/gfs2/
[root@dash-01 ~]# setquota -u test 10 20 0 0 /mnt/gfs2/
[root@dash-01 ~]# chmod 777 /mnt/gfs2
[root@dash-01 ~]# sync
[root@dash-01 ~]# su test
[test@dash-01 root]$ /tmp/creator /mnt/gfs2 123 20
write(/mnt/gfs2/lrbbmqbhcdarzowkky) failed: Disk quota exceeded
create_and_write(/mnt/gfs2/lrbbmqbhcdarzowkky) failed: Disk quota exceeded
[test@dash-01 root]$ exit
exit
[root@dash-01 ~]# sync
[root@dash-01 ~]# repquota -vu /mnt/gfs2
*** Report for user quotas on device /dev/dm-5
Block grace time: 00:00; Inode grace time: 00:00
                        Block limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --       4       0       0              0     0     0       
test      +-      24       8      20              0     0     0       

*** Status for user quotas on device /dev/dm-5
Accounting: ON; Enforcement: ON
Inode: #33132 (24 blocks, 1 extents)
[root@dash-01 ~]# rm -rf /mnt/gfs2/*
[root@dash-01 ~]# umount /mnt/gfs2/

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-1392.html