Hide Forgot
Created attachment 548
Created attachment 549
I'm new to gluster, and have been evaluating it for use in a Beowulf cluster environment. I have done extensive performance testing with distributed iozone, and saw wonderful performance. My main issue is that once I enable quotas (doesn't matter if there is a quota set), I have been triggering what looks to be an issue with locking inodes. The error itself indicates it was successful at attaining the lock, but the log claims it as a failure, and as such it cycles trying to attain the lock over again. This may be related to line like "[2011-07-07 10:00:48.989978] I [server3_1-fops.c:1240:server_writev_cbk] 0-testing-server: 4661391: WRITEV 0 (18350082) ==> -1 (Cannot allocate memory)" that occur just before the spewing of the "lock setting failed (Success)" lines, however I have no indication as to why gluster would be unable to allocate memory. Attached is the log file from the server, and memory graphs of both of my fileservers as the failure started.
After extensive testing, I think this is the actual issue: If you delete a file before updates to the quota ACL have finished writing, glusterfs seems to loop trying to get an inode lock on the file to update the ACL, but as the file doesn't exist the lock fails. This may not be exactly what happens internally, but it may give you more insight into the issue. Reproduction requirements: 1 glusterfs server with quota on (mine was set to distributed mode). 2+ client machines iozone configured in distributed mode: iozone command: RSH=`which ssh` /usr/sbin/iozone -+m host_file -t 2 -r 8192 -s 128g contents of host_file: client01 /mnt/point /usr/sbin/iozone client02 /mnt/point /usr/sbin/iozone Once the test has been going for a while, but preferably on a writing test, <ctrl-c> out of iozone. This will cause iozone to clean up its child files on the mounts, and trigger the bug. Thanks for your time.
I'm not sure how to explain this bug short of this: When using a gluster distributed volume the quota xattr can take some time to update. If the files get deleted before the the quota can be re-calculated, glusterfs doesn't forget about the files. It continues to try to poll them for their size, even after it is told the file no longer exists, it then gets hung up on re-calculating, causing increased user CPU load on the server, with i/o throughput gradually decreasing to nothing. Here is a link to debug logs of all clients and servers involved: http://beocat.cis.ksu.edu/glusterfs-logs.tar.bz2 iozone command: RSH=`which ssh` /usr/sbin/iozone -+m host_file -i 0 -i 2 -t 10 -r 8192 -s 40g host_file contents: scout01 /tmp/gluster /usr/sbin/iozone paladin01 /tmp/gluster /usr/sbin/iozone rogue01 /tmp/gluster /usr/sbin/iozone titan01 /tmp/gluster /usr/sbin/iozone ranger01 /tmp/gluster /usr/sbin/iozone scout02 /tmp/gluster /usr/sbin/iozone paladin02 /tmp/gluster /usr/sbin/iozone rogue02 /tmp/gluster /usr/sbin/iozone titan02 /tmp/gluster /usr/sbin/iozone ranger02 /tmp/gluster /usr/sbin/iozone Gluster Volume info: root@[tiamat] /mnt/data/testing $ gluster volume info testing Volume Name: testing Type: Distribute Status: Stopped Number of Bricks: 2 Transport-type: tcp,rdma Bricks: Brick1: minotaur:/mnt/data/testing Brick2: tiamat:/mnt/data/testing Options Reconfigured: diagnostics.client-log-level: DEBUG diagnostics.brick-log-level: DEBUG features.quota: on features.quota-timeout: 0 xattrs of the filesystem on minotaur after the test: root@[minotaur] /mnt/data/testing $ getfattr -m . -d -e hex . # file: . trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000007ffffffe trusted.glusterfs.quota.dirty=0x3100 trusted.glusterfs.quota.size=0x0000000ae0a8a000 trusted.glusterfs.test=0x776f726b696e6700 root@[minotaur] /mnt/data/testing $ ls -lah total 12K drwxr-xr-x 2 root root 4.0K Jul 13 13:38 ./ drwxr-xr-x 6 root root 4.0K Jul 7 16:10 ../ xattrs of the filesystem on tiamat after the test: root@[tiamat] /mnt/data/testing $ getfattr -m . -d -e hex . # file: . trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.dirty=0x3100 trusted.glusterfs.quota.size=0x000000b34c94e000 trusted.glusterfs.test=0x776f726b696e6700 root@[tiamat] /mnt/data/testing $ ls -lah total 12K drwxr-xr-x 2 root root 4.0K Jul 13 13:38 ./ drwxr-xr-x 4 root root 4.0K Jul 6 19:51 ../
CHANGE: http://review.gluster.com/42 (Change-Id: I483caadbfd77d3830b1ea930a31eedddd0808fca) merged in release-3.2 by Anand Avati (avati)
CHANGE: http://review.gluster.com/32 (Change-Id: I6733586ba56ed4fdc2abbef63c38b8c491f89819) merged in master by Anand Avati (avati)
There are two issues involved in here: 1. memory leak. 2. Excessive logging. Patches have been sent to fix 2. Hence marking this bug as duplicate of another bug which traces mem-leak fixes in marker/quota. *** This bug has been marked as a duplicate of bug 3169 ***