After starting the client, a rapid sequence of these errors results in the last entry never being fully written an the log never goes any further. After a couple of days, this client is dying but I have no idea why as there's no log beyond where it halts. [root@mysql1 glusterfs]# tail -n 5 /var/log/glusterfs/mnt-gluster-mysql1.log [2014-07-05 15:00:02.238570] E [iobuf.c:733:iobuf_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7f028b9e0492]))) 0-iobuf: invalid argument: iobuf [2014-07-05 15:00:02.238613] E [iobuf.c:828:iobref_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7f028b9e0492]))) 0-iobuf: invalid argument: iobref [2014-07-05 15:00:02.243550] E [iobuf.c:733:iobuf_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7f028b9e0492]))) 0-iobuf: invalid argument: iobuf [2014-07-05 15:00:02.243594] E [iobuf.c:828:iobref_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7f028b9e0492]))) 0-iobuf: invalid argument: iobref [2014-07-05 15:00:02.295156] E [iobuf.c:733:iobuf_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/pe[root@mysql1 glusterfs]# Version-Release number of selected component (if applicable): 3.4.4-2 Volume Name: mysql1 Type: Distributed-Replicate Volume ID: 713e6220-7eb2-4d12-9cd7-fe800b9b741a Status: Started Number of Bricks: 4 x 3 = 12 Transport-type: tcp Bricks: Brick1: ewcs2:/var/spool/glusterfs/a_mysql1 Brick2: ewcs10:/data/glusterfs/mysql1/a/brick Brick3: ewcs7:/data/glusterfs/mysql1/a/brick Brick4: ewcs2:/var/spool/glusterfs/b_mysql1 Brick5: ewcs10:/data/glusterfs/mysql1/b/brick Brick6: ewcs7:/data/glusterfs/mysql1/b/brick Brick7: ewcs2:/var/spool/glusterfs/c_mysql1 Brick8: ewcs10:/data/glusterfs/mysql1/c/brick Brick9: ewcs7:/data/glusterfs/mysql1/c/brick Brick10: ewcs2:/data/glusterfs/mysql1/d/brick Brick11: ewcs10:/data/glusterfs/mysql1/d/brick Brick12: ewcs7:/data/glusterfs/mysql1/d/brick Options Reconfigured: server.allow-insecure: on performance.write-behind: off nfs.disable: on performance.cache-size: 8MB cluster.background-self-heal-count: 2 cluster.data-self-heal-algorithm: full
REVIEW: http://review.gluster.org/8242 (performance/quick-read: Check for iobuf and iobref before unref.) posted (#1) for review on release-3.4 by Vijay Bellur (vbellur)
COMMIT: http://review.gluster.org/8242 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) ------ commit b3bf2cfaa8e932f02a10a42e0281290a7cbdc960 Author: Vijay Bellur <vbellur> Date: Sat Jul 5 21:19:01 2014 +0530 performance/quick-read: Check for iobuf and iobref before unref. In the absence of this check, iobuf_unref() can cause a log flood for read operations. Change-Id: Ib8beef6e55b69ecfb566bf61a96215ef0d0d74dc BUG: 1116514 Signed-off-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/8242 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Humble Devassy Chirammal <humble.devassy> Reviewed-by: Prashanth Pai <ppai> Reviewed-by: Kaleb KEITHLEY <kkeithle>