Hide Forgot
Description of problem: ======================== when running dbench on fuse mount for a replicate volume ( 1 x 2 )and one of the bricks goes offline and comes back online, the brick reports the following log messages : [2013-03-01 11:34:28.029222] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993888: OPEN (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.031579] I [server3_1-fops.c:252:server_inodelk_cbk] 0-vol-rep-1-server: 993896: INODELK (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.034041] I [server3_1-fops.c:252:server_inodelk_cbk] 0-vol-rep-1-server: 993914: INODELK (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.040435] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 993938: FLUSH -2 (--) ==> -1 (No such file or directory) Version-Release number of selected component (if applicable): ============================================================ [root@luigi:~] Mar-01-2013 12:35:05 $ gluster --version glusterfs 3.3.0.5rhs built on Feb 19 2013 05:31:58 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. How reproducible: ================== Often Steps to Reproduce: =================== 1. Create replicate volime ( 1 x 2 ) . start the volume 2. Create fuse mount. 3. run : " dbench -c /usr/local/share/doc/dbench/loadfiles/client.txt -D <mount_ponit> --stat-check --per-client-results 10 4. while dbench is running, bring offline one of the bricks. 5. bring back the the brick online Actual results: ================= [2013-03-01 11:34:28.029222] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993888: OPEN (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.031579] I [server3_1-fops.c:252:server_inodelk_cbk] 0-vol-rep-1-server: 993896: INODELK (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.034041] I [server3_1-fops.c:252:server_inodelk_cbk] 0-vol-rep-1-server: 993914: INODELK (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.040435] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 993938: FLUSH -2 (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.040546] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993939: OPEN (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.041595] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993952: OPEN (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.043890] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993964: OPEN (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.045784] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 993975: FLUSH -2 (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.076810] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 994130: OPEN (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.078079] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 994142: FLUSH -2 (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.297736] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 994954: FLUSH -2 (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.329278] E [posix.c:361:posix_setattr] 0-vol-rep-1-posix: setattr (lstat) on /home/brick0/.glusterfs/56/cb/56cb58de-5d1d-4d72-a264-d93f74b40345 fail ed: No such file or directory Expected Output:- ================= OPEN, INODELK, FLUSH shouldn't refer to null gfid/path
REVIEW: http://review.gluster.org/6252 (server/rpc: bricks goes offline and comes back online, with lots of "No such file or directory" log messages) posted (#1) for review on master by Vikhyat Umrao (vumrao)
COMMIT: http://review.gluster.org/6252 committed in master by Anand Avati (avati) ------ commit eeac099cf405849f07989c410968e77f95acdc77 Author: Vikhyat Umrao <vumrao> Date: Tue Nov 12 18:11:23 2013 +0530 server/rpc: bricks goes offline and comes back online, with lots of "No such file or directory" log messages Problem: Messages were getting logged very frequently at log level INFO. [2013-03-01 11:34:28.029222] I [server3_1-fops.c:1541:server_open_cbk] vol-server: 993888: OPEN (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.031579] I [server3_1-fops.c:252:server_inodelk_cbk] vol-server: 993896: INODELK (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.034041] I [server3_1-fops.c:252:server_inodelk_cbk] vol-server: 993914: INODELK (null) (--) ==> -1 (No such file or directory) [2013-03-01 11:34:28.040435] I [server3_1-fops.c:1338:server_flush_cbk] vol-server: 993938: FLUSH -2 (--) ==> -1 (No such file or directory) Solution: Moved them to DEBUG log level if error number equlas to ENOENT else to ERROR log level. It will help in decreasing the size of log files at INFO log level. For server_open_cbk and for some other functions we already have a patch, below is the URL for it. URL- http://review.gluster.org/6241 This patch solves logging problem for functions server_inodelk_cbk and server_flush_cbk. Change-Id: I57372e851371e466f1674726015e28378b826f5f BUG: 1029372 Signed-off-by: Vikhyat Umrao<vumrao> Reviewed-on: http://review.gluster.org/6252 Reviewed-by: Niels de Vos <ndevos> Tested-by: Gluster Build System <jenkins.com>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report. glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user