Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 969353

Summary:	Excessive logging in the self-heal daemon after a replace-brick
Product:	[Community] GlusterFS	Reporter:	Niels de Vos <ndevos>
Component:	replicate	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED EOL	QA Contact:
Severity:	high	Docs Contact:
Priority:	medium
Version:	3.4.0-alpha	CC:	bugs, gluster-bugs, spandura
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-10-07 14:05:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	969355, 1151303, 1155027, 1155073

Description Niels de Vos 2013-05-31 09:47:14 UTC

Description of problem:
After a replace-brick, the glustershd.log grew to 8GB in less one week. Almost
all entries look like (different gfids):

[2013-05-20 20:36:01.730851] W [client3_1-fops.c:473:client3_1_open_cbk] 0-web-client-1: remote operation failed: No such file or directory. Path: <gfid:4cfbc34f-9227-476d-bd23-b50613c94170> (00000000-0000-0000-0000-000000000000)
[2013-05-20 20:36:01.730883] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-web-replicate-0: open of <gfid:4cfbc34f-9227-476d-bd23-b50613c94170> failed on child web-client-1 (No such file or directory)
[2013-05-20 20:36:01.734192] W [client3_1-fops.c:1556:client3_1_inodelk_cbk] 0-web-client-1: remote operation failed: No such file or directory
[2013-05-20 20:36:01.735905] W [client3_1-fops.c:1656:client3_1_entrylk_cbk] 0-web-client-1: remote operation failed: No such file or directory
[2013-05-20 20:36:01.736096] E [afr-self-heal-entry.c:2352:afr_sh_post_nonblocking_entry_cbk] 0-web-replicate-0: Non Blocking entrylks failed for <gfid:351b61dc-29ed-4ef0-8b48-0c413b3fc370>.
[2013-05-20 20:36:01.739690] W [client3_1-fops.c:1556:client3_1_inodelk_cbk] 0-web-client-1: remote operation failed: No such file or directory
[2013-05-20 20:36:01.741376] W [client3_1-fops.c:1656:client3_1_entrylk_cbk] 0-web-client-1: remote operation failed: No such file or directory
[2013-05-20 20:36:01.741565] E [afr-self-heal-entry.c:2352:afr_sh_post_nonblocking_entry_cbk] 0-web-replicate-0: Non Blocking entrylks failed for <gfid:a47a96d9-556c-4d5c-b428-04be961ee19a>.
[2013-05-20 20:36:01.744961] W [client3_1-fops.c:1556:client3_1_inodelk_cbk] 0-web-client-1: remote operation failed: No such file or directory
[2013-05-20 20:36:01.746670] W [client3_1-fops.c:473:client3_1_open_cbk] 0-web-client-1: remote operation failed: No such file or directory. Path: <gfid:d0a94a9c-80d2-4778-8f25-7f9de6bce6a8> (00000000-0000-0000-0000-000000000000)
[2013-05-20 20:36:01.746702] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-web-replicate-0: open of <gfid:d0a94a9c-80d2-4778-8f25-7f9de6bce6a8> failed on child web-client-1 (No such file or directory)
[2013-05-20 20:36:01.750047] W [client3_1-fops.c:1556:client3_1_inodelk_cbk] 0-web-client-1: remote operation failed: No such file or directory
[2013-05-20 20:36:01.751756] W [client3_1-fops.c:473:client3_1_open_cbk] 0-web-client-1: remote operation failed: No such file or directory. Path: <gfid:f032cd09-afc3-4b72-8b62-51b4854ab37e> (00000000-0000-0000-0000-000000000000)
[2013-05-20 20:36:01.751787] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-web-replicate-0: open of <gfid:f032cd09-afc3-4b72-8b62-51b4854ab37e> failed on child web-client-1 (No such file or directory)
[2013-05-20 20:36:01.755104] W [client3_1-fops.c:1556:client3_1_inodelk_cbk] 0-web-client-1: remote operation failed: No such file or directory
[2013-05-20 20:36:01.756803] W [client3_1-fops.c:473:client3_1_open_cbk] 0-web-client-1: remote operation failed: No such file or directory. Path: <gfid:0d0129ba-4c2f-48e7-8c24-5d2886e52654> (00000000-0000-0000-0000-000000000000)


Version-Release number of selected component (if applicable):
current master

How reproducible:
100%

Steps to Reproduce:
1. replace a brick of a replicated volume
2. incorrectly heal with 'gluster volume MYVOL heal' (missing 'full')

Actual results:
Immense big logs.

Expected results:
Logs don't grow this enormously.

Additional info:

Comment 1 Niels de Vos 2015-05-17 22:01:19 UTC

GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5.

This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "bugs".

If there is no response by the end of the month, this bug will get automatically closed.

Comment 2 Kaleb KEITHLEY 2015-10-07 14:05:11 UTC

GlusterFS 3.4.x has reached end-of-life.

If this bug still exists in a later release please reopen this and change the version or open a new bug.