Bug 809715

Summary:	[ 3c5aed7d8226e50296c6a7106e0d59682732e9b5] ENOTCONN after replace-brick commit
Product:	[Community] GlusterFS	Reporter:	Anush Shetty <ashetty>
Component:	glusterd	Assignee:	krishnan parthasarathi <kparthas>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	gluster-bugs, nsathyan
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-04-19 04:25:45 UTC	Type:	Bug
Regression:	---	Mount Type:	fuse
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Anush Shetty 2012-04-04 07:19:11 UTC

Description of problem: After committing replace-brick operation, any operation on the mount point returned ENOTCONN


Version-Release number of selected component (if applicable): Upstream


How reproducible: Consistently


Steps to Reproduce:
1. start replace-brick
2. commit replace-brick
3. ls /mnt/gluster
  
Actual results: Return ENOTCONN


Expected results: Shouldn't error out


Additional info:

Client log:
[2012-04-04 12:43:27.444919] D [glusterfsd-mgmt.c:1608:mgmt_getspec_cbk] 0-glusterfsd-mgmt: No need to re-load volfile, reconfigure done
[2012-04-04 12:43:27.444956] I [glusterfsd-mgmt.c:1584:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2012-04-04 12:43:29.582652] D [client.c:189:client_submit_request] 0-test2-client-0: connection in disconnected state
[2012-04-04 12:43:29.582715] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-test2-client-0: remote operation failed: Transport endpoint is not connected. Path: /
[2012-04-04 12:43:29.582742] W [fuse-bridge.c:694:fuse_fd_cbk] 0-glusterfs-fuse: 54: OPENDIR() / => -1 (Transport endpoint is not connected)
[2012-04-04 12:43:30.160428] D [client.c:189:client_submit_request] 0-test2-client-0: connection in disconnected state
[2012-04-04 12:43:30.160485] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-test2-client-0: remote operation failed: Transport endpoint is not connected. Path: /
[2012-04-04 12:43:30.160511] W [fuse-bridge.c:694:fuse_fd_cbk] 0-glusterfs-fuse: 55: OPENDIR() / => -1 (Transport endpoint is not connected)
[2012-04-04 12:43:30.571687] D [client.c:189:client_submit_request] 0-test2-client-0: connection in disconnected state
[2012-04-04 12:43:30.571746] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-test2-client-0: remote operation failed: Transport endpoint is not connected. Path: /
[2012-04-04 12:43:30.571772] W [fuse-bridge.c:694:fuse_fd_cbk] 0-glusterfs-fuse: 56: OPENDIR() / => -1 (Transport endpoint is not connected)
[2012-04-04 12:43:32.248989] D [client.c:189:client_submit_request] 0-test2-client-0: connection in disconnected state
[2012-04-04 12:43:32.249069] W [client3_1-fops.c:2607:client3_1_lookup_cbk] 0-test2-client-0: remote operation failed: Transport endpoint is not connected. Path: /dot
[2012-04-04 12:43:32.249155] D [client.c:189:client_submit_request] 0-test2-client-0: connection in disconnected state
[2012-04-04 12:43:32.249178] W [client3_1-fops.c:2607:client3_1_lookup_cbk] 0-test2-client-0: remote operation failed: Transport endpoint is not connected. Path: /dot
[2012-04-04 12:43:32.249203] W [fuse-bridge.c:272:fuse_entry_cbk] 0-glusterfs-fuse: 57: LOOKUP() /dot => -1 (Transport endpoint is not connected)
[2012-04-04 12:43:32.249349] D [client.c:189:client_submit_request] 0-test2-client-0: connection in disconnected state
[2012-04-04 12:43:32.249374] W [client3_1-fops.c:2607:client3_1_lookup_cbk] 0-test2-client-0: remote operation failed: Transport endpoint is not connected. Path: /dot
[2012-04-04 12:43:32.249439] D [client.c:189:client_submit_request] 0-test2-client-0: connection in disconnected state
[2012-04-04 12:43:32.249460] W [client3_1-fops.c:2607:client3_1_lookup_cbk] 0-test2-client-0: remote operation failed: Transport endpoint is not connected. Path: /dot
[2012-04-04 12:43:32.249484] W [fuse-bridge.c:272:fuse_entry_cbk] 0-glusterfs-fuse: 58: LOOKUP() /dot => -1 (Transport endpoint is not connected)

Comment 1 krishnan parthasarathi 2012-04-16 13:15:05 UTC

Works for me in dc4c44d51e3eb4b53e78840bf223911e48ce62b5.

Comment 2 Anush Shetty 2012-04-18 11:59:30 UTC

Hi KP, I still see this issue with 7ef32ae76d1c1e4a5ff47899d175be9fdeb73bc8

Reopening it.

Comment 3 krishnan parthasarathi 2012-04-19 02:50:17 UTC

Anush,
Could you provide more information, like volume configuration (volume info)? It would be helpful if you could attach all logs (client, bricks and glusterd) when the problem is observed.

Comment 4 krishnan parthasarathi 2012-04-19 04:25:45 UTC

The reason the lookup on the mount point fails for a short period of time after replace-brick commit, is because the commit operation restarts the destination brick. This 'disconnection' is noticed by the application only when you perform replace-brick on a 'single-brick volume'. This is a known limitation of replace-brick.