Bug 818167

Summary:	Too many open fds after the graph change
Product:	[Community] GlusterFS	Reporter:	Anush Shetty <ashetty>
Component:	fuse	Assignee:	Raghavendra G <rgowdapp>
Status:	CLOSED UPSTREAM	QA Contact:
Severity:	medium	Docs Contact:
Priority:	high
Version:	mainline	CC:	amarts, gluster-bugs, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-05-14 10:49:15 UTC	Type:	Bug
Regression:	---	Mount Type:	fuse
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Anush Shetty 2012-05-02 11:14:38 UTC

Description of problem: While running tests and doing graph change using volume set in a loop, the I/O on the fuse client exited with an ENOTCONN. It also made glusterd unresponsive.

This was a 2x2 Distributed-replicate setup.


Version-Release number of selected component (if applicable): 3.3.0qa38 (RHS AMI)


Steps to Reproduce:
1.  while true; do gluster volume set amitest performance.write-behind off; sleep 1; gluster volume set amitest performance.write-behind on; done

2.  Run I/O on the mount point
3.
  
Actual results:

Mount point returned ENOTCONN.

Glusterd was unresponsive.


Expected results:


Additional info:

Glusterd log:

[2012-05-02 10:32:48.484114] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484132] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484150] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484168] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484199] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484220] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484238] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484256] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484273] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484291] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484309] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484327] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484345] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484363] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484381] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484399] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484416] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484434] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
[2012-05-02 10:32:48.484452] W [socket.c:1851:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)

Comment 1 Raghavendra G 2012-05-10 06:06:43 UTC

This is most likely a duplicate of bug #810089 - the fix to which did not go into v3.3.0qa38. I checked it on v3.3.0qa39 (which had the fix) and the number of sockets opened by glusterd did not go up.

@Anush,

Can you confirm my observations and if true, please mark this bug as duplicate of bug #810089.

regards,
Raghavendra.

Comment 2 Amar Tumballi 2012-05-11 07:10:08 UTC

need a update with qa40 or beyond.

Comment 3 Anush Shetty 2012-05-14 10:49:15 UTC

Don't see this issue with qa41. Closing it now.