Bug 1503244

Summary:	socket poller error in glusterd logs
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	SATHEESARAN <sasundar>
Component:	glusterd	Assignee:	Mohit Agrawal <moagrawa>
Status:	CLOSED ERRATA	QA Contact:	Rajesh Madaka <rmadaka>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.3	CC:	amukherj, nchilaka, nerawat, rhinduja, rhs-bugs, sasundar, storage-qa-internal, vbellur
Target Milestone:	---
Target Release:	RHGS 3.4.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1526371 (view as bug list)		Environment:
Last Closed:	2018-09-04 06:38:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1503246, 1507747, 1507749
Bug Blocks:	1503134, 1526371, 1530512

Description SATHEESARAN 2017-10-17 15:56:59 UTC

Description of problem:
-----------------------
With RHHI setup of 3 node gluster cluster in place with SSL/TLS encryption enabled, observed continuous error messages in glusterd logs in one of the node as follows:
[2017-10-13 13:01:35.114409] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53419 failed (Input/output error)

Version-Release number of selected component (if applicable):
--------------------------------------------------------------
glusterfs-3.8.4-45.el7rhgs

How reproducible:
-----------------
Just observed it once, haven't tried to reproduce it

Steps to Reproduce:
-------------------
1. Create a 3 node gluster cluster with RHGS 3.3.0 with TLS/SSL encryption enabled
2. Update the cluster to RHGS 3.3.1 interim build
3. After sometime, observe the glusterd logs

Actual results:
---------------
Socket poller errors seen with gluserd logs


Expected results:
-----------------
No errors in glusterd logs

Additional info:

Comment 1 SATHEESARAN 2017-10-17 15:57:34 UTC

I see such a continuous error messages logged only in one node.

[2017-10-13 13:01:35.114409] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53419 failed (Input/output error)
[2017-10-13 13:01:43.127455] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53528 failed (Input/output error)
[2017-10-13 13:01:53.245466] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53646 failed (Input/output error)
[2017-10-13 13:02:01.992096] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53754 failed (Input/output error)
[2017-10-13 13:02:10.152669] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.73:1007 failed (Input/output error)
[2017-10-13 13:02:19.175127] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53950 failed (Input/output error)
[2017-10-13 13:02:27.191875] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:54054 failed (Input/output error)
[2017-10-13 13:02:37.723957] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.73:58632 failed (Input/output error)
[2017-10-13 13:02:46.043035] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.75:59950 failed (Input/output error)

Comment 2 Atin Mukherjee 2017-10-17 16:11:46 UTC

Thanks to Mohit for the analysis. This had happened because of glusterd reporting back a stale port.

An upstream patch https://review.gluster.org/#/c/18541 is posted for review.

Comment 5 Raghavendra G 2017-12-01 04:19:57 UTC

*** Bug 1301460 has been marked as a duplicate of this bug. ***

Comment 12 Rajesh Madaka 2018-05-15 17:49:33 UTC

Followed steps mentioned in above desc.

i have found only one message of socket_poller error after brick restart in glusterd.log file.

socket_poller errors are not generating continuously in glusterd.log file.

Verified in below verison:


glusterfs-server-3.12.2-9

Comment 14 errata-xmlrpc 2018-09-04 06:38:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607