Description of problem: ----------------------- With RHHI setup of 3 node gluster cluster in place with SSL/TLS encryption enabled, observed continuous error messages in glusterd logs in one of the node as follows: [2017-10-13 13:01:35.114409] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53419 failed (Input/output error) Version-Release number of selected component (if applicable): -------------------------------------------------------------- glusterfs-3.8.4-45.el7rhgs How reproducible: ----------------- Just observed it once, haven't tried to reproduce it Steps to Reproduce: ------------------- 1. Create a 3 node gluster cluster with RHGS 3.3.0 with TLS/SSL encryption enabled 2. Update the cluster to RHGS 3.3.1 interim build 3. After sometime, observe the glusterd logs Actual results: --------------- Socket poller errors seen with gluserd logs Expected results: ----------------- No errors in glusterd logs Additional info:
I see such a continuous error messages logged only in one node. [2017-10-13 13:01:35.114409] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53419 failed (Input/output error) [2017-10-13 13:01:43.127455] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53528 failed (Input/output error) [2017-10-13 13:01:53.245466] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53646 failed (Input/output error) [2017-10-13 13:02:01.992096] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53754 failed (Input/output error) [2017-10-13 13:02:10.152669] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.73:1007 failed (Input/output error) [2017-10-13 13:02:19.175127] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:53950 failed (Input/output error) [2017-10-13 13:02:27.191875] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.74:54054 failed (Input/output error) [2017-10-13 13:02:37.723957] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.73:58632 failed (Input/output error) [2017-10-13 13:02:46.043035] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.75:59950 failed (Input/output error)
Thanks to Mohit for the analysis. This had happened because of glusterd reporting back a stale port. An upstream patch https://review.gluster.org/#/c/18541 is posted for review.
*** Bug 1301460 has been marked as a duplicate of this bug. ***
Followed steps mentioned in above desc. i have found only one message of socket_poller error after brick restart in glusterd.log file. socket_poller errors are not generating continuously in glusterd.log file. Verified in below verison: glusterfs-server-3.12.2-9
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607