1384363 – Polling failure errors getting when volume is started&stopped with SSL enabled setup.

Bug 1384363 - Polling failure errors getting when volume is started&stopped with SSL enabled setup.

Summary: Polling failure errors getting when volume is started&stopped with SSL enable...

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	GlusterFS
Classification:	Community
Component:	glusterd
Sub Component:
Version:	3.9
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	high
Target Milestone:	---
Assignee:	bugs@gluster.org
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1336377 1349709 1359651 1359654 1384356
TreeView+	depends on / blocked

Reported:	2016-10-13 07:51 UTC by Mohit Agrawal
Modified:	2017-03-08 10:23 UTC (History)
CC List:	8 users (show)
Fixed In Version:	glusterfs-3.9.1
Clone Of:	1336377
Environment:
Last Closed:	2017-03-08 10:23:55 UTC
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Mohit Agrawal 2016-10-13 07:51:33 UTC

+++ This bug was initially created as a clone of Bug #1336377 +++

Description of problem:
========================
Polling errors are getting when volume is start/stopped with SSL enabled setup.

and

same errors are getting during volume mount and unmounting as well.


Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.7.9-4.


How reproducible:
=================
Always


Steps to Reproduce:
===================
1. Create a SSL enabled setup.
2. Create a volume having some 16 bricks 
3. Do volume start/stop and check for polling errors in the glusterd log.

Actual results:
===============
Polling errors are getting for every volume start/stop


Expected results:
=================



Additional info:

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-05-16 06:10:27 EDT ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Byreddy on 2016-05-16 06:13:00 EDT ---

glusterd logs:
=============

<======when volume is started======>


[2016-05-16 09:52:07.483398] I [MSGID: 106499] [glusterd-handler.c:4330:__glusterd_handle_status_volume] 0-management: Received status volume req for volume rep
[2016-05-16 09:52:07.485774] E [MSGID: 106301] [glusterd-syncop.c:1281:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume rep is not started
[2016-05-16 09:52:13.325452] W [common-utils.c:1685:gf_string2boolean] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(gd_mgmt_v3_commit_fn+0xf0) [0x7fc576dfc950] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_start_volume+0x290) [0x7fc576dc28c0] -->/lib64/libglusterfs.so.0(gf_string2boolean+0x15a) [0x7fc5821fb73a] ) 0-management: argument invalid [Invalid argument]
[2016-05-16 09:52:13.398033] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:13.413800] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick0/bk0 on port 49152
[2016-05-16 09:52:13.430562] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:13.430735] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:13.431007] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:13.499345] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:13.515744] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick1/bk0 on port 49153
[2016-05-16 09:52:13.534895] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:13.535064] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:13.535395] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:13.603310] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:13.621349] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick2/bk0 on port 49154
[2016-05-16 09:52:13.638290] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:13.638460] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:13.638863] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:13.708809] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:13.724575] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick3/bk0 on port 49155
[2016-05-16 09:52:13.742071] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:13.742251] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:13.742501] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:13.813232] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:13.829702] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick0/bk1 on port 49156
[2016-05-16 09:52:13.847995] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:13.848209] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:13.848641] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:13.926354] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:13.942977] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick2/bk1 on port 49157
[2016-05-16 09:52:13.963020] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:13.963214] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:13.963564] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.038193] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:14.055262] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick3/bk1 on port 49158
[2016-05-16 09:52:14.077888] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.078092] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.078363] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.151461] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:14.168077] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick0/bk2 on port 49159
[2016-05-16 09:52:14.190317] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.190474] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.190746] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.262959] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:14.277553] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick1/bk2 on port 49160
[2016-05-16 09:52:14.295524] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.295692] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.295956] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.365725] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:14.381544] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick2/bk2 on port 49161
[2016-05-16 09:52:14.403641] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.403821] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.404088] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.470570] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:14.504401] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.504559] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.504993] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.573547] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:14.612132] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.612302] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.612554] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.683388] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:14.720135] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.720296] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.720520] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.789022] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:14.822561] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.822721] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.823279] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:14.893807] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
The message "I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick2/bk3 on port 49165" repeated 3 times between [2016-05-16 09:51:08.838486] and [2016-05-16 09:52:14.803677]
[2016-05-16 09:52:14.910684] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /bricks/brick3/bk3 on port 49166
[2016-05-16 09:52:14.934639] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-05-16 09:52:14.934801] I [socket.c:4055:socket_init] 0-management: SSL support for glusterd is ENABLED
[2016-05-16 09:52:14.935325] E [socket.c:4133:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2016-05-16 09:52:15.064920] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2016-05-16 09:52:15.124551] W [socket.c:3133:socket_connect] 0-nfs: Ignore failed connection attempt on /var/run/gluster/cc08ba1e022b08edf83e1bba70d2fc3d.socket, (No such file or directory) 
[2016-05-16 09:52:15.126045] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2016-05-16 09:52:15.174712] W [socket.c:3133:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/dc84df026f0d2dd66e8bd459fd36f37f.socket, (No such file or directory) 
[2016-05-16 09:52:15.174888] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2016-05-16 09:52:15.174960] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2016-05-16 09:52:15.175899] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.
[2016-05-16 09:52:15.176196] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd.
[2016-05-16 09:52:15.187119] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.206099] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.282269] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.283533] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.298182] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.299659] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.321826] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.323155] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.338050] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.339179] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.361984] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.365179] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.402470] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.403339] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.419347] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.456885] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.468389] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.490295] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.526012] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.526992] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.564230] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.583094] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.584671] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.609349] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.613347] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.621450] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.636984] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.651759] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.658036] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.661861] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:15.678613] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:15.683450] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.317202] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.318065] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.346238] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.357536] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.374814] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.391817] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.393537] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.396179] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.406100] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.414622] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.421938] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.423806] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.441081] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.458005] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.480529] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.508235] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.538150] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.539743] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.541647] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.543067] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.569648] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.570737] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.589567] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.603357] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.634900] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.635707] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.690965] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.696569] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:16.703307] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-214.lab.eng.blr.redhat.com
[2016-05-16 09:52:16.722765] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.131551] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(+0x1000f2) [0x7fc576dd40f2] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x4c5) [0x7fc576dd3ba5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fc5822394e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=rep --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
[2016-05-16 09:52:17.153579] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.177691] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.195720] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.207231] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.213142] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.218921] E [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(+0x1000f2) [0x7fc576dd40f2] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x426) [0x7fc576dd3b06] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fc5822394e5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=rep --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
[2016-05-16 09:52:17.220451] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.220884] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.247920] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.248238] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.277702] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.310520] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.338383] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.344333] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.345362] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.347370] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.355068] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.367232] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.369563] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.386504] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.392319] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.393038] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.397042] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.398173] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.428729] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.458737] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.491721] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.517084] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.555626] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.600577] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.694193] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.786273] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.857640] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.864333] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.877636] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.895886] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.907712] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.934290] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.942715] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.954020] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.970637] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.983603] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.993546] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:17.994475] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:17.997091] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.005923] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.009310] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.063515] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:18.089358] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:18.117129] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.123398] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:18.133915] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:18.135017] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.139068] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.146396] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:18.181822] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.192003] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:18.231796] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.233741] I [socket.c:459:ssl_setup_connection] 0-socket.management: peer CN = dhcp43-215.lab.eng.blr.redhat.com
[2016-05-16 09:52:18.235831] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop
[2016-05-16 09:52:18.248806] E [socket.c:2616:socket_poller] 0-socket.management: error in polling loop


==========when volume is stopped ======           

[2016-05-16 09:53:44.996200] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick0/bk0 on port 49152
[2016-05-16 09:53:44.998600] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick2/bk0 on port 49154
[2016-05-16 09:53:45.000977] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick1/bk0 on port 49153
[2016-05-16 09:53:45.006302] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick3/bk0 on port 49155
[2016-05-16 09:53:45.007552] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/6fdaf263eec2597f9f743fc553c59409.socket failed (No data available)
[2016-05-16 09:53:45.007788] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick0/bk0 has disconnected from glusterd.
[2016-05-16 09:53:45.007829] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/fee34b93c7ee425f5a498db5d177f543.socket failed (No data available)
[2016-05-16 09:53:45.008032] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick1/bk0 has disconnected from glusterd.
[2016-05-16 09:53:45.008991] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick0/bk1 on port 49156
[2016-05-16 09:53:45.011172] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/b99e50c3c59c7cf2291bf163c803e4f9.socket failed (No data available)
[2016-05-16 09:53:45.011454] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick2/bk0 has disconnected from glusterd.
[2016-05-16 09:53:45.012389] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick2/bk1 on port 49157
[2016-05-16 09:53:45.014542] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/2a98d47f04567a314bc8aa95fbdf848d.socket failed (No data available)
[2016-05-16 09:53:45.014788] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick3/bk0 has disconnected from glusterd.
[2016-05-16 09:53:45.015913] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick3/bk1 on port 49158
[2016-05-16 09:53:45.019323] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick0/bk2 on port 49159
[2016-05-16 09:53:45.022072] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.022575] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.022752] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/b50cce35cd8582b0e69b79362d086892.socket failed (No data available)
[2016-05-16 09:53:45.022977] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick0/bk1 has disconnected from glusterd.
[2016-05-16 09:53:45.023040] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/af549c75677de971215010adb98ffad8.socket failed (No data available)
[2016-05-16 09:53:45.023228] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick2/bk1 has disconnected from glusterd.
[2016-05-16 09:53:45.027220] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.027512] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.027804] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.028090] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.028538] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.029977] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick1/bk2 on port 49160
[2016-05-16 09:53:45.030237] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.030311] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/b2d96a91cae80eac4066026a7cf0573a.socket failed (No data available)
[2016-05-16 09:53:45.034182] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.034702] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick3/bk1 has disconnected from glusterd.
[2016-05-16 09:53:45.034742] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/5e97b991212e14459122e2fc9d17dfc3.socket failed (No data available)
[2016-05-16 09:53:45.034976] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick0/bk2 has disconnected from glusterd.
[2016-05-16 09:53:45.035034] I [socket.c:2470:socket_event_handler] 0-transport: disconnecting now
[2016-05-16 09:53:45.035221] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick1/bk2 has disconnected from glusterd.
[2016-05-16 09:53:45.036447] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick2/bk2 on port 49161
[2016-05-16 09:53:45.041737] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick3/bk2 on port 49162
[2016-05-16 09:53:45.045019] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.045518] W [socket.c:701:__socket_rwv] 0-socket.management: writev on 10.70.43.214:1012 failed (No data available)
[2016-05-16 09:53:45.045599] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.045673] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/cffdf98196e94cf02fd5b51c6b69f2a8.socket failed (No data available)
[2016-05-16 09:53:45.045952] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick2/bk2 has disconnected from glusterd.
[2016-05-16 09:53:45.045997] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/cc208c6442417fa34ffc0485abf76748.socket failed (No data available)
[2016-05-16 09:53:45.046299] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick3/bk2 has disconnected from glusterd.
[2016-05-16 09:53:45.047096] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick0/bk3 on port 49163
[2016-05-16 09:53:45.051125] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/47f70604084d9f27be623faa3f2a826e.socket failed (No data available)
[2016-05-16 09:53:45.051344] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick0/bk3 has disconnected from glusterd.
[2016-05-16 09:53:45.051635] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.052889] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick1/bk3 on port 49164
[2016-05-16 09:53:45.057592] W [socket.c:701:__socket_rwv] 0-socket.management: writev on 10.70.43.214:1010 failed (No data available)
[2016-05-16 09:53:45.057820] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/f1d49685b4cf865922baadef964924fe.socket failed (No data available)
[2016-05-16 09:53:45.057835] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.058051] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick1/bk3 has disconnected from glusterd.
[2016-05-16 09:53:45.059204] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick2/bk3 on port 49165
[2016-05-16 09:53:45.064598] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /bricks/brick3/bk3 on port 49166
[2016-05-16 09:53:45.068353] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.069567] W [socket.c:701:__socket_rwv] 0-socket.management: writev on 10.70.43.214:1008 failed (No data available)
[2016-05-16 09:53:45.069667] E [socket.c:2611:socket_poller] 0-socket.management: poll error on socket
[2016-05-16 09:53:45.069700] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/aa27a64e14ef25c095b7045b6f5486d1.socket failed (No data available)
[2016-05-16 09:53:45.070034] I [MSGID: 106005] [glusterd-handler.c:5034:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.43.214:/bricks/brick2/bk3 has disconnected from glusterd.
[2016-05-16 09:53:45.070093] W [socket.c:701:__socket_rwv] 0-management: readv on /var/run/gluster/6a606b7fad73cb22831089d115fd11c2.socket failed (No data available)
[2016-05-16 09:53:45.125912] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_commit_hook+0x195) [0x7fc576d40d75] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x4c5) [0x7fc576dd3ba5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fc5822394e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=rep --last=yes
[2016-05-16 09:53:45.182698] E [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_commit_hook+0x195) [0x7fc576d40d75] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x426) [0x7fc576dd3b06] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fc5822394e5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --

--- Additional comment from Atin Mukherjee on 2016-05-16 07:19:21 EDT ---

Since it just adds up few error logs and doesn't impact any functionality, moving it to 3.2.0

--- Additional comment from Atin Mukherjee on 2016-06-30 00:18:33 EDT ---

Upstream patch : http://review.gluster.org/14786 posted for review.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-08-09 07:24:36 EDT ---

Since this bug has been approved for the RHGS 3.2.0 release of Red Hat Gluster Storage 3, through release flag 'rhgs-3.2.0+', and through the Internal Whiteboard entry of '3.2.0', the Target Release is being automatically set to 'RHGS 3.2.0'

--- Additional comment from Atin Mukherjee on 2016-09-17 10:34:38 EDT ---

Upstream mainline : http://review.gluster.org/14786
Upstream 3.8 : http://review.gluster.org/14799

And the fix is available in rhgs-3.2.0 as part of rebase to GlusterFS 3.8.4.

--- Additional comment from Rejy M Cyriac on 2016-09-17 12:32:55 EDT ---

Moving BZ to a transitional component in preparation for removing the 'glusterd' sub-component at the 'glusterd' component

--- Additional comment from Rejy M Cyriac on 2016-09-17 12:45:33 EDT ---

Moving BZ back to the 'glusterd' component after removal of the 'glusterd' sub-component

--- Additional comment from errata-xmlrpc on 2016-09-20 08:57:38 EDT ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHEA-2016:24863-02
https://errata.devel.redhat.com/advisory/24863

--- Additional comment from errata-xmlrpc on 2016-09-20 11:10:29 EDT ---

This bug has been dropped from advisory RHEA-2016:24863 by Atin Mukherjee (amukherj)

--- Additional comment from Byreddy on 2016-09-28 04:12:38 EDT ---

Verified this bug using the build - glusterfs-3.8.4-1
When the volume is stopped, i am seeing continuous below error messages (error in polling )populating in the glusterd log.




<log>
[2016-09-28 08:00:40.081637] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2016-09-28 08:00:40.081669] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2016-09-28 08:00:40.085279] I [MSGID: 106144] [glusterd-pmap.c:295:pmap_registry_remove] 0-pmap: removing brick /bricks/brick1/g2 on port 49153
[2016-09-28 08:00:40.085483] W [socket.c:590:__socket_rwv] 0-socket.management: writev on 10.70.43.190:1021 failed (No data available)
[2016-09-28 08:00:40.085590] E [socket.c:2527:socket_poller] 0-socket.management: poll error on socket
[2016-09-28 08:00:40.086214] I [MSGID: 101053] [mem-pool.c:617:mem_pool_destroy] 0-management: size=588 max=1 total=1
[2016-09-28 08:00:40.086290] I [MSGID: 101053] [mem-pool.c:617:mem_pool_destroy] 0-management: size=124 max=1 total=1
[2016-09-28 08:00:40.086385] W [socket.c:590:__socket_rwv] 0-management: readv on /var/run/gluster/392a062d3823c492808887c960de0edc.socket failed (No data available)
[2016-09-28 08:00:40.086930] I [MSGID: 101053] [mem-pool.c:617:mem_pool_destroy] 0-management: size=588 max=1 total=1
[2016-09-28 08:00:40.086971] I [MSGID: 101053] [mem-pool.c:617:mem_pool_destroy] 0-management: size=124 max=1 total=1
[2016-09-28 08:00:50.686934] I [socket.c:343:ssl_setup_connection] 0-socket.management: peer CN = dhcp41-163.lab.eng.blr.redhat.com
[2016-09-28 08:00:50.686986] I [socket.c:346:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: )
[2016-09-28 08:00:50.693754] E [socket.c:2532:socket_poller] 0-socket.management: error in polling loop
[2016-09-28 08:00:50.698450] I [socket.c:343:ssl_setup_connection] 0-socket.management: peer CN = dhcp41-163.lab.eng.blr.redhat.com
[2016-09-28 08:00:50.698474] I [socket.c:346:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: )
[2016-09-28 08:00:50.699452] E [socket.c:2532:socket_poller] 0-socket.management: error in polling loop
[2016-09-28 08:00:50.709792] I [socket.c:343:ssl_setup_connection] 0-socket.management: peer CN = dhcp41-163.lab.eng.blr.redhat.com
[2016-09-28 08:00:50.709827] I [socket.c:346:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: )
[2016-09-28 08:00:50.710715] E [socket.c:2532:socket_poller] 0-socket.management: error in polling loop
[2016-09-28 08:00:53.760961] I [socket.c:343:ssl_setup_connection] 0-socket.management: peer CN = dhcp41-163.lab.eng.blr.redhat.com
[2016-09-28 08:00:53.760986] I [socket.c:346:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: )
[2016-09-28 08:00:53.762820] E [socket.c:2532:socket_poller] 0-socket.management: error in polling loop
[2016-09-28 08:00:53.772367] I [socket.c:343:ssl_setup_connection] 0-socket.management: peer CN = dhcp41-163.lab.eng.blr.redhat.com
[2016-09-28 08:00:53.772390] I [socket.c:346:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: )
[2016-09-28 08:00:53.773339] E [socket.c:2532:socket_poller] 0-socket.management: error in polling loop
[2016-09-28 08:00:53.789946] I [socket.c:343:ssl_setup_connection] 0-socket.management: peer CN = dhcp41-163.lab.eng.blr.redhat.com
[2016-09-28 08:00:53.789968] I [socket.c:346:ssl_setup_connection] 0-socket.management: SSL verification succeeded (client: )


<log_end>

i am not seeing these continuous errors when the volume is started.

this will consume lot of storage for the logs if volume is taken for maintenance.

Moving back to assigned state.

--- Additional comment from Mohit Agrawal on 2016-10-03 01:33:58 EDT ---

Patch is posted on upstream (mainline) 
http://review.gluster.org/#/c/15605/1

Comment 1 Worker Ant 2016-10-13 07:58:32 UTC

REVIEW: http://review.gluster.org/15632 (rpc/socket.c : Modify gf_log message in socket_poller code in case of error) posted (#1) for review on release-3.9 by MOHIT AGRAWAL (moagrawa)

Comment 2 Worker Ant 2016-10-26 13:12:41 UTC

REVIEW: http://review.gluster.org/15632 (rpc/socket.c : Modify gf_log message in socket_poller code in case of error) posted (#2) for review on release-3.9 by MOHIT AGRAWAL (moagrawa)

Comment 3 Worker Ant 2017-01-10 06:28:50 UTC

COMMIT: http://review.gluster.org/15632 committed in release-3.9 by Raghavendra G (rgowdapp) 
------
commit 0e6a8452446003e9510cfad1dbece5c0b7151fdc
Author: Mohit Agrawal <moagrawa>
Date:   Wed Oct 26 18:41:42 2016 +0530

    rpc/socket.c : Modify gf_log message in socket_poller code in case of error
    
    Problem:  In case of SSL after stopping the volume if client(mount point) is
              still trying to write the data on socket then it will throw an EIO
              error on that socket and given this log message is captured at every
              attempt this would flood the log file.
    
    Solution: To reduce the frequency of stored log message use GF_LOG_OCCASIONALLY
              instead of gf_log.
    
    > BUG: 1381115
    > Change-Id: I66151d153c2cbfb017b3ebc4c52162278c0f537c
    > Signed-off-by: Mohit Agrawal <moagrawa>
    > Reviewed-on: http://review.gluster.org/15605
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Atin Mukherjee <amukherj>
    > (cherry picked from commit 070145750006c87099f945b4990a4460d814c21f)
    
    Change-Id: I9536dcb2cbf3f8082dccf049cbfb21a034f5eac1
    BUG: 1384363
    Signed-off-by: Mohit Agrawal <moagrawa>
    Reviewed-on: http://review.gluster.org/15632
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Atin Mukherjee <amukherj>
    Reviewed-by: Raghavendra G <rgowdapp>

Comment 4 Kaushal 2017-03-08 10:23:55 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.1, please open a new bug report.

glusterfs-3.9.1 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-January/029725.html
[2] https://www.gluster.org/pipermail/gluster-users/

Note You need to log in before you can comment on or make changes to this bug.