Bug 1381439 - [Eventing]: 'gluster vol start vol1' triggers CLIENT_CONNECT/DISCONNECT events for bricks of ALL volumes of the cluster
Summary: [Eventing]: 'gluster vol start vol1' triggers CLIENT_CONNECT/DISCONNECT event...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: Bug Updates Notification Mailing List
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-04 06:29 UTC by Sweta Anandpara
Modified: 2018-11-06 08:25 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-06 08:25:02 UTC
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Sweta Anandpara 2016-10-04 06:29:06 UTC 
Description of problem:
======================
Have a 4 node cluster with eventing enabled, and create (say) 3 volumes - vol1, vol2 and vol3. Start one of the volumes, say, vol1, using 'gluster volume start vol1'. Monitor the events seen, and we see a mention of bricks which belong to 'vol2' and 'vol3'. 
'CLIENT_CONNECT' event is seen for all the brick paths which belong to 'vol2' and 'vol3'. That is followed by a CLIENT_DISCONNECT event for same paths, undoing what was done by CLIENT_CONNECT. 

It does not harm to see extra events, nor does it hamper the CLIENT_CONNECTS that are expected to be seen for the volume in question - 'vol1' (yet). But it makes no sense to have extra traffic of events which are unrelated.


Version-Release number of selected component (if applicable):
============================================================
3.8.4-2


How reproducible:
=================
Always


Steps to Reproduce:
===================
1. Have a 4 node cluster, with 3 volumes 'disp' (disperse 4+2), 'testvol' (dist-rep 2*2) and 'ozone' (distribute 1*2)
2. Configure eventing, and register a webhook.
2. If all the volume are already started, stop one of the volumes 'disp' and start it again using the command 'gluster volume start disp'
3. Monitor the events seen at the listener.

Actual results:
=================
44 CLIENT_CONNECT events are seen, and only 24 of which belong to the volume in question 'disp'. 20 of them are related to other volumes 'testvol' and 'ozone'.

20 CLIENT_DISCONNECT events are seen later, related to the volumes 'testvol' and 'ozone'. 


Expected results:
=================
CLIENT_CONNECT events should be seen only belonging to the volume in which there is a state change. The remaining 20 CLIENT_CONNECTS and the corresponding 20 CLIENT_DISCONNECTS are NA.

Additional info:
================


[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]# gluster-eventsapi status
Webhooks: 
http://10.70.46.159:9000/listen

+--------------+-------------+-----------------------+
|     NODE     | NODE STATUS | GLUSTEREVENTSD STATUS |
+--------------+-------------+-----------------------+
| 10.70.46.240 |          UP |                    UP |
| 10.70.46.242 |          UP |                    UP |
| 10.70.46.218 |          UP |                    UP |
|  localhost   |          UP |                    UP |
+--------------+-------------+-----------------------+
[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]# gluster v bitrot disp enable
Bitrot command failed : Volume is stopped, start volume before executing bit rot command.
[root@dhcp46-239 ~]# gluster v start disp
volume start: disp: success
[root@dhcp46-239 ~]# gluster v info ozone
 
Volume Name: ozone
Type: Distribute
Volume ID: f8299fbd-871f-4407-a6e1-50b59b7e716c
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.70.46.239:/bricks/brick0/ozone
Brick2: 10.70.46.240:/bricks/brick0/ozone
Options Reconfigured:
performance.readdir-ahead: on
transport.address-family: inet
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: on
features.bitrot: on
features.scrub: Active
[root@dhcp46-239 ~]# gluster v info testvol
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 458d6608-8454-4fd6-939d-6ff18f2fba8f
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.46.239:/bricks/brick2/testvol_brick0
Brick2: 10.70.46.240:/bricks/brick2/testvol_brick1
Brick3: 10.70.46.242:/bricks/brick1/testvol_brick2
Brick4: 10.70.46.218:/bricks/brick1/testvol_brick3
Options Reconfigured:
features.scrub-freq: hourly
features.scrub-throttle: aggressive
features.scrub: Inactive
features.bitrot: off
transport.address-family: inet
performance.readdir-ahead: on
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: on
[root@dhcp46-239 ~]# 


===============
EVENTS SEEN
===============

{u'message': {u'peer': u'10.70.46.239', u'volume': u'disp', u'brick': u'/bricks/brick0/disp'}, u'event': u'BRICK_CONNECTED', u'ts': 1475558483, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:1000', u'server_identifier': u'10.70.46.239:49153', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-27451-2016/10/03-09:13:05:194001-testvol-client-0-0-0', u'brick_path': u'/bricks/brick2/testvol_brick0'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558483, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:986', u'server_identifier': u'10.70.46.242:49152', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-27451-2016/10/03-09:13:05:194001-testvol-client-2-0-0', u'brick_path': u'/bricks/brick1/testvol_brick2'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558483, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.239:987', u'server_identifier': u'10.70.46.218:49152', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-27451-2016/10/03-09:13:05:194001-testvol-client-3-0-0', u'brick_path': u'/bricks/brick1/testvol_brick3'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558483, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.239:970', u'server_identifier': u'10.70.46.240:49153', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-27451-2016/10/03-09:13:05:194001-testvol-client-1-0-0', u'brick_path': u'/bricks/brick2/testvol_brick1'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558483, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.239:976', u'server_identifier': u'10.70.46.239:49152', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-27489-2016/10/03-09:13:07:209335-ozone-client-0-0-0', u'brick_path': u'/bricks/brick0/ozone'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558484, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:973', u'server_identifier': u'10.70.46.239:49152', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-27499-2016/10/03-09:13:08:216125-ozone-client-0-0-0', u'brick_path': u'/bricks/brick0/ozone'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558485, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'peer': u'10.70.46.239', u'volume': u'disp', u'brick': u'/bricks/brick1/disp'}, u'event': u'BRICK_CONNECTED', u'ts': 1475558486, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'peer': u'10.70.46.240', u'volume': u'disp', u'brick': u'/bricks/brick0/disp'}, u'event': u'BRICK_CONNECTED', u'ts': 1475558487, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.242:1017', u'server_identifier': u'10.70.46.239:49153', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-4479-2016/10/03-09:13:04:365388-testvol-client-0-0-0', u'brick_path': u'/bricks/brick2/testvol_brick0'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.242:1022', u'server_identifier': u'10.70.46.242:49152', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-4479-2016/10/03-09:13:04:365388-testvol-client-2-0-0', u'brick_path': u'/bricks/brick1/testvol_brick2'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.242:1021', u'server_identifier': u'10.70.46.218:49152', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-4479-2016/10/03-09:13:04:365388-testvol-client-3-0-0', u'brick_path': u'/bricks/brick1/testvol_brick3'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.242:999', u'server_identifier': u'10.70.46.240:49153', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-4479-2016/10/03-09:13:04:365388-testvol-client-1-0-0', u'brick_path': u'/bricks/brick2/testvol_brick1'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.218:1012', u'server_identifier': u'10.70.46.242:49152', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-27876-2016/10/03-09:12:53:23019-testvol-client-2-0-0', u'brick_path': u'/bricks/brick1/testvol_brick2'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.218:1002', u'server_identifier': u'10.70.46.239:49153', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-27876-2016/10/03-09:12:53:23019-testvol-client-0-0-0', u'brick_path': u'/bricks/brick2/testvol_brick0'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.218:984', u'server_identifier': u'10.70.46.218:49152', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-27876-2016/10/03-09:12:53:23019-testvol-client-3-0-0', u'brick_path': u'/bricks/brick1/testvol_brick3'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.218:1011', u'server_identifier': u'10.70.46.240:49153', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-27876-2016/10/03-09:12:53:23019-testvol-client-1-0-0', u'brick_path': u'/bricks/brick2/testvol_brick1'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:1009', u'server_identifier': u'10.70.46.240:49153', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-26778-2016/10/03-09:13:05:931879-testvol-client-1-0-0', u'brick_path': u'/bricks/brick2/testvol_brick1'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:1006', u'server_identifier': u'10.70.46.242:49152', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-26778-2016/10/03-09:13:05:931879-testvol-client-2-0-0', u'brick_path': u'/bricks/brick1/testvol_brick2'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.240:1017', u'server_identifier': u'10.70.46.239:49153', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-26778-2016/10/03-09:13:05:931879-testvol-client-0-0-0', u'brick_path': u'/bricks/brick2/testvol_brick0'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.240:1003', u'server_identifier': u'10.70.46.218:49152', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-26778-2016/10/03-09:13:05:931879-testvol-client-3-0-0', u'brick_path': u'/bricks/brick1/testvol_brick3'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558488, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'svc_name': u'glustershd'}, u'event': u'SVC_CONNECTED', u'ts': 1475558488, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:1020', u'server_identifier': u'10.70.46.239:49154', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-disp-client-0-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558488, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:1011', u'server_identifier': u'10.70.46.239:49155', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-disp-client-4-0-0', u'brick_path': u'/bricks/brick1/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558488, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:1007', u'server_identifier': u'10.70.46.239:49153', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-testvol-client-0-0-0', u'brick_path': u'/bricks/brick2/testvol_brick0'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558488, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'subvol': u'testvol-replicate-0'}, u'event': u'AFR_SUBVOL_UP', u'ts': 1475558488, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'peer': u'10.70.46.242', u'volume': u'disp', u'brick': u'/bricks/brick0/disp'}, u'event': u'BRICK_CONNECTED', u'ts': 1475558489, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'peer': u'10.70.46.218', u'volume': u'disp', u'brick': u'/bricks/brick0/disp'}, u'event': u'BRICK_CONNECTED', u'ts': 1475558489, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.239:1005', u'server_identifier': u'10.70.46.242:49152', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-testvol-client-2-0-0', u'brick_path': u'/bricks/brick1/testvol_brick2'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558489, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'subvol': u'testvol-replicate-1'}, u'event': u'AFR_SUBVOL_UP', u'ts': 1475558489, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:1004', u'server_identifier': u'10.70.46.242:49153', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-disp-client-2-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558489, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.239:1003', u'server_identifier': u'10.70.46.218:49153', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-disp-client-3-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558489, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.239:1002', u'server_identifier': u'10.70.46.218:49152', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-testvol-client-3-0-0', u'brick_path': u'/bricks/brick1/testvol_brick3'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558489, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.240:965', u'server_identifier': u'10.70.46.240:49152', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-26797-2016/10/03-09:13:07:956540-ozone-client-1-0-0', u'brick_path': u'/bricks/brick0/ozone'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558489, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'svc_name': u'bitd'}, u'event': u'SVC_CONNECTED', u'ts': 1475558489, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:1001', u'server_identifier': u'10.70.46.239:49152', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30107-2016/10/04-05:21:25:873367-ozone-client-0-0-0', u'brick_path': u'/bricks/brick0/ozone'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558489, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'svc_name': u'scrub'}, u'event': u'SVC_CONNECTED', u'ts': 1475558490, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.240:970', u'server_identifier': u'10.70.46.240:49152', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-26808-2016/10/03-09:13:08:964455-ozone-client-1-0-0', u'brick_path': u'/bricks/brick0/ozone'}, u'event': u'CLIENT_DISCONNECT', u'ts': 1475558490, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.239:998', u'server_identifier': u'10.70.46.239:49152', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30117-2016/10/04-05:21:26:884223-ozone-client-0-0-0', u'brick_path': u'/bricks/brick0/ozone'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558490, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'peer': u'10.70.46.240', u'volume': u'disp', u'brick': u'/bricks/brick1/disp'}, u'event': u'BRICK_CONNECTED', u'ts': 1475558491, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'force': u'0', u'name': u'disp'}, u'event': u'VOLUME_START', u'ts': 1475558491, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:995', u'server_identifier': u'10.70.46.240:49155', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-disp-client-5-0-0', u'brick_path': u'/bricks/brick1/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558491, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.239:996', u'server_identifier': u'10.70.46.240:49154', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-disp-client-1-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558491, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'subvol': u'disp-disperse-0'}, u'event': u'EC_MIN_BRICKS_UP', u'ts': 1475558491, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.239:994', u'server_identifier': u'10.70.46.240:49153', u'client_uid': u'dhcp46-239.lab.eng.blr.redhat.com-30098-2016/10/04-05:21:24:862729-testvol-client-1-0-0', u'brick_path': u'/bricks/brick2/testvol_brick1'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558491, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'svc_name': u'glustershd'}, u'event': u'SVC_CONNECTED', u'ts': 1475558492, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'svc_name': u'glustershd'}, u'event': u'SVC_CONNECTED', u'ts': 1475558492, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.242:1015', u'server_identifier': u'10.70.46.239:49154', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-disp-client-0-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.218:1021', u'server_identifier': u'10.70.46.239:49154', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-disp-client-0-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.242:1011', u'server_identifier': u'10.70.46.240:49154', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-disp-client-1-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.242:1008', u'server_identifier': u'10.70.46.242:49153', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-disp-client-2-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.218:1014', u'server_identifier': u'10.70.46.240:49154', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-disp-client-1-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.242:1006', u'server_identifier': u'10.70.46.218:49153', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-disp-client-3-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.242:1003', u'server_identifier': u'10.70.46.239:49155', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-disp-client-4-0-0', u'brick_path': u'/bricks/brick1/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.218:1009', u'server_identifier': u'10.70.46.242:49153', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-disp-client-2-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.218:1007', u'server_identifier': u'10.70.46.218:49153', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-disp-client-3-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.218:1005', u'server_identifier': u'10.70.46.239:49155', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-disp-client-4-0-0', u'brick_path': u'/bricks/brick1/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.242:995', u'server_identifier': u'10.70.46.240:49155', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-disp-client-5-0-0', u'brick_path': u'/bricks/brick1/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.218:1003', u'server_identifier': u'10.70.46.240:49155', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-disp-client-5-0-0', u'brick_path': u'/bricks/brick1/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'subvol': u'disp-disperse-0'}, u'event': u'EC_MIN_BRICKS_UP', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'subvol': u'disp-disperse-0'}, u'event': u'EC_MIN_BRICKS_UP', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.218:1000', u'server_identifier': u'10.70.46.239:49153', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-testvol-client-0-0-0', u'brick_path': u'/bricks/brick2/testvol_brick0'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.242:993', u'server_identifier': u'10.70.46.239:49153', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-testvol-client-0-0-0', u'brick_path': u'/bricks/brick2/testvol_brick0'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.242:991', u'server_identifier': u'10.70.46.240:49153', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-testvol-client-1-0-0', u'brick_path': u'/bricks/brick2/testvol_brick1'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'subvol': u'testvol-replicate-0'}, u'event': u'AFR_SUBVOL_UP', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.242:990', u'server_identifier': u'10.70.46.242:49152', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-testvol-client-2-0-0', u'brick_path': u'/bricks/brick1/testvol_brick2'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'subvol': u'testvol-replicate-1'}, u'event': u'AFR_SUBVOL_UP', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'subvol': u'testvol-replicate-0'}, u'event': u'AFR_SUBVOL_UP', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.218:998', u'server_identifier': u'10.70.46.240:49153', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-testvol-client-1-0-0', u'brick_path': u'/bricks/brick2/testvol_brick1'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.242:989', u'server_identifier': u'10.70.46.218:49152', u'client_uid': u'dhcp46-242.lab.eng.blr.redhat.com-5231-2016/10/04-05:21:29:28592-testvol-client-3-0-0', u'brick_path': u'/bricks/brick1/testvol_brick3'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.218:994', u'server_identifier': u'10.70.46.242:49152', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-testvol-client-2-0-0', u'brick_path': u'/bricks/brick1/testvol_brick2'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'subvol': u'testvol-replicate-1'}, u'event': u'AFR_SUBVOL_UP', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.218:993', u'server_identifier': u'10.70.46.218:49152', u'client_uid': u'dhcp46-218.lab.eng.blr.redhat.com-28674-2016/10/04-05:21:29:46515-testvol-client-3-0-0', u'brick_path': u'/bricks/brick1/testvol_brick3'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'svc_name': u'glustershd'}, u'event': u'SVC_CONNECTED', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:1016', u'server_identifier': u'10.70.46.239:49154', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-disp-client-0-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.240:1011', u'server_identifier': u'10.70.46.240:49154', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-disp-client-1-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:1007', u'server_identifier': u'10.70.46.242:49153', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-disp-client-2-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'client_identifier': u'10.70.46.240:1004', u'server_identifier': u'10.70.46.218:49153', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-disp-client-3-0-0', u'brick_path': u'/bricks/brick0/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'client_identifier': u'10.70.46.240:1001', u'server_identifier': u'10.70.46.239:49155', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-disp-client-4-0-0', u'brick_path': u'/bricks/brick1/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'client_identifier': u'10.70.46.240:999', u'server_identifier': u'10.70.46.240:49155', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-disp-client-5-0-0', u'brick_path': u'/bricks/brick1/disp'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:997', u'server_identifier': u'10.70.46.239:49153', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-testvol-client-0-0-0', u'brick_path': u'/bricks/brick2/testvol_brick0'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
{u'message': {u'subvol': u'disp-disperse-0'}, u'event': u'EC_MIN_BRICKS_UP', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:993', u'server_identifier': u'10.70.46.242:49152', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-testvol-client-2-0-0', u'brick_path': u'/bricks/brick1/testvol_brick2'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'1e8967ae-51b2-4c27-907e-a22a83107fd0'}
{u'message': {u'subvol': u'testvol-replicate-0'}, u'event': u'AFR_SUBVOL_UP', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:994', u'server_identifier': u'10.70.46.240:49153', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-testvol-client-1-0-0', u'brick_path': u'/bricks/brick2/testvol_brick1'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:992', u'server_identifier': u'10.70.46.218:49152', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27637-2016/10/04-05:21:29:451957-testvol-client-3-0-0', u'brick_path': u'/bricks/brick1/testvol_brick3'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558493, u'nodeid': u'0dea52e0-8c32-4616-8ef8-16db16120eaa'}
{u'message': {u'subvol': u'testvol-replicate-1'}, u'event': u'AFR_SUBVOL_UP', u'ts': 1475558493, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'svc_name': u'bitd'}, u'event': u'SVC_CONNECTED', u'ts': 1475558494, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:991', u'server_identifier': u'10.70.46.240:49152', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27646-2016/10/04-05:21:30:464132-ozone-client-1-0-0', u'brick_path': u'/bricks/brick0/ozone'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558494, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'client_identifier': u'10.70.46.240:989', u'server_identifier': u'10.70.46.240:49152', u'client_uid': u'dhcp46-240.lab.eng.blr.redhat.com-27656-2016/10/04-05:21:31:474215-ozone-client-1-0-0', u'brick_path': u'/bricks/brick0/ozone'}, u'event': u'CLIENT_CONNECT', u'ts': 1475558494, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}
{u'message': {u'svc_name': u'scrub'}, u'event': u'SVC_CONNECTED', u'ts': 1475558495, u'nodeid': u'72c4f894-61f7-433e-a546-4ad2d7f0a176'}





[root@dhcp46-159 events_seen]# cat volume_start | grep CLIENT_CONNECT | wc -l
44
[root@dhcp46-159 events_seen]# 
[root@dhcp46-159 events_seen]# cat volume_start | grep CLIENT_CONNECT | grep disp | wc -l
24
[root@dhcp46-159 events_seen]#
[root@dhcp46-159 events_seen]# cat volume_start | grep CLIENT_CONNECT | grep -v disp | wc -l
20
[root@dhcp46-159 events_seen]# 
[root@dhcp46-159 events_seen]# cat volume_start | grep CLIENT_DISCONNECT | wc -l
20
[root@dhcp46-159 events_seen]#
Comment 2 Atin Mukherjee 2016-10-12 07:08:18 UTC 
Prasanna - can you check this? the mentioned set of events were introduced by http://review.gluster.org/15294
Comment 3 Mohammed Rafi KC 2016-10-18 07:02:38 UTC 
RCA:

The events for already started bricks are generated from internal services which are per node basis, like self heal daemon,nfs.

In case of shd, basically when you start a volume, to load the volfile for that volume , you need to stop the shd and start it with new volfile. When you bring down the shd deamon, it triggers a disconnect for already connected volume meaning, for volumes which is already started and enabled self heal.
Those disconnects triggers the protocol events.

And when shd restarts with new volfile, we will get a connect event as well
Comment 5 Mohammed Rafi KC 2016-10-25 07:15:20 UTC 
As per the RCA, the events are generated from internal Gluster daemons, and for a glusterfsd process, they are clients and the events are genuine. It is the responsibility of consumers of the events to decide whether to discard or act on the event. Unfortunately we don't have the way to distinguish the correctness, once we implement protocol/client events we may able to do this.
Comment 6 Mohammed Rafi KC 2016-10-25 07:21:01 UTC 
Closing this bug as won't fix.
Comment 7 Sweta Anandpara 2016-10-28 05:43:26 UTC 
The resolution, or rather, the impact of this bug does not look clean to me, that we can live with it. 

In my limited node cluster (4), with a significantly small number of volumes created (3), which have distinctly small volume sizes/configuration (2*2, 1*4+2, 2*1), I see _44_ CLIENT_CONNECTS and CLIENT_DISCONNECTS with just the default services. This number will just increase if I enable other user-configurable services like bitrot/nfs/snapshot. And needless to say, this number is going to multiply n times in an actual customer environment.

It is becoming quite cumbersome for me to even test out the events in my small cluster with 3 volumes, when there is such a huge traffic of /unrelated/ events. The fallback that I have been exercising right now is that I delete the other volumes and start out testing my events in a single-volume-cluster. I do not see this kind of testing to scale up as the release progresses. And it definitely will not suffice testing of Eventing as a feature. 

I do not agree with this line of Comment 5 - "responsibility of consumers of events to decide whether to discard or act on an event". We are relaxing too much on the consumer to take this call and washing our hands off it. I am using a webhook as a consumer and I am feeling taxed to monitor only the events which matter and discard the rest. And I am little confident if I am doing everything right. What confidence will an external user have?!

I am affirmative wrt Comment 3 that the internal-gluster-daemons are clients for glusterfsd, and the events seen are genuine. There is no denying that. The question is: Are the internal events _really_ important?
   If no, we can do away with them.
   If yes, can we classify/group such events based on priority, may be set the log_level of such events to 'debug' rather than 'info'. At least, setting the appropriate log_level should act as a filtering mechanism.
Comment 8 Sweta Anandpara 2016-10-28 06:22:28 UTC 
Amend in the last line of above comment. 'log_level' caters to only logging of events. What I meant was setting the priority/importance of events to high-medium-low or classifying the events to info-error-debug-warning, thereby displaying or responding only to those events which match the configured setting. 

Atin and team, thoughts?
Comment 14 Amar Tumballi 2018-11-06 08:25:02 UTC 
In last 2years we don't have any progress made on this bug, and once had closed as WONTFIX. I would rather close this as UPSTREAM for now, to indicate we will try to fix it if we find the issues with gluster-prometheus in GCS integration.
Note You need to log in before you can comment on or make changes to this bug.