Bug 1711296

Summary:	Optimize glusterd code to copy dictionary in handshake code path
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Mohit Agrawal <moagrawa>
Component:	glusterd	Assignee:	Mohit Agrawal <moagrawa>
Status:	CLOSED ERRATA	QA Contact:	Bala Konda Reddy M <bmekala>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	rhgs-3.5	CC:	amukherj, rhs-bugs, sheggodu, storage-qa-internal, vbellur, vdas
Target Milestone:	---	Keywords:	Improvement, Performance
Target Release:	RHGS 3.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-6.0-4	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1711297 (view as bug list)		Environment:
Last Closed:	2019-10-30 12:21:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1696809, 1711297

Description Mohit Agrawal 2019-05-17 13:01:36 UTC

Description of problem:
glusterd is not able to handle friend request of peer node while brick_multiplex is enabled and no. of configure volumes are 2000.glusterd has some bottlenecks in the functions glusterd_add_volumes_to_export_dict and glusterd_compare_friend_data at the time of copying dictionary and peer node throwing bailing out RPC packet because it has not received RPC response.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.Setup 1x3 2000 volumes(test1...test2000)
2.Enable brick_multiplex
3.Down glusterd one node  
4. Update performance.readdir-ahead for volumes periodically like test1, test41, test81, test121...test1961
5. Start glusterd
6. Wait to finish handshake of glusterd and check peer status is Connected
7. Check the value of all volumes those were updated before start glusterd
   All values should be synced 
Actual results:
  Value for all volumes are not synced

Expected results:
  All values should be synced.

Additional info:

Comment 2 Mohit Agrawal 2019-05-17 13:03:22 UTC

Below are the function trace where it is showing glusterd has bottleneck

1)

  0x00007fdbb856a02c in dict_lookup_common () from /lib64/libglusterfs.so.0
#1  0x00007fdbb856bc72 in dict_set_lk () from /lib64/libglusterfs.so.0
#2  0x00007fdbb856be31 in dict_setn () from /lib64/libglusterfs.so.0
#3  0x00007fdbb856dbf7 in dict_foreach_match () from /lib64/libglusterfs.so.0
#4  0x00007fdbb856dda8 in dict_foreach () from /lib64/libglusterfs.so.0
#5  0x00007fdbb856dfb5 in dict_copy_with_ref () from /lib64/libglusterfs.so.0
#6  0x00007fdbac6c05d2 in glusterd_add_volumes_to_export_dict () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
#7  0x00007fdbac6e65d0 in glusterd_rpc_friend_add () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
#8  0x00007fdbac69fb19 in glusterd_ac_friend_add () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
#9  0x00007fdbac6a201e in glusterd_friend_sm () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
#10 0x00007fdbac6fe832 in __glusterd_mgmt_hndsk_version_ack_cbk () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
#11 0x00007fdbac6eb13a in glusterd_big_locked_cbk () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
#12 0x00007fdbb8320051 in rpc_clnt_handle_reply () from /lib64/libgfrpc.so.0
#13 0x00007fdbb83203b7 in rpc_clnt_notify () from /lib64/libgfrpc.so.0
#14 0x00007fdbb831ca53 in rpc_transport_notify () from /lib64/libgfrpc.so.0
#15 0x00007fdbab8b9ad5 in socket_event_handler () from /usr/lib64/glusterfs/6.0/rpc-transport/socket.so
#16 0x00007fdbb85dd176 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#17 0x00007fdbb73b5dd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fdbb6c7dead in clone () from /lib64/libc.so.6

2)

  #0  0x00007efdb9b0602c in dict_lookup_common () from /lib64/libglusterfs.so.0
  #1  0x00007efdb9b07c72 in dict_set_lk () from /lib64/libglusterfs.so.0
  #2  0x00007efdb9b07e31 in dict_setn () from /lib64/libglusterfs.so.0
  #3  0x00007efdb9b09bf7 in dict_foreach_match () from /lib64/libglusterfs.so.0
  #4  0x00007efdb9b09da8 in dict_foreach () from /lib64/libglusterfs.so.0
  #5  0x00007efdb9b09fb5 in dict_copy_with_ref () from /lib64/libglusterfs.so.0
  #6  0x00007efdadc792b0 in glusterd_compare_friend_data () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
  #7  0x00007efdadc3bafb in glusterd_ac_handle_friend_add_req () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
  #8  0x00007efdadc3d01e in glusterd_friend_sm () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
  #9  0x00007efdadc34dd4 in __glusterd_handle_incoming_friend_req () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
  #10 0x00007efdadc2ec5e in glusterd_big_locked_handler () from /usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so
  #11 0x00007efdb98b6635 in rpcsvc_handle_rpc_call () from /lib64/libgfrpc.so.0
  #12 0x00007efdb98b69ab in rpcsvc_notify () from /lib64/libgfrpc.so.0
  #13 0x00007efdb98b8a53 in rpc_transport_notify () from /lib64/libgfrpc.so.0
  #14 0x00007efdace54ad5 in socket_event_handler () from /usr/lib64/glusterfs/6.0/rpc-transport/socket.so
  #15 0x00007efdb9b79176 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
  #16 0x00007efdb8951dd5 in start_thread () from /lib64/libpthread.so.0
  #17 0x00007efdb8219ead in clone () from /lib64/libc.so.6

Comment 12 errata-xmlrpc 2019-10-30 12:21:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249