Bug 1563648 - Marshal / Un-marshal objects while saving / reading to / from etcd
Summary: Marshal / Un-marshal objects while saving / reading to / from etcd
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: web-admin-tendrl-commons
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: RHGS 3.4.0
Assignee: Shubhendu Tripathi
QA Contact: Martin Bukatovic
URL:
Whiteboard:
Depends On:
Blocks: 1503137
TreeView+ depends on / blocked
 
Reported: 2018-04-04 11:52 UTC by Shubhendu Tripathi
Modified: 2018-09-04 07:04 UTC (History)
8 users (show)

Fixed In Version: tendrl-commons-1.6.3-1.el7rhgs
Doc Type: Bug Fix
Doc Text:
Cause: At infra level earlier while persisting the object details in central store (etcd) each field used to be written separately causing so many REST calls to etcd. Consequence: This way of persisting the objects to etcd used to cause lot of REST calls resulting in lot of network calls. Also this used to end up in race conditions when other threads were trying to update an object whereas the saving thread still writing the data to etcd. Fix: As as performance improvement and getting away from race condition, now the whole object gets serialized into a single JSON and is written to etcd. While reading object state from etcd, this single JSON is read and then object is weaved back using the details. Result: As a result each object write / read to / from etcd now results in single REST call. It also avoids the critical race condition which were causing lot of issues in various flows in system.
Clone Of:
Environment:
Last Closed: 2018-09-04 07:03:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github Tendrl commons issues 893 None None None 2018-04-05 08:35:26 UTC
Red Hat Bugzilla 1538248 None None None Never
Red Hat Product Errata RHSA-2018:2616 None None None 2018-09-04 07:04:50 UTC

Internal Links: 1538248

Description Shubhendu Tripathi 2018-04-04 11:52:05 UTC
Description of problem:
Currently each field under an object gets written individually to etcd which makes REST call for each of them. Same is the case while reading an object from etcd.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
The write and read to / from etcd should marshal and un-marshal object data.

Additional info:

Comment 2 Martin Bukatovic 2018-04-05 16:07:49 UTC
How should we measure impact of the change described here? This change doesn't
have functional or directly visible impact on end user, so we can't verify that
this change has expected benefits without specifying how to measure it.

Any suggestions?

Comment 3 Shubhendu Tripathi 2018-04-05 16:11:13 UTC
This ideally involves verification across two versions, one without this fix and one with this fix. In new version we should be able to see lesser no of REST calls for writing the object data.

Comment 4 Martin Bukatovic 2018-04-06 16:41:24 UTC
It is sufficient to measure number of REST calls by logging new TCP connections 
opened with target port which etcd is listening on via custom firewall rule?
I'm asking as it's likely that the tcp connection could be reused for multiple
queries, correct? If the firewall level logging is not sufficient, we would need
to use tcpdump and count the calls directly there ...

How big improvement do we expect to see? Could I measure this in other way as
well (eg. utilization of particular resource goes down by some percent)?

Comment 13 Martin Bukatovic 2018-09-03 14:08:46 UTC
Fixing wrong docs contact.

Comment 17 errata-xmlrpc 2018-09-04 07:03:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2616


Note You need to log in before you can comment on or make changes to this bug.