Bug 1563648

Summary: Marshal / Un-marshal objects while saving / reading to / from etcd
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Shubhendu Tripathi <shtripat>
Component: web-admin-tendrl-commonsAssignee: Shubhendu Tripathi <shtripat>
Status: CLOSED ERRATA QA Contact: Martin Bukatovic <mbukatov>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: jefbrown, mbukatov, nthomas, rghatvis, rhs-bugs, shberry, shtripat, smukherj
Target Milestone: ---   
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tendrl-commons-1.6.3-1.el7rhgs Doc Type: Bug Fix
Doc Text:
Cause: At infra level earlier while persisting the object details in central store (etcd) each field used to be written separately causing so many REST calls to etcd. Consequence: This way of persisting the objects to etcd used to cause lot of REST calls resulting in lot of network calls. Also this used to end up in race conditions when other threads were trying to update an object whereas the saving thread still writing the data to etcd. Fix: As as performance improvement and getting away from race condition, now the whole object gets serialized into a single JSON and is written to etcd. While reading object state from etcd, this single JSON is read and then object is weaved back using the details. Result: As a result each object write / read to / from etcd now results in single REST call. It also avoids the critical race condition which were causing lot of issues in various flows in system.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 07:03:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1503137    

Description Shubhendu Tripathi 2018-04-04 11:52:05 UTC
Description of problem:
Currently each field under an object gets written individually to etcd which makes REST call for each of them. Same is the case while reading an object from etcd.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
The write and read to / from etcd should marshal and un-marshal object data.

Additional info:

Comment 2 Martin Bukatovic 2018-04-05 16:07:49 UTC
How should we measure impact of the change described here? This change doesn't
have functional or directly visible impact on end user, so we can't verify that
this change has expected benefits without specifying how to measure it.

Any suggestions?

Comment 3 Shubhendu Tripathi 2018-04-05 16:11:13 UTC
This ideally involves verification across two versions, one without this fix and one with this fix. In new version we should be able to see lesser no of REST calls for writing the object data.

Comment 4 Martin Bukatovic 2018-04-06 16:41:24 UTC
It is sufficient to measure number of REST calls by logging new TCP connections 
opened with target port which etcd is listening on via custom firewall rule?
I'm asking as it's likely that the tcp connection could be reused for multiple
queries, correct? If the firewall level logging is not sufficient, we would need
to use tcpdump and count the calls directly there ...

How big improvement do we expect to see? Could I measure this in other way as
well (eg. utilization of particular resource goes down by some percent)?

Comment 13 Martin Bukatovic 2018-09-03 14:08:46 UTC
Fixing wrong docs contact.

Comment 17 errata-xmlrpc 2018-09-04 07:03:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2616