Bug 1025106 - [XMLRPC][Serialization] Improve model's serialization
[XMLRPC][Serialization] Improve model's serialization
Status: VERIFIED
Product: TCMS
Classification: Other
Component: XMLRPC (Show other bugs)
Devel
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 3.8.6
Assigned To: Yang Ren
: Improvement
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-30 23:01 EDT by cqi
Modified: 2015-12-01 00:45 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description cqi 2013-10-30 23:01:00 EDT
Model's serialization is implemented in module ``tcms.core.utils.xmlrpc``. An object's all fields are serialized into a dict object recursively, which means all field with type ForeignKey and ManyToManyKey will also be serialized as well by getting related object. However, only the name or title, whatever a description text, of the related object is required besides ID.

Current implementation has two major problems introduced by the ForeignKey field.

- Each time to get related objects following the relationship represented by a ForeignKey field, a SQL query happens, and a complete object with all fields will be generated, although only the two field ID and name or title are required. Thus, this can produce amount of SQL queries if a queryset is being serialized.

- Further, too much unnecessary data is loaded from MySQL and be transferred over network from MySQL to TCMS.

From above two point, a proposal solution is a new serialization method that operates upon a explicit fields list with names rather than the field objects of a Model object. For example, the new method would operate on a QuerySet,

``TestRun.objects.filter(**query).values('run_id', 'summary', 'start_date', 'plan', 'plan__name', 'tag__pk' ...)``

the fields list here can be generated programmatically from a Model. 

This solution can covers both above problems. In this way, all necessary data is retrieved via one SQL query. Again, to serialize a TestRun object, only the id and name of related TestPlan object are required.

It's worth to consider to make an improvement to this core method of XMLRPC API. Model's change would bring less affect to it only.
Comment 1 Chaobin Tang 2014-01-16 05:33:43 EST
Yeah, this sounds doable.

So, instead of getting each field value by traversing all the fields of a model, a better way would be to calculate a list of Django LookUp field names and give it to the .values() method.

Okay, I'd like to see the how much of a difference it makes for an existing XMLRPC API.

Also, I don't suspect this introduces inconsistency into XMLRPC APIs right?
Comment 2 cqi 2014-01-16 10:23:13 EST
(In reply to Chaobin Tang from comment #1)
> Yeah, this sounds doable.
> 
> So, instead of getting each field value by traversing all the fields of a
> model, a better way would be to calculate a list of Django LookUp field
> names and give it to the .values() method.
> 
> Okay, I'd like to see the how much of a difference it makes for an existing
> XMLRPC API.
> 
> Also, I don't suspect this introduces inconsistency into XMLRPC APIs right?

Yes. Rewrite must ensure the original behavior not changed.

Cache mechanism could be introduced to cache each model's fields' names. Because, they are immutable during the runtime once TCMS begins to run.

Note You need to log in before you can comment on or make changes to this bug.