Bug 824611 - Bug.fields is not in sync with return values
Bug.fields is not in sync with return values
Status: CLOSED WONTFIX
Product: Bugzilla
Classification: Community
Component: WebService (Show other bugs)
4.2
Unspecified Unspecified
unspecified Severity unspecified (vote)
: ---
: ---
Assigned To: Simon Green
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-23 16:05 EDT by Don Zickus
Modified: 2014-10-12 18:48 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-07 20:10:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Don Zickus 2012-05-23 16:05:12 EDT
Description of problem:
So python-bugzilla tries to be clever.  After parsing bz data from a Bug.search or a Bug.get, it populates a local cache.  If a program tries to access a field that doesn't exist, instead of throwing an error, it thinks 'well.. maybe the field exists but was restricted on the last call'.  As a result the python-bugzilla programs does a Bug.fields call to determine if the field exists or not.

If the data returned shows the field exists but the cache is not populated, python-bugzilla does a Bug.get behind the scenes to populate it.  Otherwise it returns an error, invalid field.

Unfortunately, Bug.fields is not matching 1:1 with the return values and as a result causes python-bugzilla to display None for values.

The values that I have hit so far are:

fields        get/search returns
bug_status -> status
short_desc -> summary
blocked    -> blocks
dependson  -> depends_on
conditional_nak -> cf_conditional_nak

Yes I understand Bug.fields is marked UNSTABLE, but within the current release I expected it to be in sync with the data it returns.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Kevin Baker 2012-05-23 20:44:25 EDT
(In reply to comment #0)
> Description of problem:
> So python-bugzilla tries to be clever.  After parsing bz data from a
> Bug.search or a Bug.get, it populates a local cache.  If a program tries to
> access a field that doesn't exist, instead of throwing an error, it thinks
> 'well.. maybe the field exists but was restricted on the last call'.  As a
> result the python-bugzilla programs does a Bug.fields call to determine if
> the field exists or not.

yikes! Bug.field is a HUGE data structure to pull down willy-nilly. I pulled it down and dumped as a perl struct and it's 22,502,037 bytes! If all python-bugzilla apps are doing that then maybe that's why we're seeing such performance degredation.
Comment 2 Kevin Fenzi 2012-05-23 23:09:48 EDT
Which version of python-bugzilla ?
Comment 3 Don Zickus 2012-05-24 09:15:41 EDT
@comment1:

My output doesn't show the Bug.field as being that huge.  You see 22MB, I see more closer to 22KB.  Still large but it is a one time thing and only if a data field an app is requesting was not available.  I have optimized my scripts to use 'include_fields' for everything I need (used to be column_list but that changed too).

@comment2:

This should be noticable on most versions of python-bugzilla.  I am using 0.6.2 (but hacked up to deal with the new changes).

You shouldn't need python-bugzilla to see this problem.  I would think perl/python module to grab Bug.field and then Bug.get(456789).  See if bug_status is in Bug.field output and Bug.get output.  My tests show it is _not_ in Bug.get output.

Cheers,
Don
Comment 4 Kevin Baker 2012-05-24 13:00:44 EDT
(In reply to comment #3)
> @comment1:
> 
> My output doesn't show the Bug.field as being that huge.  You see 22MB, I
> see more closer to 22KB.  Still large but it is a one time thing and only if

I was simply calling Bug.field and getting everything. I'm guessing python-bugzilla is being more sensible and requesting one field. Without that it's huge   from perl:

Size of XMLRPC Call Object            802,460,565 bytes
Size of returned fields object         34,833,455 bytes

> a data field an app is requesting was not available.  I have optimized my
> scripts to use 'include_fields' for everything I need (used to be
> column_list but that changed too).

FYI - column_list is for the legacy API that Red Hat wrote. Now that the upstream is building out the WebServices functions we need to move over to them. Maintaining both is difficult, error prone and probably a security hole.

> @comment2:
> 
> This should be noticable on most versions of python-bugzilla.  I am using
> 0.6.2 (but hacked up to deal with the new changes).
> 
> You shouldn't need python-bugzilla to see this problem.  I would think
> perl/python module to grab Bug.field and then Bug.get(456789).  See if
> bug_status is in Bug.field output and Bug.get output.  My tests show it is
> _not_ in Bug.get output.

I'll leave this to the developers to investigate - sounds like we'll need to file a bug on the upstream bugzilla.
Comment 5 Will Woods 2012-05-24 14:47:27 EDT
All python-bugzilla wants is a list of valid bug field names for the bugzilla instance it's talking to, so it knows what attributes the Bug object should have.

This lets us avoid doing a network roundtrip *every* time an unknown field is accessed. Instead, we cache the field list[1] and raise an error if an unknown field is not in the list.

(I think this is a pretty reasonable request, considering that a Bugzilla instance can have a number of custom fields...)

The legacy API provided bugzilla.getBugFields() for this purpose; Bug.fields() was the closest thing I could find in the upstream API.

bugzilla.getBugFields() takes ~1s to return, and returns 91 fields.

Currently, Bug.fields({'include_fields':['name']}) takes ~51s to return(!), and returns 84 fields.

Probably the thing to do is just keep using bugzilla.getBugFields() until upstream provides a reasonable interface for this.

[1] If the CLI was clever, it would cache the returned list somewhere on-disk and only refresh it once every hour/day/week..
Comment 6 Simon Green 2012-06-07 20:10:30 EDT
(In reply to comment #0)
> The values that I have hit so far are:
> 
> fields        get/search returns
> bug_status -> status
> short_desc -> summary
> blocked    -> blocks
> dependson  -> depends_on
> conditional_nak -> cf_conditional_nak
> 
> Yes I understand Bug.fields is marked UNSTABLE, but within the current
> release I expected it to be in sync with the data it returns.

We are maintaining compatibility with upstream as much as possible with RPC calls, and this is one issue that comes from upstream. The only change we have made to the fields code is to prevent the cf_extra_component cf_extra_version and cf_last_closed fields from being reported.

You are free to raise this issue with upstream Bugzilla.

Note You need to log in before you can comment on or make changes to this bug.