Created attachment 586932 [details]
patch to change boolean search field names

As a quick test, I just tried changing the field names in /usr/bin/bugzilla and dropping the and-count and or-count values (so this only works for one search term); see the attached patch.

However, even using the proper f1, o1, and v1 field names, I'm still getting an error:

$ bugzilla query --bug_status=NEW,ASSIGNED --product=Fedora \
                 --boolean_query='keywords-substring-Patch'

Server error: <Fault 53: 'v1 is not a valid parameter for the Bugzilla::Bug::match function.'>

I'm not sure why it's complaining that v1 is not valid when it clearly uses v1=Patch as shown the URL above.

I'm wondering if the "v1 is not a valid parameter" error is a bug in the xmlrpc interface, so I also opened bug 825370

Created attachment 586942 [details]
patched python-bugzilla module

If it helps, attached is my patched version of the python-bugzilla module

Ahhh, apparently you must add the query_type=advanced parameter to your search terms.  This was not necessary with Bugzilla 3.6 (maybe it was implied), but it is required now.
 
 query = dict()
+query['query_format'] = 'advanced'
 query['bug_status'] = [ 'NEW', 'ASSIGNED' ]

This fixes the "v1 is not a valid parameter" error.

The search is really slow, though, compared to BZ 3.6...

Created attachment 586947 [details]
patch to change boolean search field names

Updated patch to also add the query_format parameter to the boolean search:

  q['query_format'] = 'advanced'

The patch still needs work:
  1. to handle multiple search terms with AND, OR, and (PARANTHESES GROUPS)
  2. boolean charts still have the old search terms like 'field%i-%i-%i'

Hi Jeff,

Yes the query_format thing is now necessary, not sure why it changed but it did.  The slowness is because of the way python-bugzilla works when fetching data on the backend when not all the data is provided by the query result.  For some reason, the response from the server for a list of valid data fields takes up to 50 seconds on bugzilla4 as opposed to a few seconds on bugzilla 3.6.

The change I have been using to compensate is putting all the required data inside an  q['include_fields']= [...] array.  This reduces the lookup delays.  Of course it isn't ideal, but it does work around the problem for now.

Cheers,
Don

Created attachment 587482 [details]
example script

The include_fields array doesn't seem to really help performance.  In the attached example, I'm requesting only 3 fields: bug_id, component, and short_desc.

It apparently only takes 3 or 4 seconds to run the query, but then it takes 52 seconds to print the first entry!  And each subsequent line of output takes another 1 or 2 seconds each.

[jbastian@localhost tmp]$ ./bz-query.py 
...
[13:30:43] Making query
[13:30:46] Printing report
[13:31:38] 72487 initscripts RFE: Add support for disabling the duplicate IP check
[13:31:39] 112727 rpm Filenames in manifest can not contain '#'
[13:31:41] 128804 rpm Implementation of per-file dependencies
[13:31:42] 129820 initscripts initscripts maclist patch
...



If I enter the debugger right after making the query (uncomment line 46), I can see that the results are _not_ populated with the requested include_fields:

(Pdb) dir(results[0])
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '__weakref__', '_dowhiteboard', 'addcc', 'addcomment', 'addtag', 'appendwhiteboard', 'autorefresh', 'bug_id', 'bugzilla', 'close', 'deletecc', 'deltag', 'get_flag_status', 'get_flag_type', 'get_flags', 'gettags', 'getwhiteboard', 'id', 'prependwhiteboard', 'refresh', 'reload', 'setassignee', 'setstatus', 'setwhiteboard', 'updateflags', 'url']


I see bug_id, but where are component and short_desc?

When I examine component, it fills in the missing data:

(Pdb) results[0].component
['initscripts']


But it also gets a *lot* more than I originally requested!

(Pdb) dir(results[0])
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '__weakref__', '_dowhiteboard', 'actual_time', 'addcc', 'addcomment', 'addtag', 'alias', 'appendwhiteboard', 'assigned_to', 'attach_data.thedata', 'attachments', 'attachments.description', 'attachments.filename', 'attachments.isobsolete', 'attachments.ispatch', 'attachments.isprivate', 'attachments.mimetype', 'attachments.submitter', 'attachstatusdefs.name', 'autorefresh', 'blocked', 'bug_file_loc', 'bug_group', 'bug_id', 'bug_severity', 'bug_status', 'bugzilla', 'build_id', 'cc', 'cclist_accessible', 'cf_build_id', 'cf_clone_of', 'cf_conditional_nak', 'cf_crm', 'cf_cust_facing', 'cf_devel_whiteboard', 'cf_documentation_action', 'cf_environment', 'cf_extra_component', 'cf_extra_version', 'cf_fixed_in', 'cf_internal_whiteboard', 'cf_mount_type', 'cf_partner', 'cf_pm_score', 'cf_qa_whiteboard', 'cf_qe_conditional_nak', 'cf_regression_status', 'cf_release_notes', 'cf_story_points', 'cf_type', 'cf_verified', 'cf_verified_branch', 'class', 'classification', 'clone_of', 'close', 'commenter', 'component', 'conditional_nak', 'content', 'creation_time', 'creation_ts', 'cust_facing', 'days_elapsed', 'deadline', 'deletecc', 'delta_ts', 'deltag', 'dependson', 'devel_whiteboard', 'estimated_time', 'everconfirmed', 'ext_bz_bug_map.ext_bz_bug_id', 'external_bugs', 'external_bugzilla.description', 'external_bugzilla.url', 'external_choices', 'fixed_in', 'flag_types', 'flagtypes.name', 'get_flag_status', 'get_flag_type', 'get_flags', 'gettags', 'getwhiteboard', 'groups', 'id', 'internal_whiteboard', 'keywords', 'last_change_time', 'lastdiffed', 'longdesc', 'longdescs', 'longdescs.count', 'longdescs.isprivate', 'op_sys', 'owner_idle_time', 'partner', 'percentage_complete', 'pm_score', 'prependwhiteboard', 'priority', 'product', 'product_id', 'qa_contact', 'qa_whiteboard', 'qe_conditional_nak', 'refresh', 'release_notes', 'reload', 'remaining_time', 'rep_platform', 'reporter', 'reporter_accessible', 'reporter_id', 'requestees.login_name', 'resolution', 'see_also', 'setassignee', 'setstatus', 'setters.login_name', 'setwhiteboard', 'short_desc', 'status_whiteboard', 'summary', 'tag', 'target_milestone', 'target_release', 'updateflags', 'url', 'vcs_commits', 'verified', 'version', 'view', 'votes', 'work_time']


No wonder it takes so long to run!

Ok, I found a bug in my unofficial build which was resetting the include_fields parameter.  With that fixed, it's getting the bug_id and component very quickly, but not short_desc.

[jbastian@localhost tmp]$ ./bz-query.py 
...
[15:20:35] Logged into bugzilla
[15:20:35] Formulating query
[15:20:35] Making query
[15:20:38] Printing report
[15:20:38] 72487 initscripts
[15:20:38] 112727 rpm
[15:20:38] 128804 rpm
[15:20:38] 129820 initscripts
...

It now generates the whole report in just a few seconds.

I'll keep poking at it and see if I can get short_desc to work also.

Hi Jeff,

Can you try using this tree:

git://git.engineering.redhat.com/users/arozansk/python-bugzilla.git

It's a little different than Will's but has some backend changes to handle various conversions (bug_id -> id, bug_status -> short_desc -> summary...).  You will still need your other patch for 'advance' query format.

Cheers,
Don

Thanks Don!  The patches in Aristeu's tree help a lot!

The short_desc -> summary change was the source of my speed problems.  My test script runs in just a few seconds now that I've changed my column_list (or include_fields) to use summary instead.


So, now this BZ is just back to the original problem of
   field0-0-0   =>   f1
   type0-0-0    =>   o1
   value0-0-0   =>   v1
in the bin/bugzilla script

Jeff

Hi Jeff,

I put up an rpm with some more changes here:

http://people.redhat.com/dzickus/python-bugzilla/

You are the second person to tell me about the field0-0-0 => f1 changes, but I can't seem to duplicate the problem on my end for some reason.  I don't have your changes but can run the sample query you had in comment0 with no problem.

I am trying to find the change that fixed that but can't.

Can you take the above rpm or the below git tree and run your failed script again?

git://git.engineering.redhat.com/users/dzickus/python-bugzilla

I am willing to take your change and pass it along to upstream, but I am trying to figure out if it could be translated by the backend instead of the front end.  However, in order to do that, I was hoping to see the failure. :-(

Cheers,
Don

Hmm, it's working for me now too.

I even tried with an older version of the module -- python-bugzilla-0.6.0-1.el6.noarch -- on RHEL6 and it worked fine.

Maybe something changed on the bugzilla side so it no longer rejects field0-0-0 syntax?

It also works fine with the python-bugzilla-0.6.3-0.dz.el6.noarch.rpm package from your people page.

I can't explain why it started working again...

*** Bug 832493 has been marked as a duplicate of this bug. ***

Just updated to 0.7.0, and I now get this on fedora-infrastructure/scripts/process-git-requests.

Hi,

with python-bugzilla-0.7.0-1.el6.noarch/python-bugzilla-0.7.0-1.fc17.noarch I'm getting 
xmlrpclib.Fault: <Fault 53: 'type0-1-0 is not a valid parameter for the Bugzilla::Bug::match function.'>

with python-bugzilla-0.6.3-0.dz.el6.noarch the error changes to 
xmlrpclib.Fault: <Fault 53: 'type0-1-0 is not a valid parameter for the Bugzilla::Bug::match function.'>

The full traceback is:
Traceback (most recent call last):
  File "./new-components", line 47, in <module>
    buglist = bugzilla.query({'product': 'Red Hat Enterprise Linux %s'% major, 'type0-0-0':'substring','field0-0-0':'cf_internal_whiteboard','value0-0-0':'NwCmp','type0-1-0':'substring','field0-1-0':'flagtypes.name','value0-1-0':'rhel-%s.0+'%release})
  File "/usr/lib/python2.6/site-packages/bugzilla/base.py", line 544, in query
    r = self._query(query)
  File "/usr/lib/python2.6/site-packages/bugzilla/rhbugzilla.py", line 578, in _query
    ret = self._proxy.Bug.search(query)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.6/site-packages/bugzilla/base.py", line 1020, in request_with_cookies
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 53: 'type0-1-0 is not a valid parameter for the Bugzilla::Bug::match function.'>


python-bugzilla-0.6.0-1.el6.noarch is ok (has other bugs but not this one :))

I failed at copy-pasting. 

Should be:

with python-bugzilla-0.6.3-0.dz.el6.noarch the error changes to 
xmlrpclib.Fault: <Fault 53: 'value0-0-0 is not a valid parameter for the Bugzilla::Bug::match function.'>

Hi Ales,

Your above output shows:

 File "./new-components", line 47, in <module>
    buglist = bugzilla.query({'product': 'Red Hat Enterprise Linux %s'% major, 'type0-0-0':'substring','field0-0-0':'cf_internal_whiteboard','value0-0-0':'NwCmp','type0-1-0':'substring','field0-1-0':'flagtypes.name','value0-1-0':'rhel-%s.0+'%release})
  File "/usr/lib/python2.6/site-packages/bugzilla/base.py", line 544, in query

It seems like new-components is building its own bugzilla.query.

If you are going to build your own queries you now need to add:

'query_format' : 'advanced'

to be able to handle those field types.

Cheers,
Don

Thank you Don, that did he trick (with python-bugzilla-0.7.0-1.el6.noarch.rpm)

I think everything here is fixed. If anyone is still hitting an issue with current python-bugzilla, please paste a command line reproducer or a code snippet that reproduces. Closing until then.