Bug 825358 - value0-0-0 is not a valid parameter for the Bugzilla::Bug::match function
value0-0-0 is not a valid parameter for the Bugzilla::Bug::match function
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: python-bugzilla (Show other bugs)
16
All Linux
unspecified Severity medium
: ---
: ---
Assigned To: Will Woods
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-25 15:19 EDT by Jeff Bastian
Modified: 2013-01-10 01:48 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-27 21:50:18 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch to change boolean search field names (1.53 KB, patch)
2012-05-25 15:32 EDT, Jeff Bastian
no flags Details | Diff
patched python-bugzilla module (122.57 KB, application/x-gzip)
2012-05-25 16:14 EDT, Jeff Bastian
no flags Details
patch to change boolean search field names (2.01 KB, patch)
2012-05-25 16:28 EDT, Jeff Bastian
no flags Details | Diff
example script (1.24 KB, text/plain)
2012-05-29 14:42 EDT, Jeff Bastian
no flags Details

  None (edit)
Description Jeff Bastian 2012-05-25 15:19:26 EDT
Description of problem:
The /usr/bin/bugzilla command gets an error when trying to do a Boolean search.  For example:

$ bugzilla query --bug_status=NEW,ASSIGNED --product=Fedora \
                 --boolean_query='keywords-substring-Patch'
Server error: <Fault 53: 'value0-0-0 is not a valid parameter for the Bugzilla::Bug::match function.'>

Using the web interface to do the same search, I see the field names have changed from
   field0-0-0
   type0-0-0
   value0-0-0 
to
   f1
   o1
   v1

Full search URL:
https://bugzilla.redhat.com/buglist.cgi?f1=keywords&list_id=38384&o1=substring&classification=Fedora&query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&v1=Patch&product=Fedora


Version-Release number of selected component (if applicable):
python-bugzilla-0.6.2-2.fc16.noarch

And python-bugzilla-0.6.3pre-2.fc16.noarch (*)

(*) This is a personal build of the latest upstream code from
      git://git.fedorahosted.org/git/python-bugzilla
    plus a patch from Michal Novotny's git repo.

How reproducible:
every time

Steps to Reproduce:
1. use bugzilla command line tool to do a boolean search
bugzilla query --bug_status=NEW,ASSIGNED --product=Fedora \
               --boolean_query='keywords-substring-Patch'
  
Actual results:
Server error: <Fault 53: 'value0-0-0 is not a valid parameter for the Bugzilla::Bug::match function.'>

Expected results:
successful query

Additional info:
Comment 1 Jeff Bastian 2012-05-25 15:32:27 EDT
Created attachment 586932 [details]
patch to change boolean search field names

As a quick test, I just tried changing the field names in /usr/bin/bugzilla and dropping the and-count and or-count values (so this only works for one search term); see the attached patch.

However, even using the proper f1, o1, and v1 field names, I'm still getting an error:

$ bugzilla query --bug_status=NEW,ASSIGNED --product=Fedora \
                 --boolean_query='keywords-substring-Patch'

Server error: <Fault 53: 'v1 is not a valid parameter for the Bugzilla::Bug::match function.'>

I'm not sure why it's complaining that v1 is not valid when it clearly uses v1=Patch as shown the URL above.
Comment 2 Jeff Bastian 2012-05-25 16:08:39 EDT
I'm wondering if the "v1 is not a valid parameter" error is a bug in the xmlrpc interface, so I also opened bug 825370
Comment 3 Jeff Bastian 2012-05-25 16:14:12 EDT
Created attachment 586942 [details]
patched python-bugzilla module

If it helps, attached is my patched version of the python-bugzilla module
Comment 4 Jeff Bastian 2012-05-25 16:22:00 EDT
Ahhh, apparently you must add the query_type=advanced parameter to your search terms.  This was not necessary with Bugzilla 3.6 (maybe it was implied), but it is required now.
 
 query = dict()
+query['query_format'] = 'advanced'
 query['bug_status'] = [ 'NEW', 'ASSIGNED' ]

This fixes the "v1 is not a valid parameter" error.

The search is really slow, though, compared to BZ 3.6...
Comment 5 Jeff Bastian 2012-05-25 16:28:30 EDT
Created attachment 586947 [details]
patch to change boolean search field names

Updated patch to also add the query_format parameter to the boolean search:

  q['query_format'] = 'advanced'

The patch still needs work:
  1. to handle multiple search terms with AND, OR, and (PARANTHESES GROUPS)
  2. boolean charts still have the old search terms like 'field%i-%i-%i'
Comment 6 Don Zickus 2012-05-29 09:58:25 EDT
Hi Jeff,

Yes the query_format thing is now necessary, not sure why it changed but it did.  The slowness is because of the way python-bugzilla works when fetching data on the backend when not all the data is provided by the query result.  For some reason, the response from the server for a list of valid data fields takes up to 50 seconds on bugzilla4 as opposed to a few seconds on bugzilla 3.6.

The change I have been using to compensate is putting all the required data inside an  q['include_fields']= [...] array.  This reduces the lookup delays.  Of course it isn't ideal, but it does work around the problem for now.

Cheers,
Don
Comment 7 Jeff Bastian 2012-05-29 14:42:17 EDT
Created attachment 587482 [details]
example script

The include_fields array doesn't seem to really help performance.  In the attached example, I'm requesting only 3 fields: bug_id, component, and short_desc.

It apparently only takes 3 or 4 seconds to run the query, but then it takes 52 seconds to print the first entry!  And each subsequent line of output takes another 1 or 2 seconds each.

[jbastian@localhost tmp]$ ./bz-query.py 
...
[13:30:43] Making query
[13:30:46] Printing report
[13:31:38] 72487 initscripts RFE: Add support for disabling the duplicate IP check
[13:31:39] 112727 rpm Filenames in manifest can not contain '#'
[13:31:41] 128804 rpm Implementation of per-file dependencies
[13:31:42] 129820 initscripts initscripts maclist patch
...



If I enter the debugger right after making the query (uncomment line 46), I can see that the results are _not_ populated with the requested include_fields:

(Pdb) dir(results[0])
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '__weakref__', '_dowhiteboard', 'addcc', 'addcomment', 'addtag', 'appendwhiteboard', 'autorefresh', 'bug_id', 'bugzilla', 'close', 'deletecc', 'deltag', 'get_flag_status', 'get_flag_type', 'get_flags', 'gettags', 'getwhiteboard', 'id', 'prependwhiteboard', 'refresh', 'reload', 'setassignee', 'setstatus', 'setwhiteboard', 'updateflags', 'url']


I see bug_id, but where are component and short_desc?

When I examine component, it fills in the missing data:

(Pdb) results[0].component
['initscripts']


But it also gets a *lot* more than I originally requested!

(Pdb) dir(results[0])
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '__weakref__', '_dowhiteboard', 'actual_time', 'addcc', 'addcomment', 'addtag', 'alias', 'appendwhiteboard', 'assigned_to', 'attach_data.thedata', 'attachments', 'attachments.description', 'attachments.filename', 'attachments.isobsolete', 'attachments.ispatch', 'attachments.isprivate', 'attachments.mimetype', 'attachments.submitter', 'attachstatusdefs.name', 'autorefresh', 'blocked', 'bug_file_loc', 'bug_group', 'bug_id', 'bug_severity', 'bug_status', 'bugzilla', 'build_id', 'cc', 'cclist_accessible', 'cf_build_id', 'cf_clone_of', 'cf_conditional_nak', 'cf_crm', 'cf_cust_facing', 'cf_devel_whiteboard', 'cf_documentation_action', 'cf_environment', 'cf_extra_component', 'cf_extra_version', 'cf_fixed_in', 'cf_internal_whiteboard', 'cf_mount_type', 'cf_partner', 'cf_pm_score', 'cf_qa_whiteboard', 'cf_qe_conditional_nak', 'cf_regression_status', 'cf_release_notes', 'cf_story_points', 'cf_type', 'cf_verified', 'cf_verified_branch', 'class', 'classification', 'clone_of', 'close', 'commenter', 'component', 'conditional_nak', 'content', 'creation_time', 'creation_ts', 'cust_facing', 'days_elapsed', 'deadline', 'deletecc', 'delta_ts', 'deltag', 'dependson', 'devel_whiteboard', 'estimated_time', 'everconfirmed', 'ext_bz_bug_map.ext_bz_bug_id', 'external_bugs', 'external_bugzilla.description', 'external_bugzilla.url', 'external_choices', 'fixed_in', 'flag_types', 'flagtypes.name', 'get_flag_status', 'get_flag_type', 'get_flags', 'gettags', 'getwhiteboard', 'groups', 'id', 'internal_whiteboard', 'keywords', 'last_change_time', 'lastdiffed', 'longdesc', 'longdescs', 'longdescs.count', 'longdescs.isprivate', 'op_sys', 'owner_idle_time', 'partner', 'percentage_complete', 'pm_score', 'prependwhiteboard', 'priority', 'product', 'product_id', 'qa_contact', 'qa_whiteboard', 'qe_conditional_nak', 'refresh', 'release_notes', 'reload', 'remaining_time', 'rep_platform', 'reporter', 'reporter_accessible', 'reporter_id', 'requestees.login_name', 'resolution', 'see_also', 'setassignee', 'setstatus', 'setters.login_name', 'setwhiteboard', 'short_desc', 'status_whiteboard', 'summary', 'tag', 'target_milestone', 'target_release', 'updateflags', 'url', 'vcs_commits', 'verified', 'version', 'view', 'votes', 'work_time']


No wonder it takes so long to run!
Comment 8 Jeff Bastian 2012-05-29 16:25:51 EDT
Ok, I found a bug in my unofficial build which was resetting the include_fields parameter.  With that fixed, it's getting the bug_id and component very quickly, but not short_desc.

[jbastian@localhost tmp]$ ./bz-query.py 
...
[15:20:35] Logged into bugzilla
[15:20:35] Formulating query
[15:20:35] Making query
[15:20:38] Printing report
[15:20:38] 72487 initscripts
[15:20:38] 112727 rpm
[15:20:38] 128804 rpm
[15:20:38] 129820 initscripts
...

It now generates the whole report in just a few seconds.

I'll keep poking at it and see if I can get short_desc to work also.
Comment 10 Don Zickus 2012-05-29 18:51:36 EDT
Hi Jeff,

Can you try using this tree:

git://git.engineering.redhat.com/users/arozansk/python-bugzilla.git

It's a little different than Will's but has some backend changes to handle various conversions (bug_id -> id, bug_status -> short_desc -> summary...).  You will still need your other patch for 'advance' query format.

Cheers,
Don
Comment 11 Jeff Bastian 2012-05-30 15:47:27 EDT
Thanks Don!  The patches in Aristeu's tree help a lot!

The short_desc -> summary change was the source of my speed problems.  My test script runs in just a few seconds now that I've changed my column_list (or include_fields) to use summary instead.


So, now this BZ is just back to the original problem of
   field0-0-0   =>   f1
   type0-0-0    =>   o1
   value0-0-0   =>   v1
in the bin/bugzilla script

Jeff
Comment 12 Don Zickus 2012-05-30 16:59:42 EDT
Hi Jeff,

I put up an rpm with some more changes here:

http://people.redhat.com/dzickus/python-bugzilla/

You are the second person to tell me about the field0-0-0 => f1 changes, but I can't seem to duplicate the problem on my end for some reason.  I don't have your changes but can run the sample query you had in comment0 with no problem.

I am trying to find the change that fixed that but can't.

Can you take the above rpm or the below git tree and run your failed script again?

git://git.engineering.redhat.com/users/dzickus/python-bugzilla

I am willing to take your change and pass it along to upstream, but I am trying to figure out if it could be translated by the backend instead of the front end.  However, in order to do that, I was hoping to see the failure. :-(

Cheers,
Don
Comment 13 Jeff Bastian 2012-05-31 14:41:42 EDT
Hmm, it's working for me now too.

I even tried with an older version of the module -- python-bugzilla-0.6.0-1.el6.noarch -- on RHEL6 and it worked fine.

Maybe something changed on the bugzilla side so it no longer rejects field0-0-0 syntax?
Comment 14 Jeff Bastian 2012-05-31 14:48:57 EDT
It also works fine with the python-bugzilla-0.6.3-0.dz.el6.noarch.rpm package from your people page.

I can't explain why it started working again...
Comment 15 Will Woods 2012-06-15 14:01:47 EDT
*** Bug 832493 has been marked as a duplicate of this bug. ***
Comment 16 Jon Ciesla 2012-06-18 10:53:06 EDT
Just updated to 0.7.0, and I now get this on fedora-infrastructure/scripts/process-git-requests.
Comment 17 Ales Zelinka 2012-06-21 10:52:52 EDT
Hi,

with python-bugzilla-0.7.0-1.el6.noarch/python-bugzilla-0.7.0-1.fc17.noarch I'm getting 
xmlrpclib.Fault: <Fault 53: 'type0-1-0 is not a valid parameter for the Bugzilla::Bug::match function.'>

with python-bugzilla-0.6.3-0.dz.el6.noarch the error changes to 
xmlrpclib.Fault: <Fault 53: 'type0-1-0 is not a valid parameter for the Bugzilla::Bug::match function.'>

The full traceback is:
Traceback (most recent call last):
  File "./new-components", line 47, in <module>
    buglist = bugzilla.query({'product': 'Red Hat Enterprise Linux %s'% major, 'type0-0-0':'substring','field0-0-0':'cf_internal_whiteboard','value0-0-0':'NwCmp','type0-1-0':'substring','field0-1-0':'flagtypes.name','value0-1-0':'rhel-%s.0+'%release})
  File "/usr/lib/python2.6/site-packages/bugzilla/base.py", line 544, in query
    r = self._query(query)
  File "/usr/lib/python2.6/site-packages/bugzilla/rhbugzilla.py", line 578, in _query
    ret = self._proxy.Bug.search(query)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.6/site-packages/bugzilla/base.py", line 1020, in request_with_cookies
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 53: 'type0-1-0 is not a valid parameter for the Bugzilla::Bug::match function.'>


python-bugzilla-0.6.0-1.el6.noarch is ok (has other bugs but not this one :))
Comment 18 Ales Zelinka 2012-06-21 10:54:54 EDT
I failed at copy-pasting. 

Should be:

with python-bugzilla-0.6.3-0.dz.el6.noarch the error changes to 
xmlrpclib.Fault: <Fault 53: 'value0-0-0 is not a valid parameter for the Bugzilla::Bug::match function.'>
Comment 19 Don Zickus 2012-06-21 12:07:50 EDT
Hi Ales,

Your above output shows:

 File "./new-components", line 47, in <module>
    buglist = bugzilla.query({'product': 'Red Hat Enterprise Linux %s'% major, 'type0-0-0':'substring','field0-0-0':'cf_internal_whiteboard','value0-0-0':'NwCmp','type0-1-0':'substring','field0-1-0':'flagtypes.name','value0-1-0':'rhel-%s.0+'%release})
  File "/usr/lib/python2.6/site-packages/bugzilla/base.py", line 544, in query

It seems like new-components is building its own bugzilla.query.

If you are going to build your own queries you now need to add:

'query_format' : 'advanced'

to be able to handle those field types.

Cheers,
Don
Comment 20 Ales Zelinka 2012-06-22 06:27:12 EDT
Thank you Don, that did he trick (with python-bugzilla-0.7.0-1.el6.noarch.rpm)
Comment 21 Cole Robinson 2012-12-27 21:50:18 EST
I think everything here is fixed. If anyone is still hitting an issue with current python-bugzilla, please paste a command line reproducer or a code snippet that reproduces. Closing until then.

Note You need to log in before you can comment on or make changes to this bug.