Bug 1418254

Summary: search filter with last change date (delta_ts) does not work properly
Product: [Community] Bugzilla Reporter: Michal Hlavinka <mhlavink>
Component: Query/Bug ListAssignee: PnT DevOps Devs <hss-ied-bugs>
Status: CLOSED NEXTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 5.0CC: crobinso, dan, huiwang, qgong, yijli
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-14 02:30:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michal Hlavinka 2017-02-01 11:39:34 UTC
Created attachment 1246615 [details]
reproducer script

Description of problem:
When using a query with 'delta_ts', the results do not honor this properly. For advanced query, the result is correct(BZ5.0) or shifted a few hours(BZ4.4). For boolean query, delta_ts is ignored. Behaviour is wrong in both rhbz4.4 and beta-rhbz5, but different.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.query for bugs with id=1000000 and get its delta_ts = 2016-12-01 00:51:08
2.query for bugs with id=1000000 AND delta_ts > 2 minutes before 1000000's bug delta_ts


Actual results:
no bug found


Expected results:
bug found

Additional info:
Result depends on what type of query I use. Whether I use boolean query:
{'boolean_query': 'delta_ts-greaterthaneq-2016-12-01 00:49:08', 'bug_id': [1000000]}
OR advanced format query:
{'value0-0-0': '2016-12-01 00:49:08', 'query_format': 'advanced', 'field0-0-0': 'delta_ts', 'id': [1000000], 'type0-0-0': 'greaterthaneq'}

In both cases, results are not reliable.
See reproducer python script for more details.

$ python3 ./test.py
Testing BETA BUGZILLA 5 https://beta.bugzilla.redhat.com/bugzilla/xmlrpc.cgi
Bug #1000000 has delta_ts = 2015-06-16 12:34:56
for 2 minutes before query= {'boolean_query': 'delta_ts-greaterthaneq-2015-06-16 12:32:56', 'Bugzilla_token': '269554-Ywk6poRceT', 'bug_id': [1000000]}
     * pass * bug matched and it should
for 2 minutes before query= {'field0-0-0': 'delta_ts', 'Bugzilla_token': '269554-Ywk6poRceT', 'id': [1000000], 'type0-0-0': 'greaterthaneq', 'query_format': 'advanced', 'value0-0-0': '2015-06-16 12:32:56'}
     * pass * bug matched and it should

for 2 minute after query= {'boolean_query': 'delta_ts-greaterthaneq-2015-06-16 12:36:56', 'Bugzilla_token': '269554-Ywk6poRceT', 'bug_id': [1000000]}
     * FAIL * bug matched and it should not match
for 2 minute after query= {'field0-0-0': 'delta_ts', 'Bugzilla_token': '269554-Ywk6poRceT', 'id': [1000000], 'type0-0-0': 'greaterthaneq', 'query_format': 'advanced', 'value0-0-0': '2015-06-16 12:36:56'}
     * pass * no match and it should not match

for 1 day before query= {'boolean_query': 'delta_ts-greaterthaneq-2015-06-15 12:34:56', 'Bugzilla_token': '269554-Ywk6poRceT', 'bug_id': [1000000]}
     * pass * bug matched and it should
for 1 day before query= {'field0-0-0': 'delta_ts', 'Bugzilla_token': '269554-Ywk6poRceT', 'id': [1000000], 'type0-0-0': 'greaterthaneq', 'query_format': 'advanced', 'value0-0-0': '2015-06-15 12:34:56'}
     * pass * bug matched and it should

for 1 day after query= {'boolean_query': 'delta_ts-greaterthaneq-2015-06-17 12:34:56', 'Bugzilla_token': '269554-Ywk6poRceT', 'bug_id': [1000000]}
     * FAIL * bug matched and it should not match
for 1 day after query= {'field0-0-0': 'delta_ts', 'Bugzilla_token': '269554-Ywk6poRceT', 'id': [1000000], 'type0-0-0': 'greaterthaneq', 'query_format': 'advanced', 'value0-0-0': '2015-06-17 12:34:56'}
     * pass * no match and it should not match


******************************************************
******************************************************

Testing production BUGZILLA 4.4 https://bugzilla.redhat.com/xmlrpc.cgi
Bug #1000000 has delta_ts = 2016-12-01 00:51:08
for 2 minutes before query= {'boolean_query': 'delta_ts-greaterthaneq-2016-12-01 00:49:08', 'Bugzilla_token': '269554-y1RGE3cZFf', 'bug_id': [1000000]}
     * pass * bug matched and it should
for 2 minutes before query= {'field0-0-0': 'delta_ts', 'Bugzilla_token': '269554-y1RGE3cZFf', 'id': [1000000], 'type0-0-0': 'greaterthaneq', 'query_format': 'advanced', 'value0-0-0': '2016-12-01 00:49:08'}
     * FAIL * no match and it should

for 2 minute after query= {'boolean_query': 'delta_ts-greaterthaneq-2016-12-01 00:53:08', 'Bugzilla_token': '269554-y1RGE3cZFf', 'bug_id': [1000000]}
     * FAIL * bug matched and it should not match
for 2 minute after query= {'field0-0-0': 'delta_ts', 'Bugzilla_token': '269554-y1RGE3cZFf', 'id': [1000000], 'type0-0-0': 'greaterthaneq', 'query_format': 'advanced', 'value0-0-0': '2016-12-01 00:53:08'}
     * pass * no match and it should not match

for 1 day before query= {'boolean_query': 'delta_ts-greaterthaneq-2016-11-30 00:51:08', 'Bugzilla_token': '269554-y1RGE3cZFf', 'bug_id': [1000000]}
     * pass * bug matched and it should
for 1 day before query= {'field0-0-0': 'delta_ts', 'Bugzilla_token': '269554-y1RGE3cZFf', 'id': [1000000], 'type0-0-0': 'greaterthaneq', 'query_format': 'advanced', 'value0-0-0': '2016-11-30 00:51:08'}
     * pass * bug matched and it should

for 1 day after query= {'boolean_query': 'delta_ts-greaterthaneq-2016-12-02 00:51:08', 'Bugzilla_token': '269554-y1RGE3cZFf', 'bug_id': [1000000]}
     * FAIL * bug matched and it should not match
for 1 day after query= {'field0-0-0': 'delta_ts', 'Bugzilla_token': '269554-y1RGE3cZFf', 'id': [1000000], 'type0-0-0': 'greaterthaneq', 'query_format': 'advanced', 'value0-0-0': '2016-12-02 00:51:08'}
     * pass * no match and it should not match



BZ5 for boolean query always matches bug whatever delta_ts is, so it ignores delta_ts completely
BZ5 for advanced query it respects delta_ts

BZ4 for boolean query always matches bug whatever delta_ts is, so it ignores delta_ts completely
BZ4 for advanced query it respects delta_ts, but value is shifted (2 minutes before is not enough, a few hours is (test uses 1 day difference, but 6 hours is enough too, probably some hidden timezone conversion))

Comment 1 Jeff Fearn 🐞 2017-02-01 12:22:42 UTC
The difference between BZ4 and BZ5 appears to be due to a time zone transformation done as part of the upgrade process. If you use EST times for BZ4 it should match the BZ5 behavior using UTC times.

"boolean_query" isn't a Bugzilla feature, it's a pyton-bugzilla feature, you should probably chase up that community to see if it's working as expected.

Comment 2 Michal Hlavinka 2017-02-01 12:30:00 UTC
(In reply to Jeff Fearn from comment #1)
> The difference between BZ4 and BZ5 appears to be due to a time zone
> transformation done as part of the upgrade process. If you use EST times for
> BZ4 it should match the BZ5 behavior using UTC times.

the problem is that the result uses one timezone and query uses different time zone. I'm not picking random time, but I use the time that bugzilla returned. Result and query should use the same time zone without need to do any magick on user side.

> "boolean_query" isn't a Bugzilla feature, it's a pyton-bugzilla feature, you
> should probably chase up that community to see if it's working as expected.

Thanks for the info. I will investigate this more.

Comment 3 Jeff Fearn 🐞 2017-02-01 12:42:06 UTC
(In reply to Michal Hlavinka from comment #2)
> (In reply to Jeff Fearn from comment #1)
> > The difference between BZ4 and BZ5 appears to be due to a time zone
> > transformation done as part of the upgrade process. If you use EST times for
> > BZ4 it should match the BZ5 behavior using UTC times.
> 
> the problem is that the result uses one timezone and query uses different
> time zone. I'm not picking random time, but I use the time that bugzilla
> returned. Result and query should use the same time zone without need to do
> any magick on user side.

Do you have your user preference for "Timezone used to display dates and times" set to "Same as Server"?

Comment 4 Michal Hlavinka 2017-02-01 14:37:45 UTC
yes, both in production and beta bugzilla. Btw, my complain is not that the time is shifted compared to time on my computee. I get bug's delta_ts from server (xmlrpc) in some timezone and when I use the same time as parameter, it's wrong from server's p.o.v., because it probably does some timezone conversion only in one direction. Either in get data on server -> send to user OR data form user -> server. It should do such conversion in all cases or never.

Comment 5 Jeff Fearn 🐞 2017-02-01 21:40:17 UTC
(In reply to Michal Hlavinka from comment #4)
> yes, both in production and beta bugzilla. Btw, my complain is not that the
> time is shifted compared to time on my computee. I get bug's delta_ts from
> server (xmlrpc) in some timezone and when I use the same time as parameter,
> it's wrong from server's p.o.v., because it probably does some timezone
> conversion only in one direction. Either in get data on server -> send to
> user OR data form user -> server. It should do such conversion in all cases
> or never.

Yeah I'm just trying to figure out which parts of the code are in play for this. Bugzilla doesn't actually store the time zone in the DB >_< But it looks like BZ5 behaves properly so it may be an upstream fix.

Comment 6 Jeff Fearn 🐞 2017-02-08 03:38:11 UTC
OK, so to summarize this bug

1: the time shifting appears to be fixed for BZ5
2: the boolean_query issue appears to be a python-bugzilla bug

If this is correct than there is nothing for us to do here.

Comment 7 Jeff Fearn 🐞 2017-02-08 03:39:17 UTC
Hey Cole, have you heard anything about boolean_query being funky with times?

Comment 8 Cole Robinson 2017-02-08 15:35:05 UTC
(In reply to Jeff Fearn from comment #7)
> Hey Cole, have you heard anything about boolean_query being funky with times?

No, but FYI to the reporter the next version of python-bugzilla is killing the boolean_query logic in favor of converting a web query URL to XMLRPC with helper functions. Details are buried in here:

https://lists.fedorahosted.org/archives/list/python-bugzilla@lists.fedorahosted.org/thread/WCYPOKJZFYOW7RRT44FCM5GQU26O56K4/

And API example here:

http://blog.wikichoon.com/2014/05/invoking-bugzilla-query-url-from.html

Comment 9 Michal Hlavinka 2017-02-13 15:26:54 UTC
(In reply to Jeff Fearn from comment #6)
> OK, so to summarize this bug
> 
> 1: the time shifting appears to be fixed for BZ5
> 2: the boolean_query issue appears to be a python-bugzilla bug
> 
> If this is correct than there is nothing for us to do here.

I agree, given the plan to switch to BZ5 soon, it probably does not make sense to waste time with bz4 fixes right now.

(In reply to Cole Robinson from comment #8)
> (In reply to Jeff Fearn from comment #7)
> No, but FYI to the reporter the next version of python-bugzilla is killing
> the boolean_query logic in favor of converting a web query URL to XMLRPC
> with helper functions.

I've noticed this in git, so I've change all queries in our tool to use {field,value,type}X-Y-Z format