Bug 453902

Summary: buglist.cgi ctype=csv dependson list truncated
Product: [Community] Bugzilla Reporter: FTBFS <ftbfs>
Component: Bugzilla GeneralAssignee: PnT DevOps Devs <hss-ied-bugs>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 3.2CC: dkl, matt_domsch
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-08-28 04:29:43 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description FTBFS 2008-07-03 03:13:47 UTC
Description of problem:
Querying using buglist.csv?ctype=csv and using a cookie that requests the
'dependson' field, the results list is incomplete.

Example:
#!/bin/bash
cookie='COLUMNLIST=alias%20bug_severity%20priority%20rep_platform%20assigned_to%20bug_status%20resolution%20component%20short_desc%20dependson'
url='https://bugzilla.redhat.com/buglist.cgi?bug_id=440169&ctype=csv'
/usr/bin/curl -k --cookie $cookie $url

resulting csv list has 128 entries in the dependson list.


The same call using python-bugzilla's bugzilla program and xmlrpc call:
$ bugzilla --verbose --debug query -f -b 440169

yields a list with 452 entries.

Version-Release number of selected component (if applicable):
bugzilla 2.18

How reproducible:
always

Comment 1 FTBFS 2008-07-03 03:14:23 UTC
Doh.  I should have filed this under my name. -Matt

Comment 2 Noura El hawary 2008-07-03 06:46:01 UTC
Problem verified, trying the url:

https://bugzilla.redhat.com/buglist.cgi?bug_id=440169&field-1-0-0=bug_id&query_format=advanced&remaction=&type-1-0-0=anyexact&value-1-0-0=440169&query_based_on=&columnlist=bug_severity%2Cpriority%2Cassigned_to%2Cbug_status%2Cresolution%2Cop_sys%2Cdependson%2Cshort_desc

the list returns only 128 bugs in the dependson column which is not the complete
list, I am assuming the problem is coming from the Function bugs.DependsOnList
created using the file bugs-functions.mysql, maybe we need to enlarge the TEXT
to be MEDIUMTEXT or LONGTEXT !!

Noura

Comment 3 Matt Domsch 2008-08-28 01:31:06 UTC
ping

Comment 4 David Lawrence 2008-08-28 04:29:43 UTC
(In reply to comment #2)
> Problem verified, trying the url:
> 
> https://bugzilla.redhat.com/buglist.cgi?bug_id=440169&field-1-0-0=bug_id&query_format=advanced&remaction=&type-1-0-0=anyexact&value-1-0-0=440169&query_based_on=&columnlist=bug_severity%2Cpriority%2Cassigned_to%2Cbug_status%2Cresolution%2Cop_sys%2Cdependson%2Cshort_desc
> 
> the list returns only 128 bugs in the dependson column which is not the complete
> list, I am assuming the problem is coming from the Function bugs.DependsOnList
> created using the file bugs-functions.mysql, maybe we need to enlarge the TEXT
> to be MEDIUMTEXT or LONGTEXT !!

Actually according to the MySQL docs, the GROUP_CONCAT function is limited by the GROUP_CONCAT_MAX_LEN setting which defaults to 1024.

"The result is truncated to the maximum length that is given by the group_concat_max_len system variable, which has a default value of 1024. The value can be set higher, although the effective maximum length of the return value is constrained by the value of max_allowed_packet. The syntax to change the value of group_concat_max_len at runtime is as follows, where val  is an unsigned integer:

SET [SESSION | GLOBAL] group_concat_max_len = val;

Beginning with MySQL 5.0.19, the type returned by GROUP_CONCAT() is always VARCHAR unless group_concat_max_len is greater than 512, in which case, it returns a BLOB. (Previously, it returned a BLOB with group_concat_max_len greater than 512 only if the query included an ORDER BY clause.)"

So I have commited the following change to CVS which fixes the truncation issue.

Index: Bugzilla/DB/Mysql.pm
===================================================================
RCS file: /cvs/qa/rh_bugzilla_3/Bugzilla/DB/Mysql.pm,v
retrieving revision 1.17
diff -u -r1.17 Mysql.pm
--- Bugzilla/DB/Mysql.pm	13 Aug 2008 19:24:55 -0000	1.17
+++ Bugzilla/DB/Mysql.pm	28 Aug 2008 04:24:57 -0000
@@ -67,6 +67,10 @@
     # return their data correctly.
     $self->do("SET NAMES utf8") if Bugzilla->params->{'utf8'};
 
+    # REDHAT EXTENSION START 453902 
+    $self->do("SET GROUP_CONCAT_MAX_LEN = 65536;"); 
+    # REDHAT EXTENSION END 453902
+
     # all class local variables stored in DBI derived class needs to have
     # a prefix 'private_'. See DBI documentation.
     $self->{private_bz_tables_locked} = "";

I set it to 65536 which should be enough for the future.

This will be fixed in the next scheduled update.

Dave