Bug 1370273

Summary:	Custom field creation failure due to proxy timeout
Product:	[Community] Bugzilla	Reporter:	Dan Raeuftlin <draeuftl>
Component:	Administration	Assignee:	PnT DevOps Devs <hss-ied-bugs>
Status:	CLOSED WONTFIX	QA Contact:	tools-bugs <tools-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.4	CC:	agk, draeuftl, huiwang, khong, mtahir, mtyson, qgong, tcarlin
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-16 06:06:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Dan Raeuftlin 2016-08-25 19:08:55 UTC

Description of problem:
Creating a new custom field in Bugzilla via the administration web interface, fails due to a proxy timeout error.

This caused the httpd service to hang under max load, which eventually caused Bugzilla to no longer accept new connections.

Version-Release number of selected component (if applicable):


How reproducible:
Custom field creation is not a common task, as there are only 31 total custom fields in Bugzilla.  This also previously occurred back in May 2015.

Steps to Reproduce:
1.  Administration -> Custom Fields -> Add a new custom field
2.  Update new custom field setting as specified per request.  In this particular case it was For a Product and mandatory.
3.  Click Create

Actual results:
httpd proxy timeout error.  Reloading the Bugzilla webui allowed temporary access until httpd eventually died.

Expected results:
Successful creation of custom field

Additional info:
http access was restored after stopping the offending httpd processes.

However, the Bugzilla administration custom field page still displays errors when attempting to view ANY custom field.

Example:

An error occurred while performing a database operation:

DBD::Pg::db selectall_arrayref failed: ERROR:  relation "cf_group" does not exist
LINE 1: ...d,value,sortkey,isactive,visibility_value_id FROM cf_group O...

Comment 4 Jeff Fearn 🐞 2016-08-25 23:56:30 UTC

The underlying problem here is that the CF code alters the schema of the bugs table, now that has a 2.5 minute processing time IF there are no constraints on the field. If there are constraints then every single row in the bugs table needs to be individually modified which invalidates all the caches on the database and has a long processing time and a huge impact on performance.

The real fix here would be to change it so that any CF that has constraints, e.g. a default value or not null, is added as a separate table and joined to the bugs table with a relationship table.

This would be very fast to add to the DB and low impact, however, it is a significant change from upstream and may require existing data being split out in to the same structure to avoid having to maintain multiple code paths for CFs.

Comment 7 Rony Gong 🔥 2016-08-26 03:44:14 UTC

Hi Dan, Could you provide the detail info of this added custom field? I am Bugzilla QE, and I'm try my best to reproduce this bug in the test env, also I will test the time of adding this custom field when bugzilla outage.

Comment 8 Matt Tyson 🤬 2016-08-28 22:31:22 UTC

It's always been the case that custom field changes need to be done with a special script during an outage window.

It was like this back in the MySQL days as well.

We should probably take away the button from the web interface so the BZ admins aren't tempted to click it.

Comment 9 Jeff Fearn 🐞 2016-09-01 10:19:54 UTC

The button has been removed in another bug, this bug will be about changing the CF code so it uses the same approach on all CFs, to create a relationship table between bugs and the new CF table and not change the bugz table schema.

Comment 11 Jeff Fearn 🐞 2018-03-16 06:06:27 UTC

This needs to be fixed upstream.