Hide Forgot
Description of problem: Creating a new custom field in Bugzilla via the administration web interface, fails due to a proxy timeout error. This caused the httpd service to hang under max load, which eventually caused Bugzilla to no longer accept new connections. Version-Release number of selected component (if applicable): How reproducible: Custom field creation is not a common task, as there are only 31 total custom fields in Bugzilla. This also previously occurred back in May 2015. Steps to Reproduce: 1. Administration -> Custom Fields -> Add a new custom field 2. Update new custom field setting as specified per request. In this particular case it was For a Product and mandatory. 3. Click Create Actual results: httpd proxy timeout error. Reloading the Bugzilla webui allowed temporary access until httpd eventually died. Expected results: Successful creation of custom field Additional info: http access was restored after stopping the offending httpd processes. However, the Bugzilla administration custom field page still displays errors when attempting to view ANY custom field. Example: An error occurred while performing a database operation: DBD::Pg::db selectall_arrayref failed: ERROR: relation "cf_group" does not exist LINE 1: ...d,value,sortkey,isactive,visibility_value_id FROM cf_group O...
The underlying problem here is that the CF code alters the schema of the bugs table, now that has a 2.5 minute processing time IF there are no constraints on the field. If there are constraints then every single row in the bugs table needs to be individually modified which invalidates all the caches on the database and has a long processing time and a huge impact on performance. The real fix here would be to change it so that any CF that has constraints, e.g. a default value or not null, is added as a separate table and joined to the bugs table with a relationship table. This would be very fast to add to the DB and low impact, however, it is a significant change from upstream and may require existing data being split out in to the same structure to avoid having to maintain multiple code paths for CFs.
Hi Dan, Could you provide the detail info of this added custom field? I am Bugzilla QE, and I'm try my best to reproduce this bug in the test env, also I will test the time of adding this custom field when bugzilla outage.
It's always been the case that custom field changes need to be done with a special script during an outage window. It was like this back in the MySQL days as well. We should probably take away the button from the web interface so the BZ admins aren't tempted to click it.
The button has been removed in another bug, this bug will be about changing the CF code so it uses the same approach on all CFs, to create a relationship table between bugs and the new CF table and not change the bugz table schema.
This needs to be fixed upstream.