Bug 1249465 - [PgPool]Can't create bug when the jobs number got to 52474
[PgPool]Can't create bug when the jobs number got to 52474
Product: Bugzilla
Classification: Community
Component: Database (Show other bugs)
Unspecified Unspecified
urgent Severity urgent (vote)
: ---
: ---
Assigned To: PnT DevOps Devs
Rony Gong
Depends On:
  Show dependency treegraph
Reported: 2015-08-03 01:13 EDT by Rony Gong
Modified: 2015-08-30 20:16 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-08-30 20:16:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Rony Gong 2015-08-03 01:13:46 EDT
Description of problem:
[PgPool]Can't create bug when the jobs number got to 52474

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.Stop all bugzilla-queue services
2.Then do a very long test against rdu pgpool environment, it will generate 52474 bugzilla jobs
3.Then try to create a new bug.

Actual results:
Page show error when create bug:

Inserting a TheSchwartz::Job=HASH(0x7f0e47eab978) job into the Job Queue failed with the following error: Failed to execute INSERT INTO ts_job (funcid, arg, uniqkey, insert_time, run_after, grabbed_until, priority, coalesce) VALUES (?, ?, ?, ?, ?, ?, ?, ?) with funcid, arg, uniqkey, insert_time, run_after, grabbed_until, priority, coalesce: DBD::Pg::st execute failed: ERROR: right sibling's left-link doesn't match: block 550 links to 1 instead of expected 555 in index "ts_job_funcid_idx" at /usr/share/perl5/vendor_perl/Data/ObjectDriver/Driver/DBI.pm line 396. 

 at /var/www/html/bugzilla/Bugzilla/JobQueue.pm line 98
	Bugzilla::JobQueue::insert(...) called at /var/www/html/bugzilla/Bugzilla/Mailer.pm line 49
	Bugzilla::Mailer::MessageToMTA(...) called at /var/www/html/bugzilla/Bugzilla/BugMail.pm line 461
	Bugzilla::BugMail::sendMail(...) called at /var/www/html/bugzilla/Bugzilla/BugMail.pm line 288
	Bugzilla::BugMail::Send(...) called at /var/www/html/bugzilla/post_bug.cgi line 259
	ModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::var_www_html_bugzilla_post_bug_2ecgi::handler(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 204
	eval {...} called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 204
	ModPerl::RegistryCooker::run(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 170
	ModPerl::RegistryCooker::default_handler(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/Registry.pm line 31
	ModPerl::Registry::handler(...) called at /var/www/html/bugzilla/mod_perl.pl line 134
	Bugzilla::ModPerl::ResponseHandler::handler(...) called at -e line 0
	eval {...} called at -e line 0

Expected results:
52474 jobs in the queue

Additional info:
Comment 3 Jeff Fearn 2015-08-16 20:23:21 EDT
After fixing Bug 1250305, Bug 1251315, Bug 1251317, Bug 1252181, and Bug 1245411, we can not duplicate this. None of those bugs affect the DB directly and none should trigger this issue. It's possible some combination of them trigger the error but we can't prove or test that.

We really need a broken DB, or a reliable way of breaking the DB indexes, to test or triage this issue.
Comment 4 Matt Tyson 2015-08-30 19:54:21 EDT
The index corruption issue is either a bug in postgres, or a hardware fault.

It's not a problem caused by bugzilla or the job queues.
Comment 5 Jason McDonald 2015-08-30 20:16:04 EDT
This issue hasn't been seen for a few weeks now, and is very unlikely to be a problem with the Bugzilla codebase anyway, so let's close it until someone is able to reproduce it.

Note You need to log in before you can comment on or make changes to this bug.