Description of problem: [PgPool]Can't create bug when the jobs number got to 52474 Version-Release number of selected component (if applicable): 4.4.9038-3 How reproducible: Always Steps to Reproduce: 1.Stop all bugzilla-queue services 2.Then do a very long test against rdu pgpool environment, it will generate 52474 bugzilla jobs 3.Then try to create a new bug. Actual results: Page show error when create bug: Inserting a TheSchwartz::Job=HASH(0x7f0e47eab978) job into the Job Queue failed with the following error: Failed to execute INSERT INTO ts_job (funcid, arg, uniqkey, insert_time, run_after, grabbed_until, priority, coalesce) VALUES (?, ?, ?, ?, ?, ?, ?, ?) with funcid, arg, uniqkey, insert_time, run_after, grabbed_until, priority, coalesce: DBD::Pg::st execute failed: ERROR: right sibling's left-link doesn't match: block 550 links to 1 instead of expected 555 in index "ts_job_funcid_idx" at /usr/share/perl5/vendor_perl/Data/ObjectDriver/Driver/DBI.pm line 396. Traceback: at /var/www/html/bugzilla/Bugzilla/JobQueue.pm line 98 Bugzilla::JobQueue::insert(...) called at /var/www/html/bugzilla/Bugzilla/Mailer.pm line 49 Bugzilla::Mailer::MessageToMTA(...) called at /var/www/html/bugzilla/Bugzilla/BugMail.pm line 461 Bugzilla::BugMail::sendMail(...) called at /var/www/html/bugzilla/Bugzilla/BugMail.pm line 288 Bugzilla::BugMail::Send(...) called at /var/www/html/bugzilla/post_bug.cgi line 259 ModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::var_www_html_bugzilla_post_bug_2ecgi::handler(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 204 eval {...} called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 204 ModPerl::RegistryCooker::run(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 170 ModPerl::RegistryCooker::default_handler(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/Registry.pm line 31 ModPerl::Registry::handler(...) called at /var/www/html/bugzilla/mod_perl.pl line 134 Bugzilla::ModPerl::ResponseHandler::handler(...) called at -e line 0 eval {...} called at -e line 0 Expected results: 52474 jobs in the queue Additional info:
After fixing Bug 1250305, Bug 1251315, Bug 1251317, Bug 1252181, and Bug 1245411, we can not duplicate this. None of those bugs affect the DB directly and none should trigger this issue. It's possible some combination of them trigger the error but we can't prove or test that. We really need a broken DB, or a reliable way of breaking the DB indexes, to test or triage this issue.
The index corruption issue is either a bug in postgres, or a hardware fault. It's not a problem caused by bugzilla or the job queues.
This issue hasn't been seen for a few weeks now, and is very unlikely to be a problem with the Bugzilla codebase anyway, so let's close it until someone is able to reproduce it.