Bug 1296181 - Bugzilla rule engine - kill switch activated due to deadlock in DB
Bugzilla rule engine - kill switch activated due to deadlock in DB
Product: Bugzilla
Classification: Community
Component: Internal Tools (Show other bugs)
Unspecified Unspecified
medium Severity medium (vote)
: ---
: ---
Assigned To: Matt Tyson
Depends On:
  Show dependency treegraph
Reported: 2016-01-06 09:27 EST by Libor Miksik
Modified: 2016-02-14 19:59 EST (History)
5 users (show)

See Also:
Fixed In Version: 4.4.11048.2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-02-14 19:57:23 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Libor Miksik 2016-01-06 09:27:23 EST
kill switch was automatically activated with following error message:
Please for investigation what was wrong. I've disabled kill switch to unblock all the rules.


DBD::Pg::db do failed: ERROR: deadlock detected DETAIL: Process 11342 waits for ShareLock on transaction 44851708; blocked by process 11386. Process 11386 waits for ExclusiveLock on tuple (56238,1) of relation 527931 of database 16384; blocked by process 11342. HINT: See server log for query details. [for Statement "UPDATE bugs SET delta_ts = ? WHERE bug_id = ?"] at Bugzilla/Bug.pm line 1318 Bugzilla::Bug::update('Bugzilla::Bug=HASH(0x6ff4c78)') called at /loader/0x5c0e818/Bugzilla/Extension/RuleEngine/Job.pm line 1006 Bugzilla::Extension::RuleEngine::Job::__ANON__() called at /usr/share/perl5/vendor_perl/Try/Tiny.pm line 76 eval {...} called at /usr/share/perl5/vendor_perl/Try/Tiny.pm line 67 Try::Tiny::try('CODE(0x6dc9518)', 'Try::Tiny::Catch=REF(0x4b4ab08)') called at /loader/0x5c0e818/Bugzilla/Extension/RuleEngine/Job.pm line 1024 Bugzilla::Extension::RuleEngine::Job::_rule_do_action('Bugzilla::Extension::RuleEngine::Job', 'Bugzilla::Bug=HASH(0x6ff4c78)', 'Bugzilla::Extension::RuleEngine::Rule=HASH(0x6c62a10)', 'ARRAY(0x5cec860)') called at /loader/0x5c0e818/Bugzilla/Extension/RuleEngine/Job.pm line 297 Bugzilla::Extension::RuleEngine::Job::do_job_on_bug('Bugzilla::Extension::RuleEngine::Job', 'Bugzilla::Bug=HASH(0x6ff4c78)', 'ARRAY(0x5cec128)') called at /loader/0x5c0e818/Bugzilla/Extension/RuleEngine/Job.pm line 224 Bugzilla::Extension::RuleEngine::Job::do_job('Bugzilla::Extension::RuleEngine::Job', 'HASH(0x6a79d98)') called at /loader/0x5c0e818/Bugzilla/Extension/RuleEngine/Job.pm line 141 eval {...} called at /loader/0x5c0e818/Bugzilla/Extension/RuleEngine/Job.pm line 136 Bugzilla::Extension::RuleEngine::Job::work('Bugzilla::Extension::RuleEngine::Job', 'TheSchwartz::Job=HASH(0x457fba8)') called at /usr/local/share/perl5/TheSchwartz/Worker.pm line 29 eval {...} called at /usr/local/share/perl5/TheSchwartz/Worker.pm line 29 TheSchwartz::Worker::work_safely('Bugzilla::Extension::RuleEngine::Job', 'TheSchwartz::Job=HASH(0x457fba8)') called at /usr/local/share/perl5/TheSchwartz.pm line 678 TheSchwartz::work_once('Bugzilla::JobQueue=HASH(0x64dce10)') called at Bugzilla/JobQueue.pm line 176 Bugzilla::JobQueue::work_once('Bugzilla::JobQueue=HASH(0x64dce10)') called at /usr/local/share/perl5/TheSchwartz.pm line 634 TheSchwartz::work_until_done('Bugzilla::JobQueue=HASH(0x64dce10)') called at Bugzilla/JobQueue/Runner.pm line 229 eval {...} called at Bugzilla/JobQueue/Runner.pm line 226 Bugzilla::JobQueue::Runner::_do_work('Bugzilla::JobQueue::Runner=HASH(0x25d3f40)', 'work_until_done') called at Bugzilla/JobQueue/Runner.pm line 207 Bugzilla::JobQueue::Runner::gd_run('Bugzilla::JobQueue::Runner=HASH(0x25d3f40)') called at /usr/share/perl5/vendor_perl/Daemon/Generic.pm line 190 Daemon::Generic::new('Bugzilla::JobQueue::Runner') called at /var/www/html/bugzilla/jobqueue.pl line 23
Comment 1 Matt Tyson 2016-01-08 00:36:59 EST
Bugzilla should be detecting and retrying later since this error is transient.

It seems the code isn't properly detecting the message.  I've tweaked it a bit which should stop this from re-occurring.
Comment 3 Hui Wang 2016-01-25 05:11:12 EST
I have tried to reproduce the issue in comment#0, but failed.
Change the status to verified at that moment. If you can reproduce this issue, pls reopen this issue and pls help to give the reproduceable scenario.
Comment 5 Matt Tyson 2016-02-14 19:57:23 EST
This change is now live. If there are any issues, do not reopen this bug.
Instead, you should create a new bug and reference this bug.

Note You need to log in before you can comment on or make changes to this bug.