Bug 951309 - beaker-provision sometimes runs commands twice
Summary: beaker-provision sometimes runs commands twice
Alias: None
Product: Beaker
Classification: Community
Component: lab controller
Version: 0.12
Hardware: Unspecified
OS: Unspecified
unspecified vote
Target Milestone: HOTFIX
Assignee: Dan Callaghan
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2013-04-12 03:44 UTC by Dan Callaghan
Modified: 2018-02-06 00:41 UTC (History)
6 users (show)

Clone Of:
Last Closed: 2013-04-23 05:05:11 UTC

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 903902 None None None Never

Internal Trackers: 903902

Description Dan Callaghan 2013-04-12 03:44:21 UTC
beaker-provision sometimes runs commands more than once.

In order to trigger this, beaker-provision must receive several new commands at once that take longer than the polling interval (20 seconds) to complete. I can reproduce it in my devel environment just by queueing a job with two recipes. (Adding more recipes should make it more likely to be triggered, in labs where the mirror is faster or for power types which are faster than virsh.)

The problem was introduced in commit 1a180c0f in Beaker 0.10.1 but became more noticeable in 0.12.0 because of the new code for enforcing task status consistency:

2013-04-12 12:51:55,991 bkr.server.model ERROR command callback failed: Status for T:1084 updated in another transaction

This is the also the root cause of the duplicate system_status_duration rows in bug 903902.

The problem is with these lines in the poll() method in LabController/src/bkr/labcontroller/provision.py:

 73)             greenlet.link(lambda greenlet: self._handler_complete(
 74)                     greenlet, command['id']))

Because of Python's late binding, the 'command' variable in the lambda references the final command in the loop, not the one being handled at the time the lambda was created. So commands might be removed from the self.commands dict before they are actually completed, causing them to be run again on the next iteration of the polling loop.

Comment 1 Dan Callaghan 2013-04-17 06:51:33 UTC
On Gerrit: http://gerrit.beaker-project.org/1892

Comment 3 Raymond Mancy 2013-04-22 13:26:45 UTC


Comment 5 Dan Callaghan 2013-04-23 05:05:11 UTC
Beaker 0.12.1 has been released.

Note You need to log in before you can comment on or make changes to this bug.