Bug 974319

Summary: clear_running_commands XML-RPC call fails with MemoryError
Product: [Retired] Beaker Reporter: Nick Coghlan <ncoghlan>
Component: lab controllerAssignee: Nick Coghlan <ncoghlan>
Status: CLOSED CURRENTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 0.12CC: asaha, dcallagh, llim, qwan, rglasz, rmancy, xjia
Target Milestone: 0.13.x   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 974352 (view as bug list) Environment:
Last Closed: 2013-07-11 02:44:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nick Coghlan 2013-06-14 00:21:05 UTC 
Description of problem:

beaker-provision was restarted with 51 Running operations in the command queue. It failed to restart because the "clear_running_commands" call back to the main server was failing with MemoryError.

Version-Release number of selected component (if applicable):

0.12.1

How reproducible:

Always (with those 51 commands listed as "Running" on the server)

Steps to Reproduce:
1.
2.
3.

Actual results:

XML-RPC call reports MemoryError and beaker-provision fails to start

Expected results:

Running commands are marked as Failed (aborting the associated recipes), beaker-provision starts up and begins processing commands.

Additional info:
Comment 1 Nick Coghlan 2013-06-20 04:48:44 UTC 
Investigation has suggested that this only arises if many stale commands (see bug 974352) have accumulated in the server's command queue, and then the provisioning daemon on the lab controller is restarted.

Due to the lack of a direct link between commands and recipes, it's also risky to execute the state update callbacks for such stale commands - if the system has since been reallocated, the stale callback may end up aborting an unrelated recipe.

Accordingly, the "clear_running_commands" operation will be updated to directly execute the following SQL before moving on to handling more recent commands:

  UPDATE command_queue
  JOIN activity ON activity.id=command_queue.id
  SET command_queue.status="Aborted"
  WHERE command_queue.status = "Running"
    AND activity.created < DATE_ADD(UTC_TIMESTAMP(), INTERVAL -1 DAY);

Any remaining commands (those less than 24 hours old) will then be processed in independent transactions.
Comment 4 Nick Coghlan 2013-06-24 05:22:51 UTC 
On Gerrit: http://gerrit.beaker-project.org/#/c/2036/
Comment 7 xjia 2013-06-28 02:23:03 UTC 
For this bug, I will verify it on July 2nd while doing acceptance test.
Comment 9 Amit Saha 2013-07-11 02:44:31 UTC 
Beaker 0.13.2 has been released. (http://beaker-project.org/docs/whats-new/release-0.13.html#beaker-0-13-2).