Bug 974319 - clear_running_commands XML-RPC call fails with MemoryError
clear_running_commands XML-RPC call fails with MemoryError
Status: CLOSED CURRENTRELEASE
Product: Beaker
Classification: Community
Component: lab controller (Show other bugs)
0.12
Unspecified Unspecified
unspecified Severity unspecified (vote)
: 0.13.x
: ---
Assigned To: Nick Coghlan
tools-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-13 20:21 EDT by Nick Coghlan
Modified: 2018-02-05 19:41 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 974352 (view as bug list)
Environment:
Last Closed: 2013-07-10 22:44:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nick Coghlan 2013-06-13 20:21:05 EDT
Description of problem:

beaker-provision was restarted with 51 Running operations in the command queue. It failed to restart because the "clear_running_commands" call back to the main server was failing with MemoryError.

Version-Release number of selected component (if applicable):

0.12.1

How reproducible:

Always (with those 51 commands listed as "Running" on the server)

Steps to Reproduce:
1.
2.
3.

Actual results:

XML-RPC call reports MemoryError and beaker-provision fails to start

Expected results:

Running commands are marked as Failed (aborting the associated recipes), beaker-provision starts up and begins processing commands.

Additional info:
Comment 1 Nick Coghlan 2013-06-20 00:48:44 EDT
Investigation has suggested that this only arises if many stale commands (see bug 974352) have accumulated in the server's command queue, and then the provisioning daemon on the lab controller is restarted.

Due to the lack of a direct link between commands and recipes, it's also risky to execute the state update callbacks for such stale commands - if the system has since been reallocated, the stale callback may end up aborting an unrelated recipe.

Accordingly, the "clear_running_commands" operation will be updated to directly execute the following SQL before moving on to handling more recent commands:

  UPDATE command_queue
  JOIN activity ON activity.id=command_queue.id
  SET command_queue.status="Aborted"
  WHERE command_queue.status = "Running"
    AND activity.created < DATE_ADD(UTC_TIMESTAMP(), INTERVAL -1 DAY);

Any remaining commands (those less than 24 hours old) will then be processed in independent transactions.
Comment 4 Nick Coghlan 2013-06-24 01:22:51 EDT
On Gerrit: http://gerrit.beaker-project.org/#/c/2036/
Comment 7 xjia 2013-06-27 22:23:03 EDT
For this bug, I will verify it on July 2nd while doing acceptance test.
Comment 9 Amit Saha 2013-07-10 22:44:31 EDT
Beaker 0.13.2 has been released. (http://beaker-project.org/docs/whats-new/release-0.13.html#beaker-0-13-2).

Note You need to log in before you can comment on or make changes to this bug.