Bug 974352 - Log implicit XML-RPC retries on lab controller
Log implicit XML-RPC retries on lab controller
Status: CLOSED CURRENTRELEASE
Product: Beaker
Classification: Community
Component: lab controller (Show other bugs)
0.12
Unspecified Unspecified
unspecified Severity unspecified (vote)
: 0.13.x
: ---
Assigned To: Nick Coghlan
tools-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-14 00:13 EDT by Nick Coghlan
Modified: 2018-02-05 19:41 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 974319
Environment:
Last Closed: 2013-07-10 22:44:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nick Coghlan 2013-06-14 00:13:07 EDT
+++ This bug was initially created as a clone of Bug #974319 +++

Description of problem:

beaker-provision was restarted with 51 Running operations in the command queue. 
It failed to restart because the "clear_running_commands" call back to the main server was failing with MemoryError.

Further investigation showed that the supposedly "Running" commands were anywhere up to 3 weeks old, and almost certainly due to their status not being updated correctly when a failure is encountered while updating the command status in mark_command_completed or mark_command_failed.

Version-Release number of selected component (if applicable):

0.12.1

How reproducible:

Timing related (requires the callback or command status update to fail, rolling back the transaction)

Steps to Reproduce:
1.
2.
3.

Actual results:

Commands are left in "Running" state (unless/until cleared out by clear_running_commands)

Expected results:

Running commands are marked as Completed or Failed as appropriate.

Additional info:
Comment 1 Nick Coghlan 2013-06-20 00:56:42 EDT
Analysis of the available logs suggests that this failure mode occurs when the "mark_command_running" operation succeeds on the server, and then one of the following happens:

1. The lab controller fails to receive the reply and thus assumes the request failed and either retries (giving a "Command X already run" failure) or else abandons the operation.
2. The lab controller receives the reply and handles the command, but the subsequent call to "mark_command_running" or "mark_command_failed" doesn't work.

Thus, this appears to be a secondary failure that arises only in the presence of network stability issues for the main server, the lab controller or both.

The available logs are currently confused by the "implicit retry" behaviour in the lab controller's XML-RPC client code. Since those retries are unlikely to work, and make analysis of the underlying networking misbehaviour more difficult, we will disable them for the next 0.13 maintenance release.
Comment 4 Nick Coghlan 2013-06-25 00:40:00 EDT
On Gerrit: http://gerrit.beaker-project.org/#/c/2057

I updated title to reflect the planned changes, as we're not going to do anything more sophisticated at this point. The only major surgery we're likely to consider for the command queue system is replacing it with a point-to-point messaging system like fedmsg, and that's well beyond the scope of this bug report.
Comment 6 Nick Coghlan 2013-06-26 23:23:39 EDT
These retries are actually needed to allow the lab controllers to handle deliberate restarts of the main web server and genuine network glitches for more remote labs.

So, the current fix isn't appropriate, and needs to be replaced with one that keeps the retry mechanism, and just ensures they're logged appropriately.
Comment 7 Nick Coghlan 2013-07-01 05:25:29 EDT
Completed review of new and improved version on Gerrit:
http://gerrit.beaker-project.org/#/c/2066/
Comment 10 Amit Saha 2013-07-10 22:44:42 EDT
Beaker 0.13.2 has been released. (http://beaker-project.org/docs/whats-new/release-0.13.html#beaker-0-13-2).

Note You need to log in before you can comment on or make changes to this bug.