Bug 1120052 - queue_processed_recipesets can fail to abort the recipe set if not-enough-systems logic triggers: Invalid state transition: Aborted -> Queued
Summary: queue_processed_recipesets can fail to abort the recipe set if not-enough-sys...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Community
Component: scheduler
Version: 0.17
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: 0.18
Assignee: Dan Callaghan
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-16 07:40 UTC by Dan Callaghan
Modified: 2018-02-06 00:41 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-09-04 05:40:34 UTC


Attachments (Terms of Use)

Description Dan Callaghan 2014-07-16 07:40:54 UTC
The recipe set gets stuck at Processed status, instead of Aborted.

bkr.server.tools.beakerd DEBUG Entering queue_processed_recipesets
bkr.server.tools.beakerd DEBUG recipe: 7619 labController:lab-devel-02.censored.example.com entering not enough systems logic
bkr.server.tools.beakerd DEBUG recipe: 7619 labController:lab-devel-02.censored.example.com Removing system intel-d3c4702-02.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7619 labController:lab-devel-02.censored.example.com Removing system dev-pe1950-01.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7623 labController:lab-devel-02.censored.example.com Removing system intel-d3c4702-02.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7623 labController:lab-devel-02.censored.example.com Removing system dev-pe1950-01.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7620 labController:lab-devel-02.censored.example.com Removing system intel-d3c4702-02.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7620 labController:lab-devel-02.censored.example.com Removing system dev-pe1950-01.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7620 labController:lab-devel-02.censored.example.com Removing system dev-pe2850-01.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7620 labController:lab-devel-02.censored.example.com Removing system dev-pe1950-04.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7620 labController:lab-devel-02.censored.example.com Removing system dev-pe1950-03.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7621 labController:lab-devel-02.censored.example.com Removing system intel-d3c4702-02.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7621 labController:lab-devel-02.censored.example.com Removing system dev-pe1950-01.censored.example.com
bkr.server.tools.beakerd DEBUG recipe: 7621 labController:lab-devel-02.censored.example.com Removing system dev-pe2850-01.censored.example.com
bkr.server.tools.beakerd INFO recipe: 7619 moved from Processed to Queued
bkr.server.tools.beakerd INFO recipe ID 7620 moved from Processed to Aborted
bkr.server.tools.beakerd INFO recipe ID 7621 moved from Processed to Aborted
bkr.server.tools.beakerd INFO recipe ID 7622 moved from Processed to Aborted
bkr.server.tools.beakerd INFO recipe: 7623 moved from Processed to Queued
bkr.server.tools.beakerd ERROR Error in queue_processed_recipeset(6645)
 Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/bkr/server/tools/beakerd.py", line 182, in queue_processed_recipesets
     queue_processed_recipeset(rs_id)
   File "/usr/lib/python2.6/site-packages/bkr/server/tools/beakerd.py", line 288, in queue_processed_recipeset
     recipe.queue()
   File "/usr/lib/python2.6/site-packages/bkr/server/model/scheduler.py", line 2087, in queue
     task._change_status(TaskStatus.queued)
   File "/usr/lib/python2.6/site-packages/bkr/server/model/scheduler.py", line 379, in _change_status
     % (self.t_id, current_status, new_status))
 ValueError: Invalid state transition for T:116867: Aborted -> Queued
bkr.server.tools.beakerd DEBUG Exiting queue_processed_recipesets

Comment 2 Dan Callaghan 2014-08-07 06:36:35 UTC
On Gerrit: http://gerrit.beaker-project.org/3234

Comment 3 Dan Callaghan 2014-08-18 08:06:35 UTC
Steps to reproduce:
1. Pick 3 systems which you have access to reserve, all on the same lab controller.
2. Schedule a job containing 1 recipe set with 4 recipes. Set <hostRequires/> as follows:

Recipe 1:
<hostname value="system1.example.com"/>

Recipe 2:
<hostname value="system2.example.com"/>

Recipe 3:
<or>
  <hostname value="system1.example.com"/>
  <hostname value="system2.example.com"/>
</or>

Recipe 4:
(none)

Expected results:
Recipe set is aborted because the host requirements are not satisfiable (recipe 3 will have no candidate systems, they are used by recipes 1 and 2 instead).

Actual results:
Recipe set is stuck in "processed", beakerd repeatedly spews the error given in comment 0.

Comment 8 Dan Callaghan 2014-09-04 05:40:34 UTC
This bug fix was applied to the release-0.17 branch, but we are not doing any more maintenance releases of the 0.17.x series. So this bug is fixed in Beaker 0.18.0.


Note You need to log in before you can comment on or make changes to this bug.