The schema upgrade notes for Beaker 0.10 included steps for populating the new guest_resource table based on Virtual system rows. This works correctly for recipes which are already running or completed, but it also created guest_resource rows where the recipe is still queued or being scheduled, which is not correct. In Beaker 0.10 the guest_resource row is created when the *host* recipe changes Queued->Scheduled (since the guest recipes themselves are no longer scheduled individually). If the guest recipes already have guest_resource rows, when beakerd tries to move the host recipe Queued->Scheduled it replaces the guest_resource rows resulting in the following exception (and the recipe stays stuck in Queued): 2012-12-02 20:00:04,897 beakerd INFO recipe ID 693117 moved from Queued to Scheduled 2012-12-02 20:00:05,205 beakerd ERROR Failed to commit in queued_recipes Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/bkr/server/tools/beakerd.py", line 409, in queued_recipes guestrecipe.resource.allocate() File "/usr/lib/python2.6/site-packages/bkr/server/model.py", line 6045, in allocate self.mac_address = self._lowest_free_mac() File "/usr/lib/python2.6/site-packages/bkr/server/model.py", line 5914, in _lowest_free_mac session.flush() [...] IntegrityError: (IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`beaker/recipe_resource`, CONSTRAINT `recipe_resource_recipe_id_fk` FOREIGN KEY (`recipe_id`) REFERENCES `recipe` (`id`) ON DELETE CASCADE ON UPDATE CASCADE)') 'UPDATE recipe_resource SET recipe_id=%s WHERE recipe_resource.id = %s' (None, 574512L) This will need a database fix to remove the offending rows, plus the schema upgrade notes for 0.10 should be corrected. No code changes needed.
Query to remove offending rows: DELETE FROM recipe_resource USING recipe_resource INNER JOIN recipe ON recipe_resource.recipe_id = recipe.id INNER JOIN machine_guest_map ON machine_guest_map.guest_recipe_id = recipe.id INNER JOIN recipe parent ON machine_guest_map.machine_recipe_id = parent.id WHERE recipe_resource.type = 'guest' AND parent.status IN ('New', 'Processed', 'Queued'); Patch for 0.10 schema update notes: http://gerrit.beaker-project.org/1528
Actually this is not quite complete, once the guest_resource rows are fixed up there is another problem: 2012-12-03 01:45:15,368 beakerd ERROR Failed to commit in queued_recipes Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/bkr/server/tools/beakerd.py", line 411, in queued_recipes guestrecipe.schedule() File "/usr/lib/python2.6/site-packages/bkr/server/model.py", line 4988, in schedule raise BX(_('Invalid state transition for Recipe ID %s' % self.id)) BX: 'Invalid state transition for Recipe ID 693118'
(In reply to comment #1) One extra query is needed to remove the guest_resource rows with foreign keys to recipe_resource. And another two queries are needed afterwards, to move the guest recipes back to Queued as the scheduler is now expecting. The complete sequence is: DELETE FROM guest_resource USING guest_resource INNER JOIN recipe_resource ON guest_resource.id = recipe_resource.id INNER JOIN recipe ON recipe_resource.recipe_id = recipe.id INNER JOIN machine_guest_map ON machine_guest_map.guest_recipe_id = recipe.id INNER JOIN recipe parent ON machine_guest_map.machine_recipe_id = parent.id WHERE recipe_resource.type = 'guest' AND parent.status IN ('New', 'Processed', 'Queued'); DELETE FROM recipe_resource USING recipe_resource INNER JOIN recipe ON recipe_resource.recipe_id = recipe.id INNER JOIN machine_guest_map ON machine_guest_map.guest_recipe_id = recipe.id INNER JOIN recipe parent ON machine_guest_map.machine_recipe_id = parent.id WHERE recipe_resource.type = 'guest' AND parent.status IN ('New', 'Processed', 'Queued'); DELETE FROM watchdog USING watchdog INNER JOIN recipe ON watchdog.recipe_id = recipe.id INNER JOIN machine_guest_map ON machine_guest_map.guest_recipe_id = recipe.id INNER JOIN recipe parent ON machine_guest_map.machine_recipe_id = parent.id WHERE parent.status = 'Queued' AND recipe.status != parent.status; UPDATE recipe INNER JOIN machine_guest_map ON machine_guest_map.guest_recipe_id = recipe.id INNER JOIN recipe parent ON machine_guest_map.machine_recipe_id = parent.id SET recipe.status = 'Queued' WHERE parent.status = 'Queued' AND recipe.status != parent.status;
Upgrade notes fixed in beaker-server-0.10.5-2.