Bug 1014875
| Summary: | database deadlock in XML-RPC method recipes.tasks.register_result_file | ||
|---|---|---|---|
| Product: | [Retired] Beaker | Reporter: | Dan Callaghan <dcallagh> |
| Component: | scheduler | Assignee: | Raymond Mancy <rmancy> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Nick Coghlan <ncoghlan> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 0.15 | CC: | aigao, asaha, dcallagh, ebaak, jingwang, llim, qwan, rmancy |
| Target Milestone: | 0.15.1 | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-10-23 06:57:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Dan Callaghan
2013-10-03 01:19:02 UTC
On Gerrit: http://gerrit.beaker-project.org/#/c/2325/ For future reference, a summary of what Ray and I found while looking at this deadlock today:
The deadlock is caused by the "conditional insert" statement from two different LogRecipeTaskResult.lazy_create calls running concurrently. Note that the statements themselves must be executed concurrently by MySQL, not just in concurrent transactions (which makes this unlike all the previous race conditions we've fixed).
The problem can be simulated as follows using two mysql clients. I say "simulated" because the only way to guarantee the "bad" interleaving of the two pieces of the conditional insert is to split them up into two separate queries: first SELECT 1 ... FOR UPDATE, then INSERT.
In terminal 1:
mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT 1 FROM log_recipe_task_result WHERE log_recipe_task_result.path = '/' AND log_recipe_task_result.filename = 'test.log' AND log_recipe_task_result.recipe_task_result_id = 5046 FOR UPDATE;
Empty set (0.00 sec)
In terminal 2:
mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT 1 FROM log_recipe_task_result WHERE log_recipe_task_result.path = '/' AND log_recipe_task_result.filename = 'test.log' AND log_recipe_task_result.recipe_task_result_id = 5047 FOR UPDATE;
Empty set (0.00 sec)
In terminal 1:
mysql> INSERT INTO log_recipe_task_result (recipe_task_result_id, path, filename, start_time) SELECT 5046, '/', 'test.log', '2013-10-03 01:04:00';
(Terminal 1 blocks)
In terminal 2:
mysql> INSERT INTO log_recipe_task_result (recipe_task_result_id, path, filename, start_time) SELECT 5047, '/', 'test.log', '2013-10-03 01:04:00';
Query OK, 1 row affected (2.69 sec)
In terminal 1:
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
have talked with Dan, now QE have no env to reproduce the deadlock to test the fix, so set qa_ack-. HSS-QE has reviewed and declined this request. QE for this bug will be handled by IED. Marking as VERIFIED based on the new automated regression test included with the patch: http://gerrit.beaker-project.org/#/c/2325/3/IntegrationTests/src/bkr/inttest/server/test_model.py beaker 0.15.1 has been released. |