Bug 1012783

Summary:	beaker-sync-tasks should use a staging area when downloading files
Product:	[Retired] Beaker	Reporter:	Nick Coghlan <ncoghlan>
Component:	general	Assignee:	Raymond Mancy <rmancy>
Status:	CLOSED CURRENTRELEASE	QA Contact:	tools-bugs <tools-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	0.14	CC:	aigao, asaha, dcallagh, ebaak, llim, qwan, rmancy, xjia
Target Milestone:	0.15.2
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-12-19 07:04:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nick Coghlan 2013-09-27 06:41:01 UTC

The beaker-sync-tasks utility currently downloads files directly into the main Beaker RPMs directory, bypassing the file lock normally used to protect changes to the task library.

Instead, downloads should be transferred to a staging area and the copied over to the main directory using TaskLibrary.update_task(). (This is already called, but using it on a file that is already in that directory means it doesn't properly synchronise with the harness repo creation process)

This can be verified based on automated tests and code review - no need to contrive a complicated manual test scenario.

Comment 2 Raymond Mancy 2013-12-02 04:48:43 UTC

Just so that I'm clear about this, at least one of the problems is the following(?):

  TaskA is uploaded
  TaskB is uploaded
  TaskB gets flock and updates repo (with TaskB and TaskA)
  TaskA fails for unknown reason and is rolled back # Except that the repo has already been updated with TaskA!

Are there any problems with passing '--includepkg' to createrepo to make sure that each Task upload is only updating the repo for that particular task?

Comment 3 Nick Coghlan 2013-12-02 05:39:44 UTC

No, task upload is fine (that was a previous bug, though). The problem is that the background sync task is dumping random crap in the task library directory without holding the flock and it's having weird side effects, since the repo management assumes that nothing will be added to the directory without holding the flock first.

Comment 4 Nick Coghlan 2013-12-02 05:41:57 UTC

If you're worried about testing for this, I wouldn't worry - just fix it, and make sure it's holding the flock whenever it is modifying the task library.

Since syncs may be rather large, I also suggest it may be appropriate to do the library updates in batches of 100 or so tasks rather than waiting until everything has been downloaded.

Comment 5 Raymond Mancy 2013-12-03 02:42:22 UTC

http://gerrit.beaker-project.org/#/c/2578/

Comment 8 Amit Saha 2013-12-19 07:04:19 UTC

Release announcement: https://lists.fedorahosted.org/pipermail/beaker-devel/2013-December/000923.html