Bug 855735
Summary: | Race between beaker-transfer, recipeset finish time, and job delete | ||
---|---|---|---|
Product: | [Retired] Beaker | Reporter: | Raymond Mancy <rmancy> |
Component: | lab controller | Assignee: | beaker-dev-list |
Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | |
Severity: | unspecified | Docs Contact: | |
Priority: | low | ||
Version: | 0.9 | CC: | bpeck, dcallagh, ebaak, rjoost, tools-bugs |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | LogStorage | ||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-05-08 03:11:52 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Raymond Mancy
2012-09-10 06:54:00 UTC
Is this fixed by the recent 0.9.3-7 hotfix? http://git.beaker-project.org/cgit/beaker/commit/?h=release-0.9.3&id=6a149822706ef40cda17edd228ba3b2be75e3c16 Yes that traceback will be, but it still won't be right. I think what will happen, is we will end up with logs on our archive server which are undeletable. This is because we can't delete the same recipe twice, and if we're hitting this problem it's because the recipe has already been deleted, but it's logs still live on disk. Ideally we should make the process more robust so this scenario cannot come about. If that's too hard, than we should at least be able to signal to the LC that this recipe is in fact deleted, and any references you have to it's logs locally should also be deleted. As it stands with that hotfix, the rsync process has already synced it to the archive server. (In reply to comment #2) > Yes that traceback will be, but it still won't be right. > > I think what will happen, is we will end up with logs on our archive server > which are undeletable. This is because we can't delete the same recipe > twice, and if we're hitting this problem it's because the recipe has already > been deleted, but it's logs still live on disk. We don't have webdav configured for our lab controllers do we? So if beaker-transfer hasn't run in a long time for some reason then log-delete could try and delete some logs from the lab controller which have expired. If webdav is configured they will be deleted from there, if not then they should still be in the DB right? We don't remove log entries for logs we haven't been able to delete right? Once they are moved from the lab controller to the archive server log-delete should be able to complete the delete via webdav. Am I missing something? > > Ideally we should make the process more robust so this scenario cannot come > about. If that's too hard, than we should at least be able to signal to the > LC that this recipe is in fact deleted, and any references you have to it's > logs locally should also be deleted. > > As it stands with that hotfix, the rsync process has already synced it to > the archive server. (In reply to comment #3) > (In reply to comment #2) > > Yes that traceback will be, but it still won't be right. > > > > I think what will happen, is we will end up with logs on our archive server > > which are undeletable. This is because we can't delete the same recipe > > twice, and if we're hitting this problem it's because the recipe has already > > been deleted, but it's logs still live on disk. > > We don't have webdav configured for our lab controllers do we? So if > beaker-transfer hasn't run in a long time for some reason then log-delete > could try and delete some logs from the lab controller which have expired. > If webdav is configured they will be deleted from there, if not then they > should still be in the DB right? We don't remove log entries for logs we > haven't been able to delete right? Once they are moved from the lab > controller to the archive server log-delete should be able to complete the > delete via webdav. > > Am I missing something? > This is all correct if webdav is configured for the LC (Which I've just checked with mgrigull, and it is). However this it kind of besides the point here, what I mean to say is this; the traceback that originates from recipes.change_files(), is called _after_ we have synced the logs to the archive server: rc = self.rsync('%s/' % tmpdir, '%s' % self.conf.get("ARCHIVE_RSYNC")) logger.debug("rsync rc=%s", rc) if rc == 0: # if the logs have been transfered then tell the server the new location self.hub.recipes.change_files(recipe_id, self.conf.get("ARCHIVE_SERVER"),self.conf.get("ARCHIVE_BASEPATH")) So this means that we've just synced the files of a deleted recipe to the archive server. This recipe will never be picked up again by a log-delete run, so those files will live on the archive server forever. This is how I read the situation, perhaps I'm missing something though? > > > > Ideally we should make the process more robust so this scenario cannot come > > about. If that's too hard, than we should at least be able to signal to the > > LC that this recipe is in fact deleted, and any references you have to it's > > logs locally should also be deleted. > > > > As it stands with that hotfix, the rsync process has already synced it to > > the archive server. (In reply to comment #4) Did the rsync actually sync any files across? Or did it just create an empty directory? Bulk reassignment of issues as Bill has moved to another team. Guess we'll never know. I'm closing this bug, since we won't be able to reproduce it with the information here. |