Bug 1402861 - Cleanup of dist-git and results
Summary: Cleanup of dist-git and results
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Copr
Classification: Community
Component: backend
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miroslav Suchý
QA Contact:
URL:
Whiteboard:
Depends On: 1427431
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-08 13:38 UTC by Thomas Moschny
Modified: 2019-07-26 09:55 UTC (History)
1 user (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2019-07-26 09:54:27 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1427431 0 unspecified CLOSED RFE: dist-git: policy for garbage collecting of lookaside cache tarballs 2023-04-03 18:45:02 UTC

Internal Links: 1427431

Description Thomas Moschny 2016-12-08 13:38:55 UTC
When I delete a project, in some cases it seems the dist-git and the result directories remain.

Looking at http://copr-dist-git.fedorainfracloud.org/cgit/thm/, I see repositories for projects I already deleted. After creating a project with the same name, dist-git carries a mixture of old and new project files.

For http://copr-dist-git.fedorainfracloud.org/cgit/thm/, the situation is even worse: The directories there are deleted asynchronously it seems. So, if I delete a project, re-create it with the same name, build something, and later have a look at the results directory, the garbage collection seems to have kicked in in between, and the *new* builds are gone :(

Comment 1 clime 2017-01-04 14:42:06 UTC
(In reply to Thomas Moschny from comment #0)
> When I delete a project, in some cases it seems the dist-git and the result
> directories remain.

Dist-git data always remain, there is no deletion of them at the moment.

> 
> Looking at http://copr-dist-git.fedorainfracloud.org/cgit/thm/, I see
> repositories for projects I already deleted. After creating a project with
> the same name, dist-git carries a mixture of old and new project files.
> 
> For http://copr-dist-git.fedorainfracloud.org/cgit/thm/, the situation is
> even worse: The directories there are deleted asynchronously it seems. So,
> if I delete a project, re-create it with the same name, build something, and
> later have a look at the results directory, the garbage collection seems to
> have kicked in in between, and the *new* builds are gone :(

Is the link http://copr-dist-git.fedorainfracloud.org/cgit/thm/ correct? Note that it is the same link as in the first paragraph. Anyway, there is no garbage collection in place, so data (srpm + spec) from the latest builds should be present. Can you recheck, please?

Comment 2 Thomas Moschny 2017-01-11 09:33:55 UTC
(In reply to clime from comment #1)
> (In reply to Thomas Moschny from comment #0)
> Dist-git data always remain, there is no deletion of them at the moment.

Is this planned? If not, consider this an RFE ;)

> > For http://copr-dist-git.fedorainfracloud.org/cgit/thm/, the situation is
> > even worse: The directories there are deleted asynchronously it seems. So,
> > if I delete a project, re-create it with the same name, build something, and
> > later have a look at the results directory, the garbage collection seems to
> > have kicked in in between, and the *new* builds are gone :(
> 
> Is the link http://copr-dist-git.fedorainfracloud.org/cgit/thm/ correct?
> Note that it is the same link as in the first paragraph. Anyway, there is no
> garbage collection in place, so data (srpm + spec) from the latest builds
> should be present. Can you recheck, please?

Copy and paste error, sorry. It should have read
https://copr-be.cloud.fedoraproject.org/results/thm/
This has been cleaned up in the meantime.

Still it looks to me as there is a significant delay between deleting a project from the web GUI and the time when that result directory gets cleaned, leading to new builds being deleted.

From memory, here is what happened:

1. I deleted the shotwell project
2. I recreated the shotwell project
3. Made some builds within this project
4. The result dir got cleaned (some hours later?)
5. Builds from (3.) are missing

Comment 3 clime 2017-01-11 13:17:09 UTC
> Copy and paste error, sorry. It should have read
> https://copr-be.cloud.fedoraproject.org/results/thm/
> This has been cleaned up in the meantime.
> 
> Still it looks to me as there is a significant delay between deleting a
> project from the web GUI and the time when that result directory gets
> cleaned, leading to new builds being deleted.
> 
> From memory, here is what happened:
> 
> 1. I deleted the shotwell project
> 2. I recreated the shotwell project
> 3. Made some builds within this project
> 4. The result dir got cleaned (some hours later?)
> 5. Builds from (3.) are missing

Oh, this scenario is possible if action queue gets stuck for some reason and there was (still is) a problem with cancelling job action that could cause the jamming up so that your deletion-action was delayed. This is an interesting problem.

Comment 5 Pavel Raiskup 2019-07-26 09:54:27 UTC
Dist-git GC topic is moved to bug 1427431.

The backend (results) part should be fixed nowadays, I hope.  Please reopen
if that isn't the case.


Note You need to log in before you can comment on or make changes to this bug.