Bug 1411402 - koji-gc leaving behind dangling symlinks
Summary: koji-gc leaving behind dangling symlinks
Alias: None
Product: Fedora
Classification: Fedora
Component: koji
Version: 25
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Mike McLean
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2017-01-09 16:05 UTC by John Florian
Modified: 2017-12-01 18:00 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-12-01 18:00:25 UTC
Type: Bug

Attachments (Terms of Use)

Description John Florian 2017-01-09 16:05:47 UTC
Description of problem:
I am find lots of broken symlinks like this one example:

> $ ls -l /mnt/koji/work/tasks/9588/19588/root.log
> lrwxrwxrwx. 1 apache apache 84 May 12  2016 /mnt/koji/work/tasks/9588/19588/root.log -> /mnt/koji/packages/python3-mdct/5.3.0/1.git.2.f7e5fbe.fc23/data/logs/noarch/root.log
> $ ls -l /mnt/koji/packages/python3-mdct/5.3.0/1.git.2.f7e5fbe.fc23/data/logs/noarch/root.log
> /bin/ls: cannot access /mnt/koji/packages/python3-mdct/5.3.0/1.git.2.f7e5fbe.fc23/data/logs/noarch/root.log: No such file or directory

In all cases, the broken symlink exists somewhere in /mnt/koji/work/tasks.

Version-Release number of selected component (if applicable):

How reproducible:
seems very consistent after pruning

Steps to Reproduce:

My complete GC policy is:

> [prune]
> policy =
>     age < 1 day :: skip
>     tag *-testing && age < 4 weeks :: keep
>     order > 6:: untag

Actual results:
Over time, lots of broken links.  These can be easily seen with:

find /mnt/koji/work/tasks/ -xtype l

Expected results:

I guess I would expect Koji to look after itself better and not leave such cruft behind, but I also realize it's mostly harmless so perhaps this is just expected behavior.  I don't know.

Additional info:

Comment 1 Mike McLean 2017-01-09 21:18:50 UTC
At the moment the gc does not deal with the contents under work/tasks. The work directory is typically cleaned up by by an instance specific cron job.

Comment 2 John Florian 2017-01-09 21:28:49 UTC
(In reply to Mike McLean from comment #1)
> At the moment the gc does not deal with the contents under work/tasks. The
> work directory is typically cleaned up by by an instance specific cron job.

Any recommendations as to what that cron job might look like?  I'm guessing lots of the old work/tasks could be removed but maybe there's reason to keep them around ... and I certainly don't want to interfere with what Koji needs.

Comment 3 Mike McLean 2017-01-09 21:36:41 UTC
Content under work is expected to be transient, but it's useful for it to stick around for a short time for debugging failed tasks and such.

I was trying to find the link to the script that Fedora infra uses, but failing. 

Something like....

TIMEARG="-mtime +14"

# for tasks, try to remove as a unit
for x in $(find "$TOPDIR"/work/tasks/ -mindepth 2 -maxdepth 2 -type d $TIMEARG); do
    find "$x" -xdev '!' -type d -print0 | xargs -0 -r rm -f
    find "$x" -xdev -depth -type d -print0 | xargs -0 -r rmdir

# for anything else, just remove old stuff
# but don't remove the top level dirs (e.g. cli-build)
for x in $(find "$TOPDIR"/work -maxdepth 1 -mindepth 1 \! -name tasks); do
    find "$x" -xdev '!' -type d $TIMEARG -print0 | xargs -0 -r rm -f
    find "$x" -xdev -depth -mindepth 1 -type d -empty -print0 | xargs -0 -r rmdir

...salt to taste

Comment 4 John Florian 2017-01-09 21:38:45 UTC
Thanks Mike, that's exactly what I was looking for.

Comment 5 Fedora End Of Life 2017-11-16 19:33:06 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Note You need to log in before you can comment on or make changes to this bug.