Bug 1411402

Summary: koji-gc leaving behind dangling symlinks
Product: [Fedora] Fedora Reporter: John Florian <trailtotale>
Component: kojiAssignee: Mike McLean <mikem>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 25CC: dennis, mikem
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-01 18:00:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description John Florian 2017-01-09 16:05:47 UTC
Description of problem:
I am find lots of broken symlinks like this one example:

> $ ls -l /mnt/koji/work/tasks/9588/19588/root.log
> lrwxrwxrwx. 1 apache apache 84 May 12  2016 /mnt/koji/work/tasks/9588/19588/root.log -> /mnt/koji/packages/python3-mdct/5.3.0/1.git.2.f7e5fbe.fc23/data/logs/noarch/root.log
> $ ls -l /mnt/koji/packages/python3-mdct/5.3.0/1.git.2.f7e5fbe.fc23/data/logs/noarch/root.log
> /bin/ls: cannot access /mnt/koji/packages/python3-mdct/5.3.0/1.git.2.f7e5fbe.fc23/data/logs/noarch/root.log: No such file or directory

In all cases, the broken symlink exists somewhere in /mnt/koji/work/tasks.


Version-Release number of selected component (if applicable):
koji-utils-1.11.0-1.fc25.noarch.rpm

How reproducible:
seems very consistent after pruning

Steps to Reproduce:

My complete GC policy is:

> [prune]
> policy =
>     age < 1 day :: skip
>     tag *-testing && age < 4 weeks :: keep
>     order > 6:: untag


Actual results:
Over time, lots of broken links.  These can be easily seen with:

find /mnt/koji/work/tasks/ -xtype l

Expected results:

I guess I would expect Koji to look after itself better and not leave such cruft behind, but I also realize it's mostly harmless so perhaps this is just expected behavior.  I don't know.

Additional info:

Comment 1 Mike McLean 2017-01-09 21:18:50 UTC
At the moment the gc does not deal with the contents under work/tasks. The work directory is typically cleaned up by by an instance specific cron job.

Comment 2 John Florian 2017-01-09 21:28:49 UTC
(In reply to Mike McLean from comment #1)
> At the moment the gc does not deal with the contents under work/tasks. The
> work directory is typically cleaned up by by an instance specific cron job.

Any recommendations as to what that cron job might look like?  I'm guessing lots of the old work/tasks could be removed but maybe there's reason to keep them around ... and I certainly don't want to interfere with what Koji needs.

Comment 3 Mike McLean 2017-01-09 21:36:41 UTC
Content under work is expected to be transient, but it's useful for it to stick around for a short time for debugging failed tasks and such.

I was trying to find the link to the script that Fedora infra uses, but failing. 

Something like....

TOPDIR=/mnt/koji
TIMEARG="-mtime +14"

# for tasks, try to remove as a unit
for x in $(find "$TOPDIR"/work/tasks/ -mindepth 2 -maxdepth 2 -type d $TIMEARG); do
    find "$x" -xdev '!' -type d -print0 | xargs -0 -r rm -f
    find "$x" -xdev -depth -type d -print0 | xargs -0 -r rmdir
done

# for anything else, just remove old stuff
# but don't remove the top level dirs (e.g. cli-build)
for x in $(find "$TOPDIR"/work -maxdepth 1 -mindepth 1 \! -name tasks); do
    find "$x" -xdev '!' -type d $TIMEARG -print0 | xargs -0 -r rm -f
    find "$x" -xdev -depth -mindepth 1 -type d -empty -print0 | xargs -0 -r rmdir
done

...salt to taste

Comment 4 John Florian 2017-01-09 21:38:45 UTC
Thanks Mike, that's exactly what I was looking for.

Comment 5 Fedora End Of Life 2017-11-16 19:33:06 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.