Bug 728584 - createrepo not cleaning up temporary files
Summary: createrepo not cleaning up temporary files
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: createrepo
Version: 20
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: Luke Macken
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 749107 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-05 17:29 UTC by Paul Howarth
Modified: 2016-09-20 02:42 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-29 11:36:49 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Patch (3.34 KB, patch)
2014-01-10 13:37 UTC, Miroslav Suchý
no flags Details | Diff

Description Paul Howarth 2011-08-05 17:29:30 UTC
Description of problem:
Createrepo leaves behind temporary directories:

 * Empty "garbageid" as a sibling of "repodata"

 * Non-empty "/var/tmp/createrepoXXXXXX/garbageid" containing repo metadata

The XXXXXX of the latter case is random so this is gradually filling up my /var filesystem.

Version-Release number of selected component (if applicable):
createrepo-0.9.9-4.fc15

Comment 1 seth vidal 2011-08-08 16:19:44 UTC
Thank you - I've figured out what I left out - setting base_persisdir in the repo setup for --update.

It's fixed in upstream with this patch.
http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=commitdiff;h=8bac55ba0417669fe87343493de5168545a5ea3a


I'll push out a new release soon.

thanks

Comment 2 Paul Howarth 2011-08-09 18:50:26 UTC
I applied the patch on top of createrepo-0.9.9-4.fc15 and it resolved the first problem but not the second one - /var/tmp is still full of temp createrepo* directories.

Comment 3 seth vidal 2011-08-10 16:04:54 UTC
the /var/tmp dirs are not a bug.

They will be cleaned up through normal tmpreaper activities.

Comment 4 Paul Howarth 2011-08-10 16:54:52 UTC
I run a lot of createrepo jobs; after a quiet day I have this:

# ls -lrtd /var/tmp/createrepo*
drwx------. 3 paul    paul    4096 Aug  9 19:47 /var/tmp/createrepox7pZLu
drwx------. 3 paul    paul    4096 Aug  9 19:47 /var/tmp/createrepo-pWI0ZZA
drwx------. 3 paul    paul    4096 Aug  9 19:47 /var/tmp/createrepo-pIVg1EJ
drwx------. 3 paul    paul    4096 Aug  9 19:47 /var/tmp/createrepo2GSgd8
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepoU0Nlg3
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepo-pW61eQX
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepo-phtVZFr
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepoEizrlI
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepo-p5DcAEb
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepo4lCOyj
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepo-pG4AgxE
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepolpRNGY
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepo-pvmB5qx
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepoDeflqe
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepoZ71Pn2
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepo-pUXMKyO
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepo-pJtOnrk
drwx------. 3 paul    paul    4096 Aug  9 19:48 /var/tmp/createrepognSJ3V
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepoYJy4RE
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepo-pDnEpXY
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepo-pU9DYS1
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepoGSILo5
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepoUUvIxg
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepo-pP0K1s_
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepo-p3aPFgr
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepo4s855D
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepo-pZPJnBm
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepofQDXWI
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepo-puYvz5X
drwx------. 3 paul    paul    4096 Aug  9 19:49 /var/tmp/createrepoAud1d_
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepo-pW8n7UX
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepo0BSBsf
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepo-plO3GIZ
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepoj_y9cF
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepoqfNKdA
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepo-p6HqFwx
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepow9bLBq
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepo-pGGSl7q
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepo-pzHX3A1
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepodNrPyM
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepoufukOO
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepo-puRB4aC
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepowi4zX6
drwx------. 3 paul    paul    4096 Aug  9 19:50 /var/tmp/createrepo-p4TPZoM
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo-pzW90Am
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepob2mlUr
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo-pwajAmC
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepok97n4x
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo-p7HYDN2
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepoiACG5m
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo-pX2k7Uk
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo59vNa1
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepoTn5GCW
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo-pgg7DIb
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo-pCmYNcj
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo0CX_3s
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepopVnYU4
drwx------. 3 paul    paul    4096 Aug  9 19:51 /var/tmp/createrepo-pGLT9JM
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepoPXFZvt
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepo-pmL8X7D
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepo-pMNTIrG
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepoB7FQt_
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepoqf516p
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepo-pMUxhBs
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepo-pYpUDEv
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepo96R8Hw
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepo-pV4N_Es
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepomvlHpL
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepo-pQYP9Mt
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepoatVaT_
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepoR9ap1C
drwx------. 3 paul    paul    4096 Aug  9 19:52 /var/tmp/createrepo-pvV35pN
drwx------. 3 paul    paul    4096 Aug  9 19:53 /var/tmp/createrepo-pfPkHIu
drwx------. 3 paul    paul    4096 Aug  9 19:53 /var/tmp/createrepooCHnu1
drwx------. 3 paul    paul    4096 Aug  9 19:53 /var/tmp/createrepo-pUnWM6R
drwx------. 3 paul    paul    4096 Aug  9 19:53 /var/tmp/createrepo6jXapY
drwx------. 3 paul    paul    4096 Aug  9 19:53 /var/tmp/createrepowfp8q6
drwx------. 3 paul    paul    4096 Aug  9 19:53 /var/tmp/createrepo-pqGF1xc
drwx------. 3 paul    paul    4096 Aug  9 19:53 /var/tmp/createrepo-pNtB7Rl
drwx------. 3 paul    paul    4096 Aug  9 19:53 /var/tmp/createrepo1NbC2f
drwx------. 3 softlib softlib 4096 Aug 10 02:31 /var/tmp/createrepo-pcyX62q
drwx------. 3 softlib softlib 4096 Aug 10 02:31 /var/tmp/createrepolc1Og9
drwx------. 3 softlib softlib 4096 Aug 10 02:32 /var/tmp/createrepo-p2wQ4ij
drwx------. 3 softlib softlib 4096 Aug 10 02:32 /var/tmp/createrepoGywb5f
drwx------. 3 softlib softlib 4096 Aug 10 02:33 /var/tmp/createrepo-p1Y15uu
drwx------. 3 softlib softlib 4096 Aug 10 02:33 /var/tmp/createrepo03e1VR
drwx------. 3 softlib softlib 4096 Aug 10 02:35 /var/tmp/createrepo-pLQIT51
drwx------. 3 softlib softlib 4096 Aug 10 02:35 /var/tmp/createrepoky9NsB
drwx------. 3 softlib softlib 4096 Aug 10 04:46 /var/tmp/createrepobwEqJY
drwx------. 3 softlib softlib 4096 Aug 10 04:46 /var/tmp/createrepo-pUMMgMR
drwx------. 3 softlib softlib 4096 Aug 10 04:48 /var/tmp/createreposidWWD
drwx------. 3 softlib softlib 4096 Aug 10 04:48 /var/tmp/createrepo-pxQBHUb
drwx------. 3 softlib softlib 4096 Aug 10 04:50 /var/tmp/createrepoX0rw9D
drwx------. 3 softlib softlib 4096 Aug 10 04:50 /var/tmp/createrepo-pOoGGtS
drwx------. 3 softlib softlib 4096 Aug 10 04:51 /var/tmp/createrepoyKOWmL
drwx------. 3 softlib softlib 4096 Aug 10 04:51 /var/tmp/createrepo-p932M1_

# du -ks /var/tmp/createrepo* | sort -n | tail 
3780	/var/tmp/createrepow9bLBq
3844	/var/tmp/createrepob2mlUr
7288	/var/tmp/createrepoGywb5f
7312	/var/tmp/createrepoky9NsB
9944	/var/tmp/createrepo03e1VR
9944	/var/tmp/createrepolc1Og9
16604	/var/tmp/createreposidWWD
18776	/var/tmp/createrepobwEqJY
20656	/var/tmp/createrepoyKOWmL
23568	/var/tmp/createrepoX0rw9D

After a few busy days this is going to fill up /var before tmpwatch gets round to doing something about it, which is why I noticed the problem in the first place. Why can't createrepo just clean up after itself anyway?

Comment 5 seth vidal 2011-08-10 18:42:44 UTC
fair enough - I've added a cleanup function for these tmpdirs upstream

however, could I encourage you to run your tmp reaper more often? :)



patch:
http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=commitdiff;h=850b368ced63b2c0b7f2a4f2fb014d1daba7c80c

Comment 6 Paul Howarth 2011-08-10 21:44:16 UTC
Thanks, that's much better now :-)

Comment 7 Von Fugal 2011-09-15 15:02:39 UTC
Is that patch going to be in a release soon or do we have to download and patch our own systems?

Comment 8 seth vidal 2011-10-04 17:00:21 UTC
It should be in f16.

thanks

Comment 9 Zdeněk Pavlas 2011-10-26 10:15:06 UTC
*** Bug 749107 has been marked as a duplicate of this bug. ***

Comment 10 A. Mani 2012-02-19 02:29:59 UTC
No wonder. Even yum does not clean up the rpms it downloads to /var/lib/yum. 
X86-64 F15

Comment 11 Miroslav Suchý 2014-01-08 14:34:11 UTC
Reopening.
This is still happen on 
  createrepo-0.10-1.fc19.noarch

It is problem for my on Copr server where we run creatrepo quite often.
/var/tmp/createrepo* have size of 1.5 GB there.

Deleting those file more often is no option for me. I would have to delete it every 5 minutes and I can delete files of some currently running createrepo.

Can you please delete those files. Or if you hesitate to do it - at least please implement something like --clean-after-myself.

Comment 12 Miroslav Suchý 2014-01-10 08:36:03 UTC
Raising priority. Copr had run out of disk space again due this behaviour. I will have to allocate some storage for /var/tmp just because of those temporary files.

Comment 13 Miroslav Suchý 2014-01-10 13:37:07 UTC
Created attachment 848204 [details]
Patch

This *should* fix the problem. However I did not test it - because I discovered createrepo_c and I will try to use that one.

Comment 14 Luke Macken 2014-01-10 21:36:55 UTC
(In reply to Miroslav Suchý from comment #13)
> Created attachment 848204 [details]
> Patch
> 
> This *should* fix the problem. However I did not test it - because I
> discovered createrepo_c and I will try to use that one.

The patch looks good at a first glance. I'm interested to hear if createrepo_c works out for you, I've been thinking about testing it with mash + bodhi at some point soon.

Comment 15 Miroslav Suchý 2014-01-15 08:19:34 UTC
After two days with createrepo_c, I do not see any residual temporary file.

Lowering priority as createrepo_c is very good workaround :)

Comment 16 Fedora End Of Life 2015-05-29 08:39:47 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 17 Fedora End Of Life 2015-06-29 11:36:49 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.