This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 811978 - abrt does not clean its cache
abrt does not clean its cache
Status: NEW
Product: Fedora
Classification: Fedora
Component: abrt (Show other bugs)
25
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Matej Habrnal
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-12 08:57 EDT by Michal Hlavinka
Modified: 2017-09-11 02:06 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-19 06:05:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Michal Hlavinka 2012-04-12 08:57:13 EDT
Description of problem:
I don't reinstall my machine, I update it using yum. I've found out, there is
not enough space left on / partition and where looking around, I've found
/var/cache/abrt-di to be eating 1.9 GB data. I did not generate backtrace locally since updating to F17 a few months ago. Most data (maybe all) actually present in abrt-di cache directory are useless - it contains data for packages that are not installed on my machine for a long time.

Version-Release number of selected component (if applicable):
abrt-2.0.10-1.fc17.x86_64
  
Actual results:
abrt does not clean its cache directory

Expected results:
abrt should periodically clean its cache removing data for packages that are no longer installed
Comment 1 Bengt Lüers 2013-01-26 14:42:58 EST
I can confirm this issue. I have use abrt a lot with F17 and found my .cache/abrt to weigh in at 6.1 GB. This is on a SSD and therefore waste of precious disk space. The biggest folder is .cache/abrt/spool.
Comment 2 collura 2013-06-10 01:26:06 EDT
this still seems to be around in fc18:

   $ du --summarize /var/cache/abrt-di/
   4520604	/var/cache/abrt-di/

the /var/cache/abrt-di at about 4.3GB was the big one here but the .cache/abrt seemed pretty upto date and empty.

currently running abrt-2.1.4-3.fc18.x86_64 with updates and updates-testing
Comment 3 Fedora End Of Life 2013-07-03 15:15:48 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 4 Fedora End Of Life 2013-07-31 13:32:48 EDT
Fedora 17 changed to end-of-life (EOL) status on 2013-07-30. Fedora 17 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 5 Fedora End Of Life 2013-12-21 10:01:45 EST
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 6 Brian J. Murrell 2014-02-03 11:52:29 EST
Can we update the version on this ticket since I am seeing this on F19 still.
Comment 7 Frank Büttner 2014-08-13 11:31:16 EDT
I my case:
du -sh /var/cache/abrt-di
6,7G	/var/cache/abrt-di
Can I safe remove the folder?
Comment 8 Jakub Filak 2014-08-13 11:50:39 EDT
(In reply to Frank Büttner from comment #7)
Yes, you can. The folder contains only unpacked debuginfos.
Comment 9 Brian J. Murrell 2014-08-13 12:06:23 EDT
I'm more interested in this "cache" being maintained more properly, aging out older, unused files to make room for new ones rather than just having abrt give up because "it's full".

I thought that is what was supposed to be happening.  Am I misunderstanding?
Comment 10 Fedora End Of Life 2015-05-29 04:43:28 EDT
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 11 Brian J. Murrell 2015-05-29 06:33:48 EDT
Can we update the version on this ticket since I am seeing this on F21 still?

And can someone please answer the question in comment #9?
Comment 12 Fedora End Of Life 2015-11-04 10:32:08 EST
This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 13 Brian J. Murrell 2015-11-04 10:34:30 EST
Can we update the version on this ticket since I am seeing this on F22 still?

And can someone please answer the question in comment #9?
Comment 14 Frank Büttner 2015-11-04 11:22:43 EST
Testing on F22 impossible until until #1277895 is fixed.
Comment 15 Brian J. Murrell 2015-11-04 11:27:00 EST
So can we set the blocking ticket field and push the Version: to 22 until this has been tested on F22 given that the ticket is in danger of being EOL'd before it can be tested?
Comment 16 Jakub Filak 2016-02-08 11:52:22 EST
The cache is maintained by abrt-action-install-debuginfo and its algorithm is the following:
  1) remove the oldest and the biggest files from the cache to trim the cache's size to 4GiB
  2) install all required debuginfo packages


I would start with a proper configuration for systemd-tmpfiles:

$ cat /usr/lib/tmpfiles.d/abrt-action-install-debuginfo.conf
# Remove data older than 30 days from the cache
R /var/cache/abrt-di/* - - - 30d
Comment 17 Brian J. Murrell 2016-02-17 14:52:48 EST
(In reply to Jakub Filak from comment #16)
> The cache is maintained by abrt-action-install-debuginfo and its algorithm
> is the following:
>   1) remove the oldest and the biggest files from the cache to trim the
> cache's size to 4GiB

Is that 4GiB free space or total use?  What if I give /var/cache/abrt-di a 100GB filesystem to increase the effectiveness of the cache?

>   2) install all required debuginfo packages

If that 4GiB is the amount of space the cache is trimmed to have free, what if the debuginfo packages needed for a particular bug report is more than 4GiB?  Maybe 4GiB is just ridiculously high enough that no amount of debuginfo packages for a given bug report would be that big?
 
> I would start with a proper configuration for systemd-tmpfiles:
> 
> $ cat /usr/lib/tmpfiles.d/abrt-action-install-debuginfo.conf

Is this file supposed to be in an RPM somewhere or are you counselling that people should create this file manually?

> # Remove data older than 30 days from the cache
> R /var/cache/abrt-di/* - - - 30d

I assume the R means remove, the 30d means files older than 30 days.  Is that the timestamp of the file as it is in the debuginfo RPM or 30 days from the time the file is installed (i.e. is it mtime and is a file's mtime the time from the RPM or the time the file was created?)?
Comment 18 Jakub Filak 2016-02-18 01:29:51 EST
(In reply to Brian J. Murrell from comment #17)
> (In reply to Jakub Filak from comment #16)
> > The cache is maintained by abrt-action-install-debuginfo and its algorithm
> > is the following:
> >   1) remove the oldest and the biggest files from the cache to trim the
> > cache's size to 4GiB
> 
> Is that 4GiB free space or total use?

Total use. The true is that the cache can be much larger than 4GiB after all required packages are installed. ABRT trims the cached and then installs new packages.

> What if I give /var/cache/abrt-di a 100GB filesystem to increase the effectiveness of the cache?

The cache size is not configurable for regular users because only root can abrt-action-install-debuginfo. Regular users use a set-uid wrapper /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache which runs abrt-action-install-debuginfo as root with the cache size argument set to 4GiB.

With larger cache you will probably need less downloading of debuginfo packages. It depends on the system you use. A higher frequency of package updates makes the cache less valuable.


> 
> >   2) install all required debuginfo packages
> 
> If that 4GiB is the amount of space the cache is trimmed to have free, what
> if the debuginfo packages needed for a particular bug report is more than
> 4GiB?  Maybe 4GiB is just ridiculously high enough that no amount of
> debuginfo packages for a given bug report would be that big?

Looks like that something has been lost in translation. /var/cache/abrt-di is a regular directory where the dowload debuginfo rpm packages are unpacked. The cache size equals amount of unpacked packages.

>  
> > I would start with a proper configuration for systemd-tmpfiles:
> > 
> > $ cat /usr/lib/tmpfiles.d/abrt-action-install-debuginfo.conf
> 
> Is this file supposed to be in an RPM somewhere or are you counselling that
> people should create this file manually?

This was just a note for me and Matej and everybody else willing to work on this bug. The file should be first created in the ABRT upstream [1][2] and we will include it in the abrt package then.

> 
> > # Remove data older than 30 days from the cache
> > R /var/cache/abrt-di/* - - - 30d
> 
> I assume the R means remove, the 30d means files older than 30 days.

Yes, you can find the specification of tmpfiles.d in man 5 tmpfiles.d (it is a systemd thingy).

>  Is that the timestamp of the file as it is in the debuginfo RPM or 30 days from
> the time the file is installed (i.e. is it mtime and is a file's mtime the
> time from the RPM or the time the file was created?)?

The time the file was created.



1: https://github.com/abrt/abrt
2: https://github.com/abrt/abrt/tree/master/init-scripts
Comment 19 Brian J. Murrell 2016-02-18 07:32:27 EST
(In reply to Jakub Filak from comment #18)
> 
> Total use. The true is that the cache can be much larger than 4GiB after all
> required packages are installed. ABRT trims the cached and then installs new
> packages.

So it is potentially trimming stuff that it will just want to re-install because it trimmed my (full) 100GiB cache down to (only) 4GiB (used, i.e. 96% free) before starting, right?

> 
> > What if I give /var/cache/abrt-di a 100GB filesystem to increase the effectiveness of the cache?
> 
> The cache size is not configurable for regular users because only root can
> abrt-action-install-debuginfo. Regular users use a set-uid wrapper
> /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache which runs
> abrt-action-install-debuginfo as root with the cache size argument set to
> 4GiB.

Why does this 4GiB limit seem to be rife throughout ABRT?

This "cache" does not sound very much like a cache at all.  Caches are maintained to always be full, with the most recent/most used artefacts.  Only the oldest/least used objects in a cache should be removed to make room in the cache for newer/more frequently used objects.

> With larger cache you will probably need less downloading of debuginfo
> packages.

But if you are always pruning the cache down to 4GiB of used space that does not seem to be very true.

> A higher frequency of package
> updates makes the cache less valuable.

Of course.

> Looks like that something has been lost in translation.

Perhaps.  Likely even.


> /var/cache/abrt-di
> is a regular directory where the dowload debuginfo rpm packages are
> unpacked.

Yes, understood.

> The cache size equals amount of unpacked packages.

Agreed.  What is not clear is what this 4GiB that is continually mentioned really means.  Maybe some clarification around that is needed.  Maybe you need to further explain (or clarify) this statement from Comment 16:

  1) remove the oldest and the biggest files from the cache to trim the cache's size to 4GiB

> This was just a note for me and Matej and everybody else willing to work on
> this bug. The file should be first created in the ABRT upstream [1][2] and
> we will include it in the abrt package then.

OK.  But to be clear, I question it's efficacy below...

> Yes, you can find the specification of tmpfiles.d in man 5 tmpfiles.d (it is
> a systemd thingy).

But a hard limit of 30 days does not seem like an effective cache management technique.  It is only going to work really effectively for people with a specific, given size of cache.  It will fail for people with really large caches by pruning more than is needed to prune and will fail for people with really small caches by not pruning enough.

Cache management is more properly done as a function of the capacity of the cache, it's "fullness" (and generally the frequency of accesses to an object but that is perhaps less useful here).  Caches are most useful when they are full and their usefulness drops as pruning gets too aggressive to effectively empty the cache.
 
> The time the file was created.

So the time the file was unpacked from the debuginfo RPM, yes?
Comment 20 Brian J. Murrell 2016-02-25 06:59:37 EST
Any response to the above comment which got no response?
Comment 21 Jakub Filak 2016-02-25 07:56:12 EST
(In reply to Brian J. Murrell from comment #19)
> (In reply to Jakub Filak from comment #18)
> > 
> > Total use. The true is that the cache can be much larger than 4GiB after all
> > required packages are installed. ABRT trims the cached and then installs new
> > packages.
> 
> So it is potentially trimming stuff that it will just want to re-install
> because it trimmed my (full) 100GiB cache down to (only) 4GiB (used, i.e.
> 96% free) before starting, right?
> 

Probably yes.

> > 
> > > What if I give /var/cache/abrt-di a 100GB filesystem to increase the effectiveness of the cache?
> > 
> > The cache size is not configurable for regular users because only root can
> > abrt-action-install-debuginfo. Regular users use a set-uid wrapper
> > /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache which runs
> > abrt-action-install-debuginfo as root with the cache size argument set to
> > 4GiB.
> 
> Why does this 4GiB limit seem to be rife throughout ABRT?

Sorry, I can't tell. These commits introduced it:
https://github.com/abrt/abrt/commit/66fdda782198bd1c7b1c110d1337cd30a31206d9
https://github.com/abrt/abrt/commit/67a3602e83af42f932e0583d7385d9bcf7ea7e16

> 
> This "cache" does not sound very much like a cache at all.  Caches are
> maintained to always be full, with the most recent/most used artefacts. 
> Only the oldest/least used objects in a cache should be removed to make room
> in the cache for newer/more frequently used objects.
> 
> > With larger cache you will probably need less downloading of debuginfo
> > packages.
> 
> But if you are always pruning the cache down to 4GiB of used space that does
> not seem to be very true.
> 
> > A higher frequency of package
> > updates makes the cache less valuable.
> 
> Of course.
> 
> > Looks like that something has been lost in translation.
> 
> Perhaps.  Likely even.
> 
> 
> > /var/cache/abrt-di
> > is a regular directory where the dowload debuginfo rpm packages are
> > unpacked.
> 
> Yes, understood.
> 
> > The cache size equals amount of unpacked packages.
> 
> Agreed.  What is not clear is what this 4GiB that is continually mentioned
> really means.  Maybe some clarification around that is needed.  Maybe you
> need to further explain (or clarify) this statement from Comment 16:
> 
>   1) remove the oldest and the biggest files from the cache to trim the
> cache's size to 4GiB

Before trimming:
# du -sh /var/cache/abrt-di
5.3G    /var/cache/abrt-di

After trimming:
# du -sh /var/cache/abrt-di
3.9G    /var/cache/abrt-di

> 
> > This was just a note for me and Matej and everybody else willing to work on
> > this bug. The file should be first created in the ABRT upstream [1][2] and
> > we will include it in the abrt package then.
> 
> OK.  But to be clear, I question it's efficacy below...
> 
> > Yes, you can find the specification of tmpfiles.d in man 5 tmpfiles.d (it is
> > a systemd thingy).
> 
> But a hard limit of 30 days does not seem like an effective cache management
> technique.  It is only going to work really effectively for people with a
> specific, given size of cache.  It will fail for people with really large
> caches by pruning more than is needed to prune and will fail for people with
> really small caches by not pruning enough.
> 
> Cache management is more properly done as a function of the capacity of the
> cache, it's "fullness" (and generally the frequency of accesses to an object
> but that is perhaps less useful here).  Caches are most useful when they are
> full and their usefulness drops as pruning gets too aggressive to
> effectively empty the cache.

OK. We do not have capacity to implement something really sophisticated and I proposed the tmpfiles.d configuration just as a first version. Anyone can come up with something better and I am sure upstream will be happy to review patches.

>  
> > The time the file was created.
> 
> So the time the file was unpacked from the debuginfo RPM, yes?

Yes.
Comment 22 Brian J. Murrell 2016-02-25 09:19:50 EST
(In reply to Jakub Filak from comment #21)
> 
> Probably yes.

That's not so good.  Throwing away 96% of a cache just to hit some arbitrary hard high-water mark.
 
> Before trimming:
> # du -sh /var/cache/abrt-di
> 5.3G    /var/cache/abrt-di
> 
> After trimming:
> # du -sh /var/cache/abrt-di
> 3.9G    /var/cache/abrt-di

So that's a use-case that looks somewhat reasonable.  What happens if before trimming is more like 50GB or 100GB?
Comment 23 Jakub Filak 2016-03-01 01:41:38 EST
(In reply to Brian J. Murrell from comment #22)
> (In reply to Jakub Filak from comment #21)
> > Before trimming:
> > # du -sh /var/cache/abrt-di
> > 5.3G    /var/cache/abrt-di
> > 
> > After trimming:
> > # du -sh /var/cache/abrt-di
> > 3.9G    /var/cache/abrt-di
> 
> So that's a use-case that looks somewhat reasonable.  What happens if before
> trimming is more like 50GB or 100GB?

The tool that manages the cache would be removing files until "du -sh /var/cache/abrt-di" returns a number lower or eaqual to 4.0 GiB.
Comment 24 Brian J. Murrell 2016-03-01 07:05:51 EST
(In reply to Jakub Filak from comment #23)
> The tool that manages the cache would be removing files until "du -sh
> /var/cache/abrt-di" returns a number lower or eaqual to 4.0 GiB.

Ahh.  The 4GB being a *free* space value, not space consumed.  That is entirely (as in completely contradictory to) different than what was said in comment #18 which was that the 4GB is "Total use.".

Pruning the cache to a low water mark is great.

So, which tool is it that is managing the cache?  How do I run it manually when I need to urgently (as in I need to push an ABRT report and don't have the space to fetch all of the RPMs needed to generate the backtrace) reduce the cache to 4GB of free space?
Comment 25 Fedora End Of Life 2016-07-19 06:05:27 EDT
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 26 Jakub Filak 2016-07-25 05:01:41 EDT
(In reply to Brian J. Murrell from comment #24)
> So, which tool is it that is managing the cache?

The /usr/bin/abrt-action-install-debuginfo utility downloads and unpacks debuginfo packages to the /var/cache/abrt-di directory. The utility also runs the /usr/bin/abrt-action-trim-files binary to prune the cache directory.

> How do I run it manually
> when I need to urgently (as in I need to push an ABRT report and don't have
> the space to fetch all of the RPMs needed to generate the backtrace) reduce
> the cache to 4GB of free space?

The abrt-action-install-debuginfo utility runs something like this:
$ sudo abrt-action-trim-files -f 4096m:/var/cache/abrt-di

and you can run it manually any time you need.
Comment 27 Jan Kurik 2016-07-26 00:44:53 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle.
Changing version to '25'.

Note You need to log in before you can comment on or make changes to this bug.