Bug 624964 - Wasting lots of diskspace for /var/lib/yum/yumdb
Wasting lots of diskspace for /var/lib/yum/yumdb
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
rawhide
All Linux
low Severity medium
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-08-18 05:02 EDT by Zdenek Kabelac
Modified: 2014-01-21 18:16 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-18 10:21:56 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Zdenek Kabelac 2010-08-18 05:02:25 EDT
Description of problem:

I've noticed directory: /var/lib/yum/yumdb   takes a lot of space considering
how much information is stored there.

I know - disk space is cheap - but is that cheap we should store ~130 bytes
in 40KB?

i.e. in subdir:

k/89766b47613acd2b60b446515d05c27fda26f7b5-kdm-4.5.0-2.fc15-x86_64

ls -la

-rw-r--r--  1 root root     5 Aug 16 11:35 changed_by
-rw-r--r--  1 root root    64 Aug 16 11:35 checksum_data
-rw-r--r--  1 root root     6 Aug 16 11:35 checksum_type
-rw-r--r--  1 root root    21 Aug 16 11:35 command_line
-rw-r--r--  1 root root     7 Aug 16 11:35 from_repo
-rw-r--r--  1 root root    10 Aug 16 11:35 from_repo_revision
-rw-r--r--  1 root root    10 Aug 16 11:35 from_repo_timestamp
-rw-r--r--  1 root root     4 Aug 16 11:35 reason
-rw-r--r--  1 root root     2 Aug 16 11:35 releasever

du -s *
 
4	command_line
4	from_repo
4	from_repo_revision
4	from_repo_timestamp
4	changed_by
4	checksum_data
4	checksum_type
4	reason
4	releasever

it looks like either stupid test file or just slightly more advanced engines like TokyoCabinet could save lots of MB of disk space here.

(in my case  120kB of information is stored in approximately 30MB)

Version-Release number of selected component (if applicable):
yum-3.2.28-3.fc15.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 James Antill 2010-08-18 10:21:56 EDT
The latest yum (3.2.28) will now hardlink the information which is similar, you can see the best case for this with:

    hardlink -c /var/lib/yumdb

...which we _might_ start calling sparodically from yum.
 It's true that even after that we will store a 4 byte file using 4k, on ext4 ... but that problem goes away with btrfs.

 As for the rationale for why yumdb was designed/created as it was, feel free to read:

    http://yum.baseurl.org/wiki/YumDB
Comment 2 Zdenek Kabelac 2010-08-18 10:47:07 EDT
IMHO wrong design is not excuse to keep it 'broken' forever.

And I seriously doubt btrfs will solve such thing (at least not in any near future) - in fact if you track lkml - you might have noticed that current btrfs is actually extremely bad in storing lots of small files.

Of course pushing users to use one specific filesystem to get decent performance from yum tools is also quite 'strange' idea....

Using plain simple easy to use text file at least for package - would significantly reduce bloat - and would be still complaint to the URL you are mentioning. And of course sqlite/libdb isn't the smartest/fastest DB for this purpose.

I really think this issue deserves to be resolved differently than saying not a bug....
Comment 3 James Antill 2010-08-18 14:53:58 EDT
> IMHO wrong design is not excuse to keep it 'broken' forever.

 Did you read the page? The most common operations are:

1. Read one entry.

2. Write one entry.

...using a single file for all addon data for a package would make #2 into a read-modify-write cycle, would introduce a lot of complexity as we'd need to serial/unserialize all the data (and likely need to change the API so we could only do a single write for multiple updates), would likely make hand editing the data with vim/etc. hard (if not virtually impossible).
 And with 3.2.28, it will almost certainly make a bunch of operations _slower_ due to not only reading much more data and deserializing ... but the fact we can't skip the reads with the hardlinks.

 Given that, I'm happy with "wrong and broken".

> And I seriously doubt btrfs will solve such thing

 You should let the btrfs developers know then, because they have explicitly told me that if you store 3 small files in a directory btrfs will store much closer to "wc -c" bytes than 12k.

> Of course pushing users to use one specific filesystem to get
> decent performance

 Here by "performance" you mean disk space. And, yeh, if users need to save 5¢ in disk space ... my recommendation is they look at btrfs.
Comment 4 Zdenek Kabelac 2010-08-19 04:23:48 EDT
(In reply to comment #3)
> > IMHO wrong design is not excuse to keep it 'broken' forever.
> 
>  Did you read the page? The most common operations are:

Sure I did.

> 
> 1. Read one entry.
> 
> 2. Write one entry.
> 
> ...using a single file for all addon data for a package would make #2 into a
> read-modify-write cycle, would introduce a lot of complexity as we'd need to

Well as I said - you could reuse lots of projects which already hide this complexity for you - so your tool will really read & write one entry - everything else stays hidden to you. It will be obviously faster then your current implementation - as only one block from filesystem will be taken.
Eveything else would be resolved in userspace land (you somehow forget that application should not offload it's task to kernel - if it can do far better job in userspace)

Noone else seems to be going in direction of storing 1 byte information in 1 file yet - as it is big waste of diskspace & CPU.

And I hope you are not going to argue it's more safe - as I could keep this 30M database probably efficiently stored in less then 64KB - that gives extra space to keep nearly 500 copies of this db...

> serial/unserialize all the data (and likely need to change the API so we could
> only do a single write for multiple updates), would likely make hand editing
> the data with vim/etc. hard (if not virtually impossible).
>  And with 3.2.28, it will almost certainly make a bunch of operations _slower_
> due to not only reading much more data and deserializing ... but the fact we
> can't skip the reads with the hardlinks.
> 
>  Given that, I'm happy with "wrong and broken".

It is so broken that I'm curios how you can't see that.

Do you know how much data is written to SSD when you create those 7 small files?
And how much CPU is wasted by looking at free space in filesystem.

> 
> > And I seriously doubt btrfs will solve such thing
> 
>  You should let the btrfs developers know then, because they have explicitly
> told me that if you store 3 small files in a directory btrfs will store much
> closer to "wc -c" bytes than 12k.

They already know: http://lkml.org/lkml/2010/6/3/313

> > Of course pushing users to use one specific filesystem to get
> > decent performance
> 
>  Here by "performance" you mean disk space. And, yeh, if users need to save 5¢
> in disk space ... my recommendation is they look at btrfs.

Well so far reiserfs is much more reliable in this case - but still it's so  inefficient that it's hard to believe you want to keep it this was forever.

Is it really that bad to reconsider to keep at least package dir in one file?

Note You need to log in before you can comment on or make changes to this bug.