Bug 1614312

Description of problem:
Poor performance when performing operations on DAR archives with many files (>100,000).  Performance drops dramatically as number of files increases.

Version-Release number of selected component (if applicable):
2.5.15

How reproducible:
Easily reproducible.

Steps to Reproduce:
1. Create a new dar archive, archiving a directory tree of a few hundred thousand to a million+ files (ie, full system backup)

dar -c backup -R / -M

2. Perform a differential backup using the full backup as reference, or extraction of a file, or list archive or add archive to dar_manager database.

dar -c backup_diff1 -A backup -R / -M
or
dar -l backup

Actual results:
There is a lengthy delay performing step two while dar reads the catalogue, in the order of 5-10 minutes or more.

Expected results:
No long delay


Additional info:

The poor performance is caused by dar's use of its internal class for storing large numbers, infinint.  This class provides very large number support, but was the authors early attempt at creating such a class.

Dar does not rely on this.  64 bit integers can be used with the --enable-mode=64 configure mode which uses native 64 bit types.  Infinint is the default of no other type is specified (unlimited integer size)

The site says that infinint has a performance cost
http://dar.linux.free.fr/doc/from_sources.html

"if set, replace "infinint" integers by 32 or 64 bits integers. This makes a faster executable and less fond of memory, but with several restrictions (about for example ability to handle large files, or high dates. See the limitations for more). Note that 64 bits integers are available on 32 bits CPU plateforms."

In practice, the performance cost seems to be very high, to the point where some operations can seem to have hung.

https://sourceforge.net/p/dar/mailman/dar-support/thread/20120113110808.52a4a8b8%40supagro.inra.fr/

Using the verson of dar supplied by, a listing of the archive containing a backup of my system takes 615 seconds (> 1 million files)

$ time /usr/bin/dar -l root > /dev/null

/usr/bin/dar -l root > /dev/null  615.30s user 1.81s system 107% cpu 9:31.64 total


Using a copy of dar compiled with the --enable-mode=64 flag, the same operation takes 28 seconds.

$ time dar -l root > /dev/null
dar -l root > /dev/null  28.89s user 0.48s system 97% cpu 30.253 total

A speed up of about 20x.

Memory usage is also improved.  The operation using dar compiled with --enable-mode=64 uses 879MB.  The operation using the RPM supplied version with infinint the operation takes over 2124MB


The limitations of using the native 64 bit values are listed here
http://dar.linux.free.fr/doc/Limitations.html
but these do not seem to warrant not using them, as the limits imposed on the dates and file sizes far, far exceed the what todays technology is capable of.


My testing indicates that this does not affect operations, apart from the improvement in speed and memory usage.

To improve the performance of dar significantly, for large archives, as well as the performance of front-ends which use dar or libdar, I recommend that --enable-mode=64 be added to the configuration option for builds of dar for significant performance increases.

I note that the MacOSX version of dar I use uses 64 bit, not unlimited integers, and again, performs satisfactorily in comparison to Fedora.