Bug 516309

Thanks for report, will check our xattrs patch - it's quite possible that this patch could be culprit of that limitation.

Any ETA on a fix - this is really causing problems with amanda backups.

I did just a quick review of the patch and found nothing obvious. It's on top of my todo list at the moment, so I hope I'll find something soon. I'll try to reproduce it myself on my machine with some smaller reproducer.

Tried to reproduce with 100 files with size of 8,5G (8912896000 bytes) with extended attributes (--dereference option just because they are symlinks ;) ) and with "tar --create --file /dev/null --directory . --one-file-system --atime-preserve=system --no-check-device --sparse --xattrs --listed-incremental temp.new --ignore-failed-read --totals --exclude-from temp.exclude --dereference mybigfile.*
" , but the error doesn't seem to occur then (just passes with Total bytes written: 900202659840 (839GiB, 76GiB/s)) on my i386 architecture.

Do you have some simple reproducer? Some specific filesystem or mount options?

Looks like it might be interaction with --sparse and an actual (or at least apparently) sparse file.  I've posted my reproducing file (20GB) here:

http://www.cora.nwra.com/~orion/paperSin38_3_g01.cdf

gdb seems to indicate that it may be using the old gnu format to dump the sparse file and maybe that's the issue?

(gdb) bt
#0  to_chars (negative=<value optimized out>, value=0, valsize=8, substitute=0, where=0x80923ca "07143137000",
    size=12, type=0x807cadd "off_t") at create.c:353
#1  0x0805093b in off_to_chars (v=965524992, p=0x80923ca "07143137000", s=12) at create.c:433
#2  0x080636c1 in oldgnu_store_sparse_info (file=0xbfffeefc, pindex=0xbfffeeac, sp=0x80923ca, sparse_size=1)
    at sparse.c:696
#3  0x080637d2 in oldgnu_dump_header (file=0xbfffeefc) at sparse.c:721
#4  0x08063f09 in tar_sparse_dump_header (file=<value optimized out>) at sparse.c:154
#5  sparse_dump_file (file=<value optimized out>) at sparse.c:395
#6  0x0805252a in dump_file0 (st=0xbffff098, p=<value optimized out>, top_level=<value optimized out>,
    parent_device=2193) at create.c:1643
#7  0x08053521 in dump_file (p=0x80918d8 "./paperSin38_3_g01.cdf", top_level=0, parent_device=1)
    at create.c:1824
#8  0x080531ba in dump_dir0 (parent_device=<value optimized out>, top_level=<value optimized out>,
    st=<value optimized out>, directory=<value optimized out>) at create.c:1244
#9  dump_dir (parent_device=<value optimized out>, top_level=<value optimized out>, st=<value optimized out>,
    directory=<value optimized out>) at create.c:1293
#10 dump_file0 (parent_device=<value optimized out>, top_level=<value optimized out>,
    st=<value optimized out>, directory=<value optimized out>) at create.c:1631
#11 0x08053521 in dump_file (p=0x8090e30 ".", top_level=1, parent_device=1) at create.c:1824
#12 0x080535b6 in create_archive () at create.c:1361
#13 0x08066b25 in main (argc=13, argv=0xbffff564) at tar.c:2561

old gnu format and gnu format have the same optab functions in sparse.c, I don't think that's the issue. Thanks for reproducer, but 20G is quite big, I'll try to reproduce it somehow without downloading it.

Any idea here?  Anything else I can do to help?

Maybe ... could you try to reproduce this problem with sparse file created via dd? 
I tried several sparse files created by dd(different sizes, different skips/seeks, more "holes"), but I was not able to reproduce this issue. Downloading your reproducer file would mean quite a lot of bandwidth and time (I started download, but with 20kB/s speed it's nonsense to download it) - so I would be happy with sparse reproducer that could be generated somehow on my machine ...

I managed to compress the cdf file to 6.5GB.  We should have 3Mbps outgoing barring other usage so you might have more luck.

http://www.cora.nwra.com/~orion/tartest.cdf.bz2

Another possible piece of the puzzle, is that this is a sparse file created by using rsync --sparse.  On the original system, it is not sparse. 

Do you know of any tools to analyze sparse files?  One possibility is to scan the file and print out a list of the holes in order to reproduce it.  Might not be too hard to write in C.

Also trying to create:
http://www.cora.nwra.com/~orion/tartest.cdf.xz

Not done yet so don't know how big that will be yet.

xz file is 3.2GB

Ok, started to download xz to my home computer, hopefully it will work with that file. Thanks, I'll let you know later ...

Bad luck with this sparse file ... 

Finally downloaded xz and 
$ xz -d tartest.cdf.xz 
xz: tartest.cdf.xz: Compressed data is corrupt

Old GNU (and POSIX) tar header has 8G limitation, but this should not be default format. Will try non-sparse large >8G file and few other things before another download ...

Turns out my original compress failed.  Re-running now but taking a long time.  Will post when it is done.

Complete, but 5.4GB now.

Any chance the bug is specific for a particular architecture? Can you e.g. compare the behavior on 32bit vs. 64bit arch? And please provide some details about your kernel, filesystem etc. Thanks in advance!

Sorry for late check, Orion, got a bit busy and bought new computer at home - and was not able to access your downloaded file for a while. However - I'm still not able to reproduce even with your file (I have F-12, but I checked F-11 tar sources). 

[Reset@localhost src]$ pwd
/home/Reset/koji/tar/F-11/tar-1.22/src
[Reset@localhost src]$ touch temp.new
[Reset@localhost src]$ touch temp.exclude
[Reset@localhost src]$ ./tar --create --file /dev/null --directory . --one-file-system --atime-preserve=system --no-check-device --sparse --xattrs --listed-incremental temp.new --ignore-failed-read --totals --exclude-from temp.exclude ~/koji/tar/devel/tartest.cdf 
./tar: Removing leading `/' from member names
Total bytes written: 21457561600 (20GiB, 154GiB/s)

Do you still experience the issue? Is there something wrong on the command syntax I used? Could it be arch specific? I have i386 architecture on my machine (but from report it seems you have the same). Are you able to experience that issue directly with tar binary or just within amanda backup (maybe ulimit there?) ?
Thanks in advance for answer...

Finally managed to break at the point of error:

#0  to_chars (negative=0, value=11318425088, valsize=8, substitute=0,
    where=0x809407c '0' <repeats 11 times>, size=12, type=0x807ca1d "off_t") at create.c:302
#1  0x0805098c in off_to_chars (v=<value optimized out>, p=<value optimized out>,
    s=<value optimized out>) at create.c:433
#2  0x08062995 in pax_dump_header_1 (file=<value optimized out>) at sparse.c:1033
#3  pax_dump_header (file=<value optimized out>) at sparse.c:1064
#4  0x08064029 in tar_sparse_dump_header (file=<value optimized out>) at sparse.c:154
#5  sparse_dump_file (file=<value optimized out>) at sparse.c:395
#6  0x080525c7 in dump_file0 (st=0xbffff168, p=<value optimized out>,
    top_level=<value optimized out>, parent_device=<value optimized out>) at create.c:1643
#7  0x08053602 in dump_file (p=<value optimized out>, top_level=<value optimized out>,
    parent_device=<value optimized out>) at create.c:1827
#8  0x08053717 in create_archive () at create.c:1320
#9  0x08066c6b in main (argc=<value optimized out>, argv=<value optimized out>) at tar.c:2561

this is now with tar-1.22-10.fc12.i686.

command is:
--create --file /dev/null --directory /data/backup1/paiute/export/home/alexand/COMMAS/alison --one-file-system --atime-preserve=system --no-check-device --listed-incremental /tmp/temp.new --sparse --xattrs --ignore-failed-read --totals --exclude-from /var/log/amanda/amgtar._export_backup1_paiute.20090825161003.exclude paperSin38_3_g01.cdf              

It's trying to store the size of the shrunken file:

sparse.c:1032
  /* Store the effective (shrunken) file size */
  OFF_TO_CHARS (file->stat_info->archive_file_size, blk->header.size);

 and it is larger than 8589934591.

Should give you some more info on how to reproduce.  Any reason why we would be limited to off_t in storing this size when we aren't in other file sizes?

This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Sorry for long delay, it got off my radar - and current mail on bug-tar mailing lists remind me that's still unresolved.

As the message is print by to_chars_subst(), maxval in the error message is clear that gnu_format variable is not set - so the archive_format global variable is not GNU_FORMAT or OLDGNU_FORMAT - so maxval limits the filesize to 8GB. However - POSIX_FORMAT should not be limited that way.
As extended attributes usage forces POSIX_FORMAT (and usual default format is GNU), that's maybe the culprit why --xattrs causes the issue...

This is still present in tar-1.26-7.fc17.x86_64 when both --acls and --sparse are used.

Also get the same error on EL6 (tar-1.23-7.el6.x86_64).
There is much simpler command to reproduce it.

$ tar -co 24GBtestfile >/dev/null
tar: value 24104711334 out of off_t range 0..8589934591
tar: Exiting with failure status due to previous errors

The same command works just fine for files smaller then 8589934591 bytes.
Thankfully the old fashion syntax works without a fail.

$ tar -cf - 24GBtestfile >/dev/null
$

Upstream proposal:

http://www.mail-archive.com/bug-tar@gnu.org/msg03905.html

(In reply to comment #22)
> Also get the same error on EL6 (tar-1.23-7.el6.x86_64).
> There is much simpler command to reproduce it.
> 
> $ tar -co 24GBtestfile >/dev/null
> tar: value 24104711334 out of off_t range 0..8589934591
> tar: Exiting with failure status due to previous errors
> 
> The same command works just fine for files smaller then 8589934591 bytes.
> Thankfully the old fashion syntax works without a fail.
> 
> $ tar -cf - 24GBtestfile >/dev/null
> $

Hi Tramada, this is not related to this bugzilla.  You have used old 'v7' format
by using the -o option.  Probably misspelled with '-O'.  This legacy format is
expected to not store files > 8GB.

Upstream fixed this issue:

http://lists.gnu.org/archive/html/bug-tar/2013-01/msg00002.html

I'll backport this patch today.  Even if there is broken listing of such
created tarballs - it is another bug.  I'll create new BZ# for this one.

Pavel

> [... add listing ...]
> I'll create new BZ# for this one.

Bug created as bug #916995.

Reported problem from this bugzilla is fixed & built in rawhide.

~> https://koji.fedoraproject.org/koji/taskinfo?taskID=5066463

Pavel

tar-1.26-12.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/tar-1.26-12.fc18

tar-1.26-12.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.