Bug 913783

Created attachment 700967 [details]
File: backtrace

Created attachment 700968 [details]
File: core_backtrace

Created attachment 700969 [details]
File: environ

Created attachment 700970 [details]
File: smolt_data

The place you are trying to save your backup to seems to be read-only? 

What kind of filesystem is /data/scooby-backup ? 

Any errors on it in dmesg or is it showing as ro in /proc/mounts ?

There may have been two problems.  One is that my filesystem mounted at /data/scooby-backup may have not been mounted at the time of backup and its mountpoint is normally read only to the user swbackup.  Two is that I found that I seemed to lose power to the hard drive yesterday when I was testing and that may have occured when I got this error.

Right now, I am running the backup again and it seems to be taking a long time as it is saying that it is "regressing the destination now".

I have another backup on the same system trying to run to another mountpoint.  I can confirm the mounted filesystem has 1.1TB of space available but yet when I run:

rdiff-backup --include-globbing-filelist /home/swbackup/rdiff_backup2_includes.txt --exclude / scooby::/ /data-2.0TB/scooby-backup

I get the following output:

Exception '[Errno 28] No space left on device' raised of class '<type 'exceptions.IOError'>':
  File "/usr/lib/python2.7/site-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/restore.py", line 462, in get_fp
    current_fp = self.get_first_fp()
  File "/usr/lib/python2.7/site-packages/rdiff_backup/restore.py", line 499, in get_first_fp
    rpath.copyfileobj(fp, current_fp)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/rpath.py", line 64, in copyfileobj
    outputfp.write(inbuf)

Exception '[Errno 28] No space left on device' raised of class '<type 'exceptions.IOError'>':
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 337, in Backup
    backup_final_init(rpout)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 501, in backup_final_init
    checkdest_if_necessary(rpout)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 920, in checkdest_if_necessary
    dest_rp.conn.regress.Regress(dest_rp)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/regress.py", line 71, in Regress
    for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/rorpiter.py", line 281, in __call__
    last_branch.fast_process(*args)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/regress.py", line 268, in fast_process
    if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/regress.py", line 296, in restore_orig_regfile
    rf.mirror_rp.write_from_fileobj(rf.get_restore_fp())
  File "/usr/lib/python2.7/site-packages/rdiff_backup/restore.py", line 488, in get_restore_fp
    return robust.check_common_error(error_handler, get_fp)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/restore.py", line 462, in get_fp
    current_fp = self.get_first_fp()
  File "/usr/lib/python2.7/site-packages/rdiff_backup/restore.py", line 499, in get_first_fp
    rpath.copyfileobj(fp, current_fp)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/rpath.py", line 64, in copyfileobj
    outputfp.write(inbuf)

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 337, in Backup
    backup_final_init(rpout)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 501, in backup_final_init
    checkdest_if_necessary(rpout)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/Main.py", line 920, in checkdest_if_necessary
    dest_rp.conn.regress.Regress(dest_rp)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/regress.py", line 71, in Regress
    for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/rorpiter.py", line 281, in __call__
    last_branch.fast_process(*args)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/regress.py", line 268, in fast_process
    if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/regress.py", line 296, in restore_orig_regfile
    rf.mirror_rp.write_from_fileobj(rf.get_restore_fp())
  File "/usr/lib/python2.7/site-packages/rdiff_backup/restore.py", line 488, in get_restore_fp
    return robust.check_common_error(error_handler, get_fp)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/restore.py", line 462, in get_fp
    current_fp = self.get_first_fp()
  File "/usr/lib/python2.7/site-packages/rdiff_backup/restore.py", line 499, in get_first_fp
    rpath.copyfileobj(fp, current_fp)
  File "/usr/lib/python2.7/site-packages/rdiff_backup/rpath.py", line 64, in copyfileobj
    outputfp.write(inbuf)
IOError: [Errno 28] No space left on device
Fatal Error: Lost connection to the remote system


This is the result of df -h:

Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                       2.0G     0  2.0G   0% /dev
tmpfs                          2.0G  152K  2.0G   1% /dev/shm
tmpfs                          2.0G  4.0M  2.0G   1% /run
tmpfs                          2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/vg_hotpot-lv_root  267G   25G  229G  10% /
/dev/sdb1                      984M   89M  844M  10% /boot
/dev/loop2                     2.0G  3.1M  1.9G   1% /tmp
/dev/loop0                     1.8T  259G  1.5T  15% /data
/dev/loop1                     1.4T  558G  748G  43% /data-1.5TB
/dev/loop3                     1.8T  616G  1.1T  36% /data-2.0TB

The following from the messages log might also be of interest:

Feb 23 11:43:38 hotpot abrt: detected unhandled Python exception in '/usr/bin/rdiff-backup'
Feb 23 11:43:39 hotpot abrtd: New client connected
Feb 23 11:43:40 hotpot abrtd: Directory 'pyhook-2013-02-23-11:43:40-4952' creation detected
Feb 23 11:43:40 hotpot abrt-server[4968]: Saved problem directory of pid 4952 to '/var/spool/abrt/pyhook-2013-02-23-11:43:40-4952'
Feb 23 11:43:40 hotpot abrtd: BDB2053 Freeing read locks for locker 0x6c: 4862/3077723968
Feb 23 11:43:40 hotpot abrtd: BDB2053 Freeing read locks for locker 0x6d: 4862/3077723968
Feb 23 11:43:40 hotpot abrtd: BDB2053 Freeing read locks for locker 0x6e: 4862/3077723968
Feb 23 11:43:40 hotpot abrtd: BDB2053 Freeing read locks for locker 0x6f: 4862/3077723968
Feb 23 11:43:41 hotpot abrtd: Duplicate: core backtrace
Feb 23 11:43:41 hotpot abrtd: DUP_OF_DIR: /var/spool/abrt/pyhook-2013-02-22-13:23:33-11784
Feb 23 11:43:41 hotpot abrtd: Deleting problem directory pyhook-2013-02-23-11:43:40-4952 (dup of pyhook-2013-02-22-13:23:33-11784)

How about 'df -i' ?

Filesystem                       Inodes   IUsed     IFree IUse% Mounted on
devtmpfs                         207176     459    206717    1% /dev
tmpfs                            211895       9    211886    1% /dev/shm
tmpfs                            211895     641    211254    1% /run
tmpfs                            211895      12    211883    1% /sys/fs/cgroup
/dev/mapper/vg_hotpot-lv_root  17760256  556120  17204136    4% /
/dev/sdb1                        260096     371    259725    1% /boot
/dev/loop2                       128000      61    127939    1% /tmp
/dev/loop0                    122101760 1001574 121100186    1% /data
/dev/loop1                     91578368  143818  91434550    1% /data-1.5TB
/dev/loop3                    122101760   43736 122058024    1% /data-2.0TB

ok, that looks fine. 

So, anything in dmesg about that filesystem?

If you do a --check-destination-dir on that directory does it correctly check the backups as ok?

Does it work again after a check-destination-dir?

Okay.  I think I figured out some of what is happening.

It is regressing the data backup to a previous date and it is working on a 4+ GB file.

I just noticed as it was working on restoring that file that my /tmp filesystem was getting filled up.  It isn't more than 2GB in size.

That is what is running out of space.

Is there something I need to do to not have /tmp used or do I just need to make sure my /tmp has at least as much space as the largest file in my backup for cases where I need to have the backup regressed?

Yeah, see --tempdir in the man page. You can also set TMPDIR env...

Cool.  Thanks so much for your help!