Bug 928869 - [abrt] rdiff-backup-1.2.8-7.fc17: Main.py:907:checkdest_need_check:AssertionError: Found too many current_mirror incs!
Summary: [abrt] rdiff-backup-1.2.8-7.fc17: Main.py:907:checkdest_need_check:AssertionE...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: rdiff-backup
Version: 17
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kevin Fenzi
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: abrt_hash:4f25dc706fe9b47e008d5ab3e3d...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-03-28 16:13 UTC by jghobrial
Modified: 2014-06-15 19:21 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2013-04-02 15:15:23 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: backtrace (1.57 KB, text/plain)
2013-03-28 16:14 UTC, jghobrial
no flags Details
File: core_backtrace (917 bytes, text/plain)
2013-03-28 16:14 UTC, jghobrial
no flags Details
File: environ (109 bytes, text/plain)
2013-03-28 16:14 UTC, jghobrial
no flags Details
File: smolt_data (5.62 KB, text/plain)
2013-03-28 16:14 UTC, jghobrial
no flags Details

Description jghobrial 2013-03-28 16:13:59 UTC
Version-Release number of selected component:
rdiff-backup-1.2.8-7.fc17

Additional info:
cmdline:        /usr/bin/python /usr/bin/rdiff-backup --print-statistics /boot /mnt/backup/boot
executable:     /usr/bin/rdiff-backup
kernel:         3.8.3-103.fc17.x86_64
uid:            0

Truncated backtrace:
Main.py:907:checkdest_need_check:AssertionError: Found too many current_mirror incs!

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 337, in Backup
    backup_final_init(rpout)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 501, in backup_final_init
    checkdest_if_necessary(rpout)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 916, in checkdest_if_necessary
    need_check = checkdest_need_check(dest_rp)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 907, in checkdest_need_check
    assert len(curmir_incs) == 2, "Found too many current_mirror incs!"
AssertionError: Found too many current_mirror incs!

Local variables in innermost frame:
curmirroot: <rdiff_backup.rpath.RPath instance at 0x1a65518>
curmir_incs: [<rdiff_backup.rpath.RPath instance at 0x1a65908>, <rdiff_backup.rpath.RPath instance at 0x1a66cb0>, <rdiff_backup.rpath.RPath instance at 0x1a66d40>]
dest_rp: <rdiff_backup.rpath.RPath instance at 0x1a60128>
filename: 'extended_attributes.2012-12-10T01:00:02-06:00.snapshot.gz'

Comment 1 jghobrial 2013-03-28 16:14:02 UTC
Created attachment 717722 [details]
File: backtrace

Comment 2 jghobrial 2013-03-28 16:14:03 UTC
Created attachment 717723 [details]
File: core_backtrace

Comment 3 jghobrial 2013-03-28 16:14:04 UTC
Created attachment 717724 [details]
File: environ

Comment 4 jghobrial 2013-03-28 16:14:06 UTC
Created attachment 717725 [details]
File: smolt_data

Comment 5 Kevin Fenzi 2013-03-28 17:06:31 UTC
So, this happens when your backup dir is messed up. ;) 

Can you attach the output of: 

ls -la /mnt/backup/boot/rdiff-backup-data/current_mirror*

You should normally have one of those showing the current backup. 
If you have 2 of them it means a backup failed and needs to regress. 
If you have more than 2, you get this error and something tampered with your backup dir somehow.

Comment 6 jghobrial 2013-03-28 20:32:16 UTC
ls -la /mnt/backup/boot/rdiff-backup-data/current_mirror*

-rw-------. 1 root root 10 Mar 15 01:06 /mnt/backup/boot/rdiff-backup-data/current_mirror.2013-03-15T01:06:02-05:00.data
-rw-------. 1 root root 10 Mar 15 01:08 /mnt/backup/boot/rdiff-backup-data/current_mirror.2013-03-15T01:07:02-05:00.data
-rw-------. 1 root root 10 Mar 15 01:08 /mnt/backup/boot/rdiff-backup-data/current_mirror.2013-03-15T01:08:02-05:00.data

I've deleted items like this before, but it keeps occurring so I thought to submit this. I'm tempted just to format the external disk and move on, but have seen enough google hits to thing something else is going on.

Thanks for the quick response. I'm not sure where to start debugging this.

Here's my bash script run at 1 AM.

#!/bin/bash
mount | grep -q /mnt/backup
if [ $? -eq 0 ]
then
   for i in /boot /etc /var/log /var/account /home /root /opt
   do
      echo /usr/bin/rdiff-backup --print-statistics ${i} /mnt/backup${i}
      /usr/bin/rdiff-backup --print-statistics ${i} /mnt/backup${i}
   done
else
   echo "External disk NOT mounted"
fi | tee /root/backup.log

Comment 7 Kevin Fenzi 2013-03-30 01:22:34 UTC
It seems like it's running 3 times? once every minute?

Check your cron log?

Comment 8 jghobrial 2013-04-01 19:29:58 UTC
There's a huge amount of runs happening. I'm not sure why. 

I've wrapped the script with some PID file magic to try and prevent multiple instances from running.

# grep rdiff-backup /var/log/cron
Apr  1 01:00:01 CROND[6226]: (root) CMD (/root/bin/rdiff-backup)
...
Apr  1 01:59:01 CROND[6226]: (root) CMD (/root/bin/rdiff-backup)

# grep rdiff-backup /var/log/cron | wc -l
60

It's running my script every minute for 1 hour.

I believe the answer is rdiff-backup may be running from the previous day if there is enough data churn. Meaning, the current day's entry will start to run creating an overlapping situation since the previous day's run is still happening. The current day's instance creates the multiple current_mirror.* file and once that happens the script will always die horribly. cron dutifully detects the failure and retries every minute until the next hour.

Does this make sense?

Maybe, rdiff-backup shouldn't die so horribly when it detects too many current_mirror.* files and provide an easier to decipher error message? A nicety, not a necessity...

If my diagnoses makes sense then excellent. If you think it's something else then let me know what you need.

Comment 9 jghobrial 2013-04-01 19:32:20 UTC
"I've wrapped the script with some PID file magic to try and prevent multiple instances from running."

Sorry, meaning that I just implemented that right now, today.

Comment 10 Kevin Fenzi 2013-04-02 00:22:21 UTC
Sure, that sounds plausable. 

I guess lets see if your fix works tonight?

Comment 11 jghobrial 2013-04-02 15:04:37 UTC
Please close this issue.

PEBKAC... :(

Comment 12 Kevin Fenzi 2013-04-02 15:15:23 UTC
Sure. It happens to us all... no problem.

Comment 13 Jan Vlug 2014-06-15 19:21:38 UTC
See also: bug 1109563


Note You need to log in before you can comment on or make changes to this bug.