My backups have recently been failing with: No space left in '/'. Looks like deja-dup is exhausting tmpfs memory/space during the backup verification phase. Doing TMPDIR=/var/tmp deja-dup --backup Seems to work
I am unable to login to launchpad but I have reported the issue via email to the upstream developer directly. Will update when I get a reply.
I believe this value is probably coming from duplicity, rather than deja-dup. If you set the temporary location in duplicity it will be reflected in deja-dup: # grep temproot /usr/lib64/python2.7/site-packages/duplicity/globals.py temproot = "/var/tmp" Should this perhaps be fixed in duplicity, if /tmp is not a sane default for it (in f18)?
(I'm the maintainer of deja-dup.) You are right that this is coming from duplicity instead of deja-dup, but deja-dup could certainly set TMPDIR differently or some such. Since /var/tmp is for persistent files, I'm not sure it's a suitable match here. Perhaps it could be used as a workaround, but we'd also need to throw in some logic to clean it up. The real fix is to have duplicity not use so much space, but that fix might take some time and investigation.
This bug is preventing people's backups from working, which qualifies this as a data loss bug. Could the severity be raised?
If you are referring to the bugzilla severity field, we don't use it in Fedora and setting it has no real effect. I agree that it is a urgent issue however and upstream developer, Michael Terry has been informed about this issue and has commented here as well. @Micheal, fyi, https://lists.fedoraproject.org/pipermail/devel/2013-January/176504.html If I can push a temporary fix while the underlying issue of Duplicity using a lot of space can be fixed later, I would prefer that. Thanks!
I'm working on a fix for Deja Dup to specify a tempdir on the same partition as the source files. We'll see how far I get this weekend.
Here's a patch for deja-dup that tries to specify a tempdir for duplicity on the same partition as the source files: https://code.launchpad.net/~mterry/deja-dup/tempdir/+merge/144032 This patch has not yet been reviewed or landed in trunk, but it seems to work for me. Since I believe Fedora is still on duplicity 0.6.18, you will also likely need this patch from duplicity 0.6.20 to stop duplicity from using /tmp for some files despite being passed a --tempdir: https://code.launchpad.net/~ed.so/duplicity/duplicity.tmpspacefix/+merge/123622
And if you have to patch duplicity anyway, I strongly encourage you to pull the following data corruption fix from duplicity trunk: https://code.launchpad.net/~mterry/duplicity/static-corruption/+merge/142044
Is that a patch against 24.0? Doesn't seem to be apply.
Created attachment 711657 [details] DD 24.0 tempdir patch Here is a patch against Deja Dup 24.0. It tries /tmp, /var/tmp, and ~/.cache/deja-dup/tmp in that order, preferring a directory on the same partition as the source files.
Note that you should not need to patch duplicity in F18 because it uses 0.6.20 (thanks for updating it in F18 before release). However! Duplicity should really be patched to deal with the corruption issue I mention in comment 8. I've filed bug 922576 about that.
*** Bug 906210 has been marked as a duplicate of this bug. ***
deja-dup-24.0-2.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/deja-dup-24.0-2.fc18
@Michael Terry Thanks for the patches. I have pushed updates of duplicity as well. Anyone running into this problem, please test and provide karma. Thanks!
Package deja-dup-24.0-2.fc18: * should fix your issue, * was pushed to the Fedora 18 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing deja-dup-24.0-2.fc18' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-4088/deja-dup-24.0-2.fc18 then log in and leave karma (feedback).
I still get gigabytes of data in /tmp/ during restore testing. Files created have this pattern: deja-dup-XXXXXX/deja-dup/xxxxx....../duplicity-full-signatures.....
Doesn't WFM 3.8G /tmp/deja-dup-WPTNUW My backups are still failing due to not enough space. duplicity-0.6.21-1.fc18.x86_64 deja-dup-24.0-2.fc18.x86_64
This also no longer works: TMPDIR=/var/tmp deja-dup --backup
Still broken here as well. duplicity-0.6.18-2.fc18.x86_64 deja-dup-24.0-1.fc18.x86_64
This is still broken almost two months after being reported, and it is causing considerable difficulty for people. Backups can't be created or restored, possibly driving people away from Deja-Dup. I tried @Chris Smart's solution of changing the "tempdir" field in "/usr/lib64/python2.7/site-packages/duplicity/globals.py" to "/var/tmp". This worked perfectly when restoring a 250GB backup. I don't think @Michael Terry's concern regarding persistence is that big of a deal, as duplicity cleared out all the data from /var/tmp when it was finished. Could the amendment to "/usr/lib64/python2.7/site-packages/duplicity/globals.py" not be put in the deja-dup package? Or could Fedora not start distributing deja-dup version 26 where this appears to be fixed?
Chris's suggested hack doesn't really solve the problem for everyone. It is just a workaround for some folks. Anyway, we are going to look into getting a 26 build out for testing soon.
(In reply to comment #21) > we are going to look into > getting a 26 build out for testing soon. Working on it.
deja-dup-26.0-1.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/deja-dup-26.0-1.fc18
(In reply to comment #23) > deja-dup-26.0-1.fc18 has been submitted as an update for Fedora 18. > https://admin.fedoraproject.org/updates/deja-dup-26.0-1.fc18 Appears to fix this problem for me.
deja-dup-26.0-1.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.
I still have this problem. Very worryingly, the error is that "/" has run out of space. It hasn't, /tmp has which is a tmpfs mount. $ rpm -q deja-dup duplicity deja-dup-26.0-1.fc18.x86_64 duplicity-0.6.21-1.fc18.x86_64
Michael Terry, do you need this to be filed upstream?
Any chance this will be fixed? This is preventing me from having any backups.
I am unable to reproduce this. I have mailed upstream.
Can you post the output of the following? df; DEJA_DUP_DEBUG=1 G_MESSAGES_DEBUG=all deja-dup --backup; df
That's a 250 gig log file! I'll sanitise it if you need it and chop out the middle. For now, here is the line you probably want: DUPLICITY: . Args: /usr/bin/duplicity ... --tempdir=/tmp --log-fd=19
Hm, then how about the output of: df; gsettings list-recursively org.gnome.DejaDup (You may want to scrub the output for any revealing info.)
Created attachment 750751 [details] df; gsettings list-recursively org.gnome.DejaDup
TMPDIR=/var/tmp deja-dup-preferences Click "Backup now" works. Not using TMPDIR=/var/tmp fails.
Please could this be fixed? I'd like backups. This works: TMP=/var/tmp/ deja-dup --backup This fails: deja-dup --backup Can someone change the release to 19?
Also the severity of this bug needs bumping since this is a data loss bug.
Data loss is a strong word for this. This bug prevents backing up, but does not affect existing backups. I could not reproduce on a fresh F19 install. Need Real Name is seemingly the only one left experiencing this bug? (Which is not to diminish the bug, just trying to figure out who can help debug.) It seems fixed for others, which indicates that the core fix is working. There is likely a corner case happening with Need Real Name's setup. Looking at the code, there are a few ways we could fallback to using /tmp: 1) We didn't detect any folders in the include list (seems unlikely) 2) We couldn't detect the filesystem ID of /home (also seems unlikely) 3) Your user's cache directory is not in /home (eh?) Is (3) true?
Although, I suppose this could stop restoring too, which would qualify as data loss. Though you could still restore via command line with duplicity.
> 3) Your user's cache directory is not in /home (eh?) Aha. Yes. ~/.cache/deja-dup is a symlink to the usb disk I am backing up to, since the cache grows too big for the SSD and fills it, preventing the backup from working.
Hmm, OK. So that's why we're falling back to /tmp. We look for a potential temp dir that is on the same partition as the source files, with the assumption that it will have enough space for us. There are other ways to find the best temp dir. I suppose we could look at actual free space, but that changes over time. Or we could have a special check if the backup location is local and try a temp dir there. But for now, Need Real Name, there is a workaround. Instead of making all of ~/.cache/deja-dup a symlink, make just the funky directory under it a symlink. That is, there should be a directory like 1ab3f49ec002d beneath it. If you make that a symlink to your external drive, deja-dup will use ~/.cache/deja-dup/tmp as its temp dir, as that will be on the same partition as $HOME.
The reason why I have the cache dir on the backup destination is that the cache directory gets so big over time that it fills the (expensive) SSD. Since I can't disable the cache for local to local backups, I used a symlink. I think it was suggested on the mailing list. I'm surprised it tries to find a tmp dir though, rather than using the system default. That seems unusual to me. I'll follow your suggestion. Many many thanks.
If I use a symlink for that directory I get the following error: Specified archive directory '/home/user/.cache/deja-dup/123...' does not exist, or is not a directory $ file /home/user/.cache/deja-dup/123 /home/user/.cache/deja-dup/123: symbolic link to `/run/media/user/EXTERNAL/deja-dup/123/'
This still isn't fixed in F20.
This message is a reminder that Fedora 18 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 18. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '18'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 18's end of life. Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 18 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior to Fedora 18's end of life. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 18 changed to end-of-life (EOL) status on 2014-01-14. Fedora 18 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
As I can't reopen this bug for F20, I've opened a new bug for it here: Bug 1086990 Someone with more privileges than I might wish to rationalise Bug 1086990 with this bug.
*** Bug 1086990 has been marked as a duplicate of this bug. ***
Thanks Rahul
Morgan, Can you take a look at comment #2 and let me know if that workaround addresses your problem? If it does, I am going to push a duplicity update with a patch as a workaround since it doesn't seem to be addressed even with the very latest duplicity and deja-dup.
Well, if I've understood Comment #2 correctly, I think I've tried that, see Comment 1 & 2 here: https://bugzilla.redhat.com/show_bug.cgi?id=1086990#c1 https://bugzilla.redhat.com/show_bug.cgi?id=1086990#c2 $ grep temproot /usr/lib64/python2.7/site-packages/duplicity/globals.py temproot = None Re https://bugzilla.redhat.com/show_bug.cgi?id=892063#c2 This seems wrong? ... ... ... [root@morgansmachine ~]# env | grep TMPDIR TMPDIR=/home/tmp [root@morgansmachine ~]# deja-dup-preferences [root@morgansmachine ~]# env | grep TMPDIR TMPDIR=/home/tmp [root@morgansmachine ~]# And, this is the response to # deja-dup-preferences: Temp space has 0 available, backup needs approx 34078720. I set the temp directory with env, correct? But, it doesn't seem to make any difference. Or, does Comment #2 suggest I should I re-write /usr/lib64/python2.7/site-packages/duplicity/globals.py because "temproot = None" isn't very helpful? I've tried one; I'll try the other.
OK, I didn't understand Comment #2 correctly, it's the other; and it works. Thank you Rahul. Oh I spoke to soon, that was before it said: Backup location is too small. Try using one with more space. Which takes me back to here: https://bugzilla.redhat.com/show_bug.cgi?id=1086363#c0 ... Just to confirm: [root@bigbadbeast backup]# df -h . Filesystem Size Used Avail Use% Mounted on /dev/md127 230G 60M 219G 1% /mnt/backup [root@bigbadbeast backup]# [morgan@morgansmachine ~]$ sudo du -sh /etc /opt /usr/local /home/morgan /home/readlegal /home/Music /home/Pictures 52M /etc 66M /opt 163M /usr/local 39G /home/morgan 27G /home/readlegal 40G /home/Music 4.1G /home/Pictures [morgan@morgansmachine ~]$ Umm...
I'm a bit befuddled by all this: From comment here- https://bugzilla.redhat.com/show_bug.cgi?id=1086990#c2 Temp space has 0 available, backup needs approx 34078720. And, from comment here- https://bugzilla.redhat.com/show_bug.cgi?id=1086363#c5 Temp space has 0 available, backup needs approx 68157440. Now, I'm no mathematical genius, but I think there's a relationship between those 2 numbers... 2 x 34078720 = 68157440 I'm sure there's an innocent explanation, but I can't see it's coincidence?
If I understand correctly, setting temp=/var/tmp in /usr/lib64/python2.7/site-packages/duplicity/globals.py and setting TMP=/var/tmp/ neither helps as a workaround for you. Correct?
(In reply to Rahul Sundaram from comment #53) > If I understand correctly, setting temp=/var/tmp in > /usr/lib64/python2.7/site-packages/duplicity/globals.py and setting > TMP=/var/tmp/ neither helps as a workaround for you. Correct? Well, I misunderstood Comment #2. I understood it to be suggesting that 'temproot = "/var/tmp"' set in duplicity/globals.py was a problem in need of fixing... If that were the setting in my duplicity/globals.py, it would in deed be a problem in need of fixing because /var/tmp is on my / partition, along with /tmp and with close to 0 space available - so, no good to me. I discovered that my duplicity/globals.py was set 'temproot = None', which seemed likely as much a problem as 'temproot = "/var/tmp"'. I assumed that to fix the 'temproot = "/var/tmp"' /problem/ I should set temp directory by the environment variable TMPDIR - as per the man page... But, that seemed to have no effect. See Comment #50 above. So, yes. Neither setting 'temp=/var/tmp' in duplicity/globals.py nor setting 'TMP=/var/tmp/' works for me. That is because I need my temp directory on a different partition than either /tmp/ and /var/tmp/. (I assume setting TMP is the same as setting TMPDIR, as TMPDIR takes precedence over TMP according to the man page - also, I notice there's a trailing '/' on 'TMP=/var/tmp/' cf 'TMPDIR=/home/tmp' and hope that makes no difference.) And, when I tried 'TMPDIR=/home/tmp' set by env variable I got: Temp space has 0 available, backup needs approx 34078720. (Comment #50 above) But $ df -h /home Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg01_mm-lv03_home 250G 184G 53G 78% /home And, when I tried 'temproot = "/home/tmp"' set in duplicity/globals.py I got: Backup location is too small. Try using one with more space. (Comment #51 above) But [root@bigbadbeast backup]# df -h . Filesystem Size Used Avail Use% Mounted on /dev/md127 230G 60M 219G 1% /mnt/backup [root@bigbadbeast backup]# $ rpm -qa deja-dup duplicity duplicity-0.6.23-2.fc20.x86_64 deja-dup-30.0-1.fc20.x86_64 $ I hope this helps to unbefuddle my befuddlement...
Hmm... only certainty in life: sin & taxes Looking at my last comment above, perhaps a trailing '/' is the answer... So, amending duplicity/globals.py 'temproot = "/home/tmp"' to 'temproot = "/home/tmp/"' and running: $ pkexec deja-dup-preferences --display=$DISPLAY (non-root gnome-terminal gdm login) Continues to result in- Backup location is too small. Try using one with more space. But, running deja-dup via root gdm login (from graphical environment, not gnome-terminal) seems to be working at the moment... I'll update with my further experience.
While running deja-dup via root gdm login (from graphical environment, not gnome-terminal) seems to be working, it also seems to be paying no respect to duplicity/globals.py 'temproot = "/home/tmp/"' [root@morgansmachine ~]# du -sh /var/tmp/duplicity-*/* 0 /var/tmp/duplicity-pTs2_b-tempdir/mkstemp-4iNbDS-1 25M /var/tmp/duplicity-pTs2_b-tempdir/mktemp-7KXAl6-671 Which is clearly active during deja-dup back-up as size keeps changing... [root@morgansmachine ~]# du -sh /home/tmp/* du: cannot access ‘/home/tmp/*’: No such file or directory [root@morgansmachine ~]# Nothing happening here... And cached in the home directory (/root) is close to 1.5G: [root@morgansmachine ~]# du -sh /root/.cache/deja-dup/* 4.0K /root/.cache/deja-dup/2fbe1999c814a85cc4595ac0b8a45c64 4.0K /root/.cache/deja-dup/43d5fd6c2ea7c6bbe46d917522b69469 103M /root/.cache/deja-dup/499ca929d5a4e9e15bc4cdca0b786e77 1.2G /root/.cache/deja-dup/6a40e0b81a1f4c5137952978869b39fc 4.0K /root/.cache/deja-dup/afe7fe6da861d5c4ded308fdcd6101b1 8.0K /root/.cache/deja-dup/metadata [root@morgansmachine ~]#
Scratch build of latest duplicity http://koji.fedoraproject.org/koji/taskinfo?taskID=6842839 please test
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
This message is a reminder that Fedora 20 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 20. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '20'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 20 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.