Bug 1492288

Summary: MemoryError during backup
Product: [Fedora] Fedora Reporter: kartochka378
Component: duplicityAssignee: Gwyn Ciesla <gwync>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 27CC: agrover, alciregi, chrisa, daryll, giacecco, gwync, haliyo, james, j.hoffmann, liblit, mbrancaleoni, olivier.crete, steve.bz, twaugh, zaitcev
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-05 17:42:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1512733    
Bug Blocks:    
Attachments:
Description Flags
Traceback none

Description kartochka378 2017-09-16 07:39:55 UTC
Created attachment 1326676 [details]
Traceback

Description of problem:

Traceback durung backup

Version-Release number of selected component (if applicable):

duplicity-0.7.13.1-2.fc26

How reproducible:
every time

Steps to Reproduce:
1.
start backup
2.
3.

Actual results:
Traceback

Expected results:
success backup

Additional info:

testing repo used
backup file in remote nfs share, rest is usual default

Comment 1 Jobst Hoffmann 2017-09-20 04:17:06 UTC
Same for me

backup file on external hdd over USB 3.0

Comment 2 kartochka378 2017-09-22 01:18:50 UTC
Tried to stop auto backup for a day, then after 2 updates to testing and manually start backup it finished with success. No idea what package fixed it. I set to auto backup and wait for traceback.

Comment 3 kartochka378 2017-09-28 06:52:19 UTC
Interesting, it still traceback when auto backup is on, and succeed in case I disable auto backup and start manually. Weird.

Comment 4 Alessio 2017-10-05 10:07:00 UTC
Same here.
duplicity-0.7.13.1-2.fc26.x86_64
Auto backup on external USB 3 HD
Encryption enabled.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1410, in do_backup
    globals.archive_dir).set_values()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 710, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 836, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 824, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 101, in add_filename
    self.set_manifest(filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 148, in set_manifest
    self.set_files_changed()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 128, in set_files_changed
    mf = self.get_manifest()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 250, in get_manifest
    return self.get_local_manifest()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 224, in get_local_manifest
    return manifest.Manifest().from_string(manifest_buffer)
  File "/usr/lib64/python2.7/site-packages/duplicity/manifest.py", line 215, in from_string
    vi = VolumeInfo().from_string(match.group(1))
MemoryError

Comment 5 Olivier Crête 2017-11-10 18:59:59 UTC
Here is the upstream bug for this

https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1727653

Comment 6 Olivier Crête 2017-11-18 16:57:27 UTC
This is fixed by 0.7.15, which I just tested, which is in updates-testing, see bug #1512733

Comment 7 Jobst Hoffmann 2017-11-20 11:29:30 UTC
Yesterday I've installed 0.7.15, the full backup succeded.

Today the incremental backup has stopped with

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1559, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1545, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1394, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1526, in do_backup
    check_last_manifest(col_stats)  # not needed for full backup
  File "/usr/bin/duplicity", line 1228, in check_last_manifest
    last_backup_set.check_manifests()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 208, in check_manifests
    remote_manifest = self.get_remote_manifest()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 249, in get_remote_manifest
    return manifest.Manifest().from_string(manifest_buffer)
  File "/usr/lib64/python2.7/site-packages/duplicity/manifest.py", line 202, in from_string
    vi = VolumeInfo().from_string(match.group(1))
  File "/usr/lib64/python2.7/site-packages/duplicity/manifest.py", line 390, in from_string
    linelist = s.strip().split("
")
 MemoryError

Comment 8 Daryll 2017-12-17 18:48:48 UTC
Still happens for me with Fedora 27 Duplicity 0.7.15

Comment 9 arnaud.bawol 2018-01-25 11:01:49 UTC
Same here for fedora 27 w/ duplicity-0.7.15-1.fc27.x86_64 on afp:// mount, was the same on smb:// mount.

Comment 10 Olivier Crête 2018-01-25 15:17:42 UTC
Also for me, see discussion with upstream at https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1730451

Comment 11 Steve Rogerson 2018-01-26 12:58:25 UTC
A me too. F 27 with duplicity  0.7.15-1.fc27 Slightly different backtrace.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1559, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1545, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1394, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1420, in do_backup
    action).set_values()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 720, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 846, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 834, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 102, in add_filename
    self.set_manifest(filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 153, in set_manifest
    self.set_files_changed()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 129, in set_files_changed
    mf = self.get_manifest()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 256, in get_manifest
    return self.get_local_manifest()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 234, in get_local_manifest
    return manifest.Manifest().from_string(manifest_buffer)
  File "/usr/lib64/python2.7/site-packages/duplicity/manifest.py", line 202, in from_string
    vi = VolumeInfo().from_string(match.group(1))
 MemoryError

Comment 12 Christopher Archer 2018-02-02 04:55:04 UTC
Another me too? 
Different backtrace on automatic incremental backup

Fedora 27, duplicity 0.7.16.1.fc27, kernel 4.14.14-300.fc27.x86_64

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1559, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1545, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1394, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1527, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 669, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 430, in write_multivol
    globals.volsize)
  File "/usr/lib64/python2.7/site-packages/duplicity/gpg.py", line 385, in GPGWriteFile
    data = block_iter.next().data
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 520, in next
    result = self.process_continued()
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 700, in process_continued
    data, last_block = self.get_data_block(self.process_fp)
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 684, in get_data_block
    buf = fp.read(read_size)
  File "/usr/lib64/python2.7/site-packages/duplicity/librsync.py", line 88, in read
    self._add_to_outbuf_once()
  File "/usr/lib64/python2.7/site-packages/duplicity/librsync.py", line 98, in _add_to_outbuf_once
    self._add_to_inbuf()
  File "/usr/lib64/python2.7/site-packages/duplicity/librsync.py", line 113, in _add_to_inbuf
    assert not self.infile.close()
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 440, in close
    self.callback(self.sig_gen.getsig(), *self.extra_args)
  File "/usr/lib64/python2.7/site-packages/duplicity/librsync.py", line 232, in getsig
    return ''.join(self.sigstring_list)
 MemoryError

Comment 13 Christopher Archer 2018-02-02 04:56:35 UTC
One more thing: backup is to Samba share, and broke only after upgrade to Fedora 27 (from 26)

Comment 14 Matteo Brancaleoni 2018-02-05 08:24:32 UTC
Another me too, same stacktrace as in comment 11

Comment 15 Gwyn Ciesla 2018-02-11 20:55:08 UTC
*** Bug 1544191 has been marked as a duplicate of this bug. ***

Comment 16 Christopher Archer 2018-03-02 18:57:56 UTC
This seems to be resolved for me (for now). Found that my backup configuration was including a very large sub-directory in my ~/Downloads folder that was explicitly excluded, yet was picked up by the backup. Re-did the configuration and now things are fine. Makes me suspect that at least in my case the volume to be backed up was an issue, or perhaps just re-doing the backup configuration.

Comment 17 Gwyn Ciesla 2018-03-05 17:42:54 UTC
Excellent!