Bug 1470873 - deja-dup/duplicity backup fails with UnicodeDecodeError
deja-dup/duplicity backup fails with UnicodeDecodeError
Status: NEW
Product: Fedora
Classification: Fedora
Component: deja-dup (Show other bugs)
26
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Gwyn Ciesla
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-13 17:46 EDT by Ralf Oltmanns
Modified: 2017-11-11 21:15 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ralf Oltmanns 2017-07-13 17:46:50 EDT
Description of problem:
With the update from Fedora 25 to Fedora 26 Workstation, backups with deja-dup/duplicity are failing


Version-Release number of selected component (if applicable):
duplicity 0.7.13.1
deja-dup 34.3

How reproducible:
Always (scheduled and manually started backup)


Steps to Reproduce:
1. Wait for the scheduled backup to start or start it via the "Backup now" button
2. Some or a large number of files might get backed up, then the backup stops with "Failed with an unknown error"
3.

Actual results:
Backup fails

Expected results:
Backup completes as it always did


Additional info:
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1516, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)
Comment 1 Gwendal 2017-07-27 05:35:39 EDT
Same problem here, also on Fedora 26, when trying to perform a backup:


Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1509, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 748: ordinal not in range(128)
Comment 2 Garrett Mitchener 2017-08-28 23:15:23 EDT
I see this too:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1516, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)
Comment 3 narco85@gmail.com 2017-09-04 17:48:20 EDT
I have same error with Fedora 26 with

duplicity 0.7.13.1
deja-dup 35.5

I've find a solution that work for me in this page

https://bugs.launchpad.net/duplicity/+bug/1386373

He say: "This error can be easily fixed by replacing body of the uexc function in the utils module with the following statement:
return ufn(unicode(e).encode('utf-8'))"
Comment 4 narco85@gmail.com 2017-09-04 18:06:16 EDT
(In reply to narco85@gmail.com from comment #3)
> I have same error with Fedora 26 with
> 
> duplicity 0.7.13.1
> deja-dup 35.5
> 
> I've find a solution that work for me in this page
> 
> https://bugs.launchpad.net/duplicity/+bug/1386373
> 
> He say: "This error can be easily fixed by replacing body of the uexc
> function in the utils module with the following statement:
> return ufn(unicode(e).encode('utf-8'))"

I correct myself. 

The solution doesn't work well.


Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1504, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)
Comment 5 Eduardo Ribeiro 2017-09-28 14:55:43 EDT
I'm having the same bug/error:

[---@+++ ~]$ uname -a
Linux +++.*** 4.12.14-300.fc26.x86_64 #1 SMP Wed Sep 20 16:28:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[---@+++ ~]$ dnf info deja-dup
Última verificação de expiração de metadados: 10 days, 2:25:23 em seg 18 set 2017 17:26:58 WEST.
Pacotes Instalados
Nome         : deja-dup
Versão       : 36.1
Lançamento   : 1.fc26
Arq.         : x86_64
Tamanho      : 4.0 M
Fonte        : deja-dup-36.1-1.fc26.src.rpm
Repositório  : @System
Do repositór : updates
Resumo       : Simple backup tool and frontend for duplicity
URL          : https://launchpad.net/deja-dup
Licença      : GPLv3+
Descrição    : Déjà Dup is a simple backup tool. It hides the complexity of
             : doing backups the 'right way' (encrypted, off-site, and regular)
             : and uses duplicity as the backend.
             : 
             : Features:
             :  • Support for local, remote, or cloud backup locations (Amazon
             : S3 or Rackspace) • Securely encrypts and compresses your data
             :  • Incrementally backs up, letting you restore from any
             : particular backup • Schedules regular backups
             :  • Integrates well into your GNOME desktop

[---@+++ ~]$ dnf info duplicity
Última verificação de expiração de metadados: 10 days, 2:26:58 em seg 18 set 2017 17:26:58 WEST.
Pacotes Instalados
Nome         : duplicity
Versão       : 0.7.13.1
Lançamento   : 2.fc26
Arq.         : x86_64
Tamanho      : 2.4 M
Fonte        : duplicity-0.7.13.1-2.fc26.src.rpm
Repositório  : @System
Do repositór : updates
Resumo       : Encrypted bandwidth-efficient backup using rsync algorithm
URL          : http://www.nongnu.org/duplicity/
Licença      : GPLv2+
Descrição    : Duplicity incrementally backs up files and directory by encrypting
             : tar-format volumes with GnuPG and uploading them to a remote (or
             : local) file server. In theory many protocols for connecting to a
             : file server could be supported; so far ssh/scp, local file access,
             : rsync, ftp, HSI, WebDAV and Amazon S3 have been written.
             : 
             : Because duplicity uses librsync, the incremental archives are space
             : efficient and only record the parts of files that have changed since
             : the last backup. Currently duplicity supports deleted files, full
             : unix permissions, directories, symbolic links, fifos, device files,
             : but not hard links.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1515, in do_backup
    check_last_manifest(col_stats)  # not needed for full backup
  File "/usr/bin/duplicity", line 1219, in check_last_manifest
    last_backup_set.check_manifests()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 199, in check_manifests
    remote_manifest = self.get_remote_manifest()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 234, in get_remote_manifest
    manifest_buffer = self.backend.get_data(self.remote_manifest_name)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 677, in get_data
    fin = self.get_fileobj_read(filename, parseresults)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 669, in get_fileobj_read
    self.get(filename, tdp)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 597: ordinal not in range(128)
Comment 6 Ralf Oltmanns 2017-10-17 18:25:06 EDT
I tried running
 DEJA_DUP_DEBUG=1 deja-dup --backup | tee ~/deja-dup-backup-debug.log
hoping the debug log would show which file would contain the 0xe2 byte in its filename so that I could delete or at least exclude it.
But although the debug.log is rather verbose it wouldn't tell which file made it fail.

Is there a way to make deja-dup reveal what file brings it down?
Comment 7 Pavel Malyshev 2017-11-11 21:15:48 EST
The same issue with duplicity 0.7.13.1 on CentOS7.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1406, in do_backup
    sync_archive(decrypt)
  File "/usr/bin/duplicity", line 1146, in sync_archive
    remlist = globals.backend.list()
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 836: ordinal not in range(128)


This is the function itself and a nice comment:
def uexc(e):
    # Exceptions in duplicity often have path names in them, which if they are
    # non-ascii will cause a UnicodeDecodeError when implicitly decoding to
    # unicode.  So we decode manually, using the filesystem encoding.
    # 99.99% of the time, this will be a fine encoding to use.
    e = unicode(e).encode('utf-8')
    return ufn(str(e))

Note You need to log in before you can comment on or make changes to this bug.