Bug 1470873 - deja-dup/duplicity backup fails with UnicodeDecodeError
deja-dup/duplicity backup fails with UnicodeDecodeError
Status: ASSIGNED
Product: Fedora
Classification: Fedora
Component: deja-dup (Show other bugs)
28
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Gwyn Ciesla
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-13 17:46 EDT by Ralf Oltmanns
Modified: 2018-05-30 13:48 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1736164 None None None 2017-12-04 09:41 EST

  None (edit)
Description Ralf Oltmanns 2017-07-13 17:46:50 EDT
Description of problem:
With the update from Fedora 25 to Fedora 26 Workstation, backups with deja-dup/duplicity are failing


Version-Release number of selected component (if applicable):
duplicity 0.7.13.1
deja-dup 34.3

How reproducible:
Always (scheduled and manually started backup)


Steps to Reproduce:
1. Wait for the scheduled backup to start or start it via the "Backup now" button
2. Some or a large number of files might get backed up, then the backup stops with "Failed with an unknown error"
3.

Actual results:
Backup fails

Expected results:
Backup completes as it always did


Additional info:
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1516, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)
Comment 1 Gwendal 2017-07-27 05:35:39 EDT
Same problem here, also on Fedora 26, when trying to perform a backup:


Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1509, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 748: ordinal not in range(128)
Comment 2 Garrett Mitchener 2017-08-28 23:15:23 EDT
I see this too:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1516, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)
Comment 3 narco85@gmail.com 2017-09-04 17:48:20 EDT
I have same error with Fedora 26 with

duplicity 0.7.13.1
deja-dup 35.5

I've find a solution that work for me in this page

https://bugs.launchpad.net/duplicity/+bug/1386373

He say: "This error can be easily fixed by replacing body of the uexc function in the utils module with the following statement:
return ufn(unicode(e).encode('utf-8'))"
Comment 4 narco85@gmail.com 2017-09-04 18:06:16 EDT
(In reply to narco85@gmail.com from comment #3)
> I have same error with Fedora 26 with
> 
> duplicity 0.7.13.1
> deja-dup 35.5
> 
> I've find a solution that work for me in this page
> 
> https://bugs.launchpad.net/duplicity/+bug/1386373
> 
> He say: "This error can be easily fixed by replacing body of the uexc
> function in the utils module with the following statement:
> return ufn(unicode(e).encode('utf-8'))"

I correct myself. 

The solution doesn't work well.


Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1504, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 453, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 452, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 341, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 746: ordinal not in range(128)
Comment 5 Eduardo Ribeiro 2017-09-28 14:55:43 EDT
I'm having the same bug/error:

[---@+++ ~]$ uname -a
Linux +++.*** 4.12.14-300.fc26.x86_64 #1 SMP Wed Sep 20 16:28:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[---@+++ ~]$ dnf info deja-dup
Última verificação de expiração de metadados: 10 days, 2:25:23 em seg 18 set 2017 17:26:58 WEST.
Pacotes Instalados
Nome         : deja-dup
Versão       : 36.1
Lançamento   : 1.fc26
Arq.         : x86_64
Tamanho      : 4.0 M
Fonte        : deja-dup-36.1-1.fc26.src.rpm
Repositório  : @System
Do repositór : updates
Resumo       : Simple backup tool and frontend for duplicity
URL          : https://launchpad.net/deja-dup
Licença      : GPLv3+
Descrição    : Déjà Dup is a simple backup tool. It hides the complexity of
             : doing backups the 'right way' (encrypted, off-site, and regular)
             : and uses duplicity as the backend.
             : 
             : Features:
             :  • Support for local, remote, or cloud backup locations (Amazon
             : S3 or Rackspace) • Securely encrypts and compresses your data
             :  • Incrementally backs up, letting you restore from any
             : particular backup • Schedules regular backups
             :  • Integrates well into your GNOME desktop

[---@+++ ~]$ dnf info duplicity
Última verificação de expiração de metadados: 10 days, 2:26:58 em seg 18 set 2017 17:26:58 WEST.
Pacotes Instalados
Nome         : duplicity
Versão       : 0.7.13.1
Lançamento   : 2.fc26
Arq.         : x86_64
Tamanho      : 2.4 M
Fonte        : duplicity-0.7.13.1-2.fc26.src.rpm
Repositório  : @System
Do repositór : updates
Resumo       : Encrypted bandwidth-efficient backup using rsync algorithm
URL          : http://www.nongnu.org/duplicity/
Licença      : GPLv2+
Descrição    : Duplicity incrementally backs up files and directory by encrypting
             : tar-format volumes with GnuPG and uploading them to a remote (or
             : local) file server. In theory many protocols for connecting to a
             : file server could be supported; so far ssh/scp, local file access,
             : rsync, ftp, HSI, WebDAV and Amazon S3 have been written.
             : 
             : Because duplicity uses librsync, the incremental archives are space
             : efficient and only record the parts of files that have changed since
             : the last backup. Currently duplicity supports deleted files, full
             : unix permissions, directories, symbolic links, fifos, device files,
             : but not hard links.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1515, in do_backup
    check_last_manifest(col_stats)  # not needed for full backup
  File "/usr/bin/duplicity", line 1219, in check_last_manifest
    last_backup_set.check_manifests()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 199, in check_manifests
    remote_manifest = self.get_remote_manifest()
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 234, in get_remote_manifest
    manifest_buffer = self.backend.get_data(self.remote_manifest_name)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 677, in get_data
    fin = self.get_fileobj_read(filename, parseresults)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 669, in get_fileobj_read
    self.get(filename, tdp)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 597: ordinal not in range(128)
Comment 6 Ralf Oltmanns 2017-10-17 18:25:06 EDT
I tried running
 DEJA_DUP_DEBUG=1 deja-dup --backup | tee ~/deja-dup-backup-debug.log
hoping the debug log would show which file would contain the 0xe2 byte in its filename so that I could delete or at least exclude it.
But although the debug.log is rather verbose it wouldn't tell which file made it fail.

Is there a way to make deja-dup reveal what file brings it down?
Comment 7 Pavel Malyshev 2017-11-11 21:15:48 EST
The same issue with duplicity 0.7.13.1 on CentOS7.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1406, in do_backup
    sync_archive(decrypt)
  File "/usr/bin/duplicity", line 1146, in sync_archive
    remlist = globals.backend.list()
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 376, in inner_retry
    % exception_traceback())
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 51, in exception_traceback
    return uexc(msg)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 79, in uexc
    e = unicode(e).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 836: ordinal not in range(128)


This is the function itself and a nice comment:
def uexc(e):
    # Exceptions in duplicity often have path names in them, which if they are
    # non-ascii will cause a UnicodeDecodeError when implicitly decoding to
    # unicode.  So we decode manually, using the filesystem encoding.
    # 99.99% of the time, this will be a fine encoding to use.
    e = unicode(e).encode('utf-8')
    return ufn(str(e))
Comment 8 Ralf Oltmanns 2017-12-02 16:02:58 EST
The problem persists in Fedora 27 Workstation:

$ uname -a
Linux hostname 4.13.16-300.fc27.x86_64 #1 SMP Mon Nov 27 18:19:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -qi deja-dup
Name        : deja-dup
Version     : 37.0
Release     : 1.fc27
Architecture: x86_64
Install Date: Fri 01 Dec 2017 04:52:44 PM CET
Group       : Applications/Archiving
Size        : 4213682
License     : GPLv3+
Signature   : RSA/SHA256, Mon 20 Nov 2017 04:25:27 PM CET, Key ID f55e7430f5282ee4
Source RPM  : deja-dup-37.0-1.fc27.src.rpm
Build Date  : Mon 20 Nov 2017 04:03:58 PM CET
Build Host  : buildvm-30.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://launchpad.net/deja-dup

$ rpm -qi duplicity
Name        : duplicity
Version     : 0.7.15
Release     : 1.fc27
Architecture: x86_64
Install Date: Tue 28 Nov 2017 09:11:38 AM CET
Group       : Unspecified
Size        : 2476520
License     : GPLv2+
Signature   : RSA/SHA256, Tue 14 Nov 2017 03:22:29 PM CET, Key ID f55e7430f5282ee4
Source RPM  : duplicity-0.7.15-1.fc27.src.rpm
Build Date  : Tue 14 Nov 2017 03:16:59 PM CET
Build Host  : buildvm-30.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.nongnu.org/duplicity/
Comment 9 Gwyn Ciesla 2017-12-04 09:41:24 EST
Filed bug upstream.
Comment 10 jonathan.m.eldridge 2018-02-18 14:31:09 EST
I am also having this same problem on Ubuntu 16.04 LTS with duplicity 0.7.06 and deja-dup 34.2.
I backup my files (using the "backup now" button) to an external hard drive every Sunday. Last week's backup was fine, but during the week I let Ubuntu install updates. My computer had not asked to run an update in a few months.
Additionally, I tried uninstalling (sudo apt-get purge) and reinstalling both duplicity and deja-dup, and then restarting. The issue still persists, and neither program was updated to a newer version.
Unlike the original poster, however, the backup fails immediately. This is the text I get:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1532, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1526, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1380, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1401, in do_backup
    sync_archive(decrypt)
  File "/usr/bin/duplicity", line 1139, in sync_archive
    remote_metafiles, ignored, rem_needpass = get_metafiles(remlist)
  File "/usr/bin/duplicity", line 1029, in get_metafiles
    pr = file_naming.parse(fn)
  File "/usr/lib/python2.7/dist-packages/duplicity/file_naming.py", line 400, in parse
    pr = check_inc()
  File "/usr/lib/python2.7/dist-packages/duplicity/file_naming.py", line 340, in check_inc
    t1 = str2time((m1 or m2).group("start_time"), short)
  File "/usr/lib/python2.7/dist-packages/duplicity/file_naming.py", line 290, in str2time
    t = dup_time.genstrtotime(timestr.upper())
  File "/usr/lib/python2.7/dist-packages/duplicity/dup_time.py", line 295, in genstrtotime
    return override_curtime - intstringtoseconds(timestr)
  File "/usr/lib/python2.7/dist-packages/duplicity/dup_time.py", line 203, in intstringtoseconds
    error()
  File "/usr/lib/python2.7/dist-packages/duplicity/dup_time.py", line 194, in error
    raise TimeException(bad_interval_string % interval_string)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 12: ordinal not in range(128)
Comment 11 Ralf Oltmanns 2018-05-02 17:24:50 EDT
The problem still exists on Fedora 28 x86_64
[ralf@iroquois ~]$ uname -a
Linux iroquois 4.16.5-300.fc28.x86_64 #1 SMP Fri Apr 27 17:38:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[ralf@iroquois ~]$ rpm -qi duplicity
Name        : duplicity
Version     : 0.7.17
Release     : 2.fc28
Architecture: x86_64
Install Date: Wed 02 May 2018 10:37:51 AM CEST
Group       : Unspecified
Size        : 2534725
License     : GPLv2+
Signature   : RSA/SHA256, Mon 26 Feb 2018 08:26:50 PM CET, Key ID e08e7e629db62fb1
Source RPM  : duplicity-0.7.17-2.fc28.src.rpm
Build Date  : Mon 26 Feb 2018 08:22:02 PM CET
Build Host  : buildhw-07.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.nongnu.org/duplicity/
Bug URL     : https://bugz.fedoraproject.org/duplicity
Summary     : Encrypted bandwidth-efficient backup using rsync algorithm
Description :
Duplicity incrementally backs up files and directory by encrypting
tar-format volumes with GnuPG and uploading them to a remote (or
local) file server. In theory many protocols for connecting to a
file server could be supported; so far ssh/scp, local file access,
rsync, ftp, HSI, WebDAV and Amazon S3 have been written.

Because duplicity uses librsync, the incremental archives are space
efficient and only record the parts of files that have changed since
the last backup. Currently duplicity supports deleted files, full
unix permissions, directories, symbolic links, fifos, device files,
but not hard links.

[ralf@iroquois ~]$ rpm -qi deja-dup
Name        : deja-dup
Version     : 38.0
Release     : 1.fc28
Architecture: x86_64
Install Date: Wed 02 May 2018 10:37:53 AM CEST
Group       : Applications/Archiving
Size        : 4278028
License     : GPLv3+
Signature   : RSA/SHA256, Tue 10 Apr 2018 03:39:48 PM CEST, Key ID e08e7e629db62fb1
Source RPM  : deja-dup-38.0-1.fc28.src.rpm
Build Date  : Tue 10 Apr 2018 03:23:50 PM CEST
Build Host  : buildvm-02.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://launchpad.net/deja-dup
Bug URL     : https://bugz.fedoraproject.org/deja-dup
Summary     : Simple backup tool and frontend for duplicity
Description :
Déjà Dup is a simple backup tool. It hides the complexity of doing backups the
'right way' (encrypted, off-site, and regular) and uses duplicity as the
backend.

Features:
 • Support for local, remote, or cloud backup locations (Amazon S3 or Rackspace)
 • Securely encrypts and compresses your data
 • Incrementally backs up, letting you restore from any particular backup
 • Schedules regular backups
 • Integrates well into your GNOME desktop
Comment 12 Levan 2018-05-30 13:48:32 EDT
i conform this problemon fedora 28.

[root@levglonti-toshiba ~]# rpm -qi deja-dup
Name        : deja-dup
Version     : 38.0
Release     : 2.fc28
Architecture: x86_64
Install Date: 2018 წლის 20 მაისი, 19:00:01 +04
Group       : Applications/Archiving
Size        : 4258724
License     : GPLv3+
Signature   : RSA/SHA256, 2018 წლის 10 მაისი, 17:45:57 +04, Key ID e08e7e629db62fb1
Source RPM  : deja-dup-38.0-2.fc28.src.rpm
Build Date  : 2018 წლის 10 მაისი, 17:36:15 +04
Build Host  : buildvm-14.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://launchpad.net/deja-dup
Bug URL     : https://bugz.fedoraproject.org/deja-dup
Summary     : Simple backup tool and frontend for duplicity
Description :
Déjà Dup is a simple backup tool. It hides the complexity of doing backups the
'right way' (encrypted, off-site, and regular) and uses duplicity as the
backend.

Features:
 • Support for local, remote, or cloud backup locations (Amazon S3 or Rackspace)
 • Securely encrypts and compresses your data
 • Incrementally backs up, letting you restore from any particular backup
 • Schedules regular backups
 • Integrates well into your GNOME desktop
[root@levglonti-toshiba ~]# 
[root@levglonti-toshiba ~]# uname -a
Linux levglonti-toshiba 4.16.12-300.fc28.x86_64 #1 SMP Fri May 25 21:13:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@levglonti-toshiba ~]#

Note You need to log in before you can comment on or make changes to this bug.