Bug 1669083

Summary: borgbackup fails in F30 due to msgpack update
Product: [Fedora] Fedora Reporter: Kevin Fenzi <kevin>
Component: borgbackupAssignee: Felix Schwarz <fschwarz>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: 5jxusjgarm, 7d28c752, abo, antoine.gemis, arjunak234, bonfostar, bugzilla.redhat.com, bzf, cfergeau, c.justin88, clasohm, craig, dev, dgibson, fredcwells, frog, fschwarz, hontonoroger+redhat, jonha87, jorti, laurent, mail, mh, mmello, peter.hutterer, pilux, rafael, reklov, releng, ricardo.arguello, Simon.Gerhards, thomas.moschny, timo, tomek, tom+f, vaaghoofdharry, vic, wpinheir
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: borgbackup-1.1.10-2.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-18 01:01:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1710715    
Bug Blocks: 1674516, 1683400    

Description Kevin Fenzi 2019-01-24 09:52:07 UTC
File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.0 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3112, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3096, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3125, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 580, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 593, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 781, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup

Comment 1 Felix Schwarz 2019-01-24 10:01:06 UTC
Ah, that is too bad as this is likely a "real" incompatibility. msgpack 0.6 is only supported in borg's master branch [1] and upstream explicitly advises against using it in production [2].

Actually I sent an email on 2019-01-17 14:41 UTC+1 to python-msgpack-owner *exactly* because I feared this might becomea  problem and asked if they could coordinate with Fedora's borgbackup maintainer if they planned an upgrade.

Now it seems as if my message got lost as Fabian Affolter pushed the 0.6 update on 2019-01-18 08:46 UTC.

Let's see what we can do here, adding Fabian as he might have an idea.

[1] https://github.com/borgbackup/borg/issues/4251
[2] https://github.com/borgbackup/borg/issues/4251#issuecomment-453777837

Comment 2 Fabian Affolter 2019-01-26 12:40:19 UTC
For Rawhide is only borgbackup 1.1.7 and not 1.1.8 available [1].

The latest version of borgbackup 1.1.8 was build at the end of December. In 1.1.8 was support added for python-msgpack 0.6.0 thus I postponed the update of python-msgpack because I was waiting for the rawhide build. As there was no build available after 2.5 week I updated python-msgpack in rawhide because back then python-msgpack 0.6.0 was out for six weeks and included security fixes.

[1] https://apps.fedoraproject.org/packages/borgbackup

Comment 3 Kevin Fenzi 2019-01-30 18:48:39 UTC
1.1.8 was committed in december, but failed to build:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1182746

Looks like a failure in tests. 
=================================== FAILURES ===================================
__________________ ArchiverTestCase.test_corrupted_repository __________________
self = <borg.testsuite.archiver.ArchiverTestCase testMethod=test_corrupted_repository>
    def test_corrupted_repository(self):
        self.cmd('init', '--encryption=repokey', self.repository_location)
        self.create_src_archive('test')
        self.cmd('extract', '--dry-run', self.repository_location + '::test')
        output = self.cmd('check', '--show-version', self.repository_location)
        self.assert_in('borgbackup version', output)  # implied output even without --info given
>       self.assert_not_in('Starting repository check', output)  # --info not given for root logger
E       AssertionError: 'Starting repository check' unexpectedly found in 'borgbackup version 1.1.8\nStarting repository check\nStarting repository index check\nCompleted repository check, no problems found.\n'
build/lib.linux-x86_64-3.7/borg/testsuite/archiver.py:1494: AssertionError
 1 failed, 2233 passed, 310 skipped, 142 deselected, 4 xfailed in 909.07 seconds 
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.sHksSs (%check)
BUILDSTDERR:     Bad exit status from /var/tmp/rpm-tmp.sHksSs (%check)

Seems like https://github.com/borgbackup/borg/issues/3805 but that was already committed to 1.1.8, so not sure.

Comment 4 Benjamin Pereto 2019-01-30 20:12:22 UTC
Sorry folks, forgot to build 1.1.8 for rawhide.

The build previously failed with some buildroot issues, but now its this failing test.
I saw the same Issue as Kevin and can be related to it.

Not shure if we can xfail this test, better report this upstream.

Comment 5 Benjamin Pereto 2019-01-30 20:35:27 UTC
reported upstream:
https://github.com/borgbackup/borg/issues/4287

Shot in the blue:
`setup_logging` contains also a logging.clear(), removing it lets all tests pass for testsuite/archvier.py

--- a/src/borg/testsuite/archiver.py	2018-12-09 03:18:19.000000000 +0100
+++ b/src/borg/testsuite/archiver.py.b	2019-01-30 21:18:48.973311799 +0100
@@ -279,7 +279,7 @@
         os.chdir(self._old_wd)
         # note: ignore_errors=True as workaround for issue #862
         shutil.rmtree(self.tmpdir, ignore_errors=True)
-        setup_logging()
+        #setup_logging()
 
     def cmd(self, *args, **kw):
         exit_code = kw.pop('exit_code', 0)

Comment 6 Felix Schwarz 2019-02-11 16:57:51 UTC
*** Bug 1674709 has been marked as a duplicate of this bug. ***

Comment 7 Benjamin Pereto 2019-03-09 11:35:18 UTC
I patched out the logging check until we figured out the issue about the logging.
seems to be the only test failing because of this.

Comment 8 Benjamin Pereto 2019-03-09 11:45:58 UTC
The Issue with msgpack persists. Only the upcoming release of borgbackup 1.2 supports msgpack > 0.5.6 (ex. 0.6)
So until the release of borgbackup 1.2 there are no further updates possible for borg in f30 and rawhide.
Especially the current Releases in f30 and rahwhide can be broken, due the wrong msgpack version.

Comment 9 Christophe Fergeau 2019-03-21 10:21:49 UTC
Is borgbackup 1.2.0 going to be released before f30 release? If not, and assuming the msgpack 0.6.0 support cannot be backported, should a python-msgpack056 compat package be introduced?

Comment 10 Peter Hutterer 2019-04-04 04:41:48 UTC
*** Bug 1694426 has been marked as a duplicate of this bug. ***

Comment 11 Christophe Fergeau 2019-04-04 07:13:55 UTC
Fwiw, I can workaround this on my f30 system with
dnf downgrade https://kojipkgs.fedoraproject.org//packages/python-msgpack/0.5.6/5.fc29/x86_64/python3-msgpack-0.5.6-5.fc29.x86_64.rpm
as I have nothing else depending on this package on my install.

Comment 12 Juan Orti 2019-04-29 14:18:04 UTC
Similar problem has been detected:

Running:

borg list ::

reporter:       libreport-2.10.0
cmdline:        /usr/bin/python3 /usr/bin/borg list ::
crash_function: resolve
exception_type: pkg_resources.ContextualVersionConflict
executable:     /usr/bin/borg
interpreter:    python3-3.7.3-1.fc30.x86_64
kernel:         5.0.9-301.fc30.x86_64
package:        borgbackup-1.1.7-2.fc30
reason:         __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
runlevel:       N 5
type:           Python3
uid:            0

Comment 13 Marcelo Moreira de Mello 2019-05-01 00:22:37 UTC
 Hitting the same problem on Fedora 30 SilverBlue

[root@storm ~]# rpm -qi borgbackup
Name        : borgbackup
Version     : 1.1.7
Release     : 2.fc30
Architecture: x86_64
Install Date: Tue 30 Apr 2019 06:13:18 PM EDT
Group       : Unspecified
Size        : 4337397
License     : BSD
Signature   : RSA/SHA256, Sun 23 Sep 2018 08:47:49 PM EDT, Key ID ef3c111fcfc659b9
Source RPM  : borgbackup-1.1.7-2.fc30.src.rpm
Build Date  : Sun 23 Sep 2018 08:27:34 PM EDT
Build Host  : buildvm-27.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://borgbackup.readthedocs.org
Bug URL     : https://bugz.fedoraproject.org/borgbackup
Summary     : A deduplicating backup program with compression and authenticated encryption
Description :
BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it
supports compression and authenticated encryption.

[root@storm ~]# borg list
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup


[root@storm ~]# rpm-ostree  status -v
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-workstation:fedora/30/x86_64/silverblue
                   Version: 30.20190427.n.0 (2019-04-27T08:57:18Z)
                BaseCommit: a602958b1097d0790361a8b907f6f6ecbb62b7bbf826b086401eafb4ffcb845d
                            ├─ repo-0 ((invalid timestamp))
                            └─ repo-1 ((invalid timestamp))
                    Commit: e60764a2f28e25c3adfb9479a41e8b009dec18d3d8dfa38552cb12ebea2e2039
                            ├─ fedora-cisco-openh264 (2019-03-21T15:16:16Z)
                            ├─ rpmfusion-free (2019-04-16T20:46:20Z)
                            ├─ rpmfusion-free-updates (2019-04-29T08:32:13Z)
                            ├─ updates (2019-04-30T01:11:12Z)
                            ├─ rpmfusion-nonfree-steam (2019-02-27T18:49:45Z)
                            ├─ rpmfusion-nonfree-nvidia-driver (2019-03-25T17:44:44Z)
                            ├─ rpmfusion-nonfree (2019-04-16T21:53:39Z)
                            ├─ fedora (2019-04-27T09:42:56Z)
                            ├─ google-chrome (2019-04-30T19:45:42Z)
                            └─ rpmfusion-nonfree-updates (2019-04-29T09:15:15Z)
                    Staged: no
                 StateRoot: fedora-workstation
              GPGSignature: 1 signature
                            Signature made Sat 27 Apr 2019 04:57:30 AM EDT using RSA key ID EF3C111FCFC659B9
                            Good signature from "Fedora <fedora-30-primary>"
          InactiveRequests: fedora-toolbox
           LayeredPackages: NetworkManager-l2tp borgbackup fedora-workstation-repositories fish fuse-exfat
                            git gnome-tweaks google-chrome-stable knock krb5-workstation libvirt
                            libvirt-daemon net-tools oathtool qemu-img qemu-kvm snapd synergy terminator
                            tilix tmux vim-enhanced virt-manager xclip xdotool xl2tpd xorg-x11-drv-nvidia

Comment 14 Marcelo Moreira de Mello 2019-05-01 01:13:49 UTC
I was able to work around the issue as posted on c#11 (https://bugzilla.redhat.com/show_bug.cgi?id=1669083#c11)

Comment 15 rugk 2019-05-01 16:41:45 UTC
Borg breaks in Fedora 30… And downgrading is no solution, so please try to fix this soon. This is a backup system after all, if this does not work, you have a problem…

```
$ borg -V
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
```

Comment 16 Albert Vernon 2019-05-01 21:02:45 UTC
According to the Borgbackup release notes, version 1.1.8 or later is compatible with python3-msgpack 0.6.1 that ships with Fedora 30. Looks like building the latest from upstream will fix this problem.

Comment 17 Felix Schwarz 2019-05-02 06:47:02 UTC
(In reply to Albert Vernon from comment #16)
> According to the Borgbackup release notes, version 1.1.8 or later is
> compatible with python3-msgpack 0.6.1 that ships with Fedora 30. Looks like
> building the latest from upstream will fix this problem.

Could you please provide a link where you read this?

To the best of my knowledge only borgbackup 1.2 (currently alpha) supports msgpack 0.6.1. That version was whitelisted in https://github.com/borgbackup/borg/commit/0ebfaa5b61a675c22cda301bc20d0b00372dd181 and I don't see any code in borgbackup 1.1.x which supports your claim.

Actually upstream is pretty clear that only borgbackup 1.2 will support msgpack 0.6+ ( https://github.com/borgbackup/borg/issues/3753#issuecomment-454011810 ).

I fear there is no elegant solution to this. Anyways Fedora 30 has been released and borgbackup is broken. We need to deal with this somehow:
1. retire borgbackup in F30 until borgbackup 1.2 is released (current version is 1.2.0 alpha6).
2. create a new package python3-msgpack05 with conflicts python3-msgpack and make Fedora's borgbackup dependent on the new package
3. vendorize a copy of msgpack 0.5.6, add a bit of import magic in borg and ship that.

@bpereto: Any preference/ideas how to proceed?

Comment 18 harry 2019-05-02 09:47:59 UTC
Similar problem has been detected:

I tried running "borg list <backup-location>" to see if my backup solution works after upgrading to fedora 30. 
(Note: I installed borgmatic from a fc29 rpm because installing it via the official repos gives a dependency error of python3-ruamel-yaml)

reporter:       libreport-2.10.0
cmdline:        /usr/bin/python3 /usr/bin/borg list /mnt/hd/linux/backups/pc/borg-backup
crash_function: resolve
exception_type: pkg_resources.ContextualVersionConflict
executable:     /usr/bin/borg
interpreter:    python3-3.7.3-1.fc30.x86_64
kernel:         5.0.9-301.fc30.x86_64
package:        borgbackup-1.1.7-2.fc30
reason:         __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
runlevel:       N 5
type:           Python3
uid:            1000

Comment 19 Felix Schwarz 2019-05-02 12:02:31 UTC
*** Bug 1705487 has been marked as a duplicate of this bug. ***

Comment 20 Albert Vernon 2019-05-02 14:36:47 UTC
I was mistaken. Borg commit 0ebfaa5 for ticket #4308 is in version 1.2 like you said and not 1.18 as I stated.

Thank you for the clarification.

Comment 21 Simo_Bonfo 2019-05-02 20:13:35 UTC
Similar problem has been detected:

Just install borgbackup on F30 and try to use it.
This problem happens with every borg command.

reporter:       libreport-2.10.0
cmdline:        /usr/bin/python3 /usr/bin/borg init --encryption=repokey Fedora-Backups/
crash_function: resolve
exception_type: pkg_resources.ContextualVersionConflict
executable:     /usr/bin/borg
interpreter:    python3-3.7.3-1.fc30.x86_64
kernel:         5.0.9-301.fc30.x86_64
package:        borgbackup-1.1.7-2.fc30
reason:         __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
runlevel:       N 5
type:           Python3
uid:            1000

Comment 22 Thomas Moschny 2019-05-03 08:09:36 UTC
(In reply to Felix Schwarz from comment #17)
> I fear there is no elegant solution to this. Anyways Fedora 30 has been
> released and borgbackup is broken. We need to deal with this somehow:
> 1. retire borgbackup in F30 until borgbackup 1.2 is released (current
> version is 1.2.0 alpha6).
> 2. create a new package python3-msgpack05 with conflicts python3-msgpack and
> make Fedora's borgbackup dependent on the new package
> 3. vendorize a copy of msgpack 0.5.6, add a bit of import magic in borg and
> ship that.

To me, (3) looks like a good solution.

Another possibility would be to put 1.2.0 alpha into F30.

Comment 23 Felix Schwarz 2019-05-03 08:15:36 UTC
(In reply to Thomas Moschny from comment #22)
> Another possibility would be to put 1.2.0 alpha into F30.

Upstream says: "borgbackup 1.2.0a6 alpha release - do not use this on production backup repositories." (https://github.com/borgbackup/borg/releases). I think upstream is just cautious but on the other hand users entrust Fedora/borgbackup to provide a safety net and corrupt backup data can cause significant losses.

> To me, (3) looks like a good solution.

That's my preferred solution as well but it needs someone to fiddle with the code. Also neither is a really good solution because upstream has no support/infrastructure to vendorize msgpack. This means we need to code something ourself which is a delicate thing to do in a backup software. (Also it costs time and I don't know if I have enough of that in the next days to implement something like that.)

Comment 24 Albert Vernon 2019-05-04 01:02:41 UTC
After downgrading to python3-msgpack 0.5.6 using Christophe's suggestion, I added the following line to the [main] section of /etc/dnf/dnf.conf to keep `dnf upgrade` from re-installing 0.6.1:

excludepkgs=python3-msgpack

Comment 25 Fred Wells 2019-05-04 19:14:42 UTC
So, a python3-msgpack 0.5.6 downgrade is IMO a suitable workaround for the many users currently stranded after F30 upgrade, assuming there is a fc30 build of same.

Comment 26 Fred Wells 2019-05-04 19:37:25 UTC
Following Albert's lead, I also downgraded (to the F29 version @ 0.5.6-5.fc29.x86_64) and is working again.

Comment 27 Christophe Fergeau 2019-05-06 12:03:00 UTC
(In reply to Felix Schwarz from comment #17)

> I fear there is no elegant solution to this. Anyways Fedora 30 has been
> released and borgbackup is broken. We need to deal with this somehow:
> 1. retire borgbackup in F30 until borgbackup 1.2 is released (current
> version is 1.2.0 alpha6).
> 2. create a new package python3-msgpack05 with conflicts python3-msgpack and
> make Fedora's borgbackup dependent on the new package
> 3. vendorize a copy of msgpack 0.5.6, add a bit of import magic in borg and
> ship that.

4. downgrade the python3-msgpack package in fedora 30 to 0.5.6 (through the use of an epoch)
imo this is better than 2, unless something else really needs 0.6.0

Comment 28 Felix Schwarz 2019-05-06 15:40:24 UTC
(In reply to Christophe Fergeau from comment #27)
> 4. downgrade the python3-msgpack package in fedora 30 to 0.5.6 (through the
> use of an epoch) imo this is better than 2, unless something else really needs 0.6.0

I omitted that option because I think Fabian deliberately updated python3-msgpack in January (see comment 2):
> I updated python-msgpack in rawhide because back then python-msgpack 0.6.0 was out for six weeks and included security fixes.

Yesterday I tried (very briefly) to bundle a pure python version of msgpack (slow) but so far without much success. If there is someone who wants to invest some time I can upload a git repo (basic Python skills needed).

Comment 29 David Gibson 2019-05-08 01:33:20 UTC
Raising priority, since it essentially makes borgbackup wholly unusable.

Comment 30 Laurent Meunier 2019-05-09 04:41:58 UTC
Similar problem has been detected:

I've tried to run borgbackup, just as I was doing with Fedora 29.
There seems to be an issue with Fedora 30 and the msgpack package:
https://github.com/borgbackup/borg/issues/3889

reporter:       libreport-2.10.0
cmdline:        /usr/bin/python3 /usr/bin/borg create -v --stats --compression lz4 --show-rc --show-version borgbackup.net:/data/backup/spring.local::{hostname}-{now} /home/laurent/.ssh/ /home/laurent/.thunderbird/ /home/laurent/Devel/ /home/laurent/Documents/ /home/laurent/Pictures/ /home/laurent/_tmp/
crash_function: resolve
exception_type: pkg_resources.ContextualVersionConflict
executable:     /usr/bin/borg
interpreter:    python3-3.7.3-1.fc30.x86_64
kernel:         5.0.10-300.fc30.x86_64
package:        borgbackup-1.1.7-2.fc30
reason:         __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
runlevel:       N 5
type:           Python3
uid:            1000

Comment 31 Felix Schwarz 2019-05-09 07:49:12 UTC
Ok, I got borgbackup to pass all the unit tests with an bundled copy of msgpack 0.5.6. I'll try to get this packaged in a COPR this evening. It will needs quite a bit of user testing before even submitting it to the testing repo.

Just a few notes on why this was more complicated than expected:
- tests for borgbackup 1.1.9 fail with the pure-python msgpack
- need to compile msgpack's Cython as part of borg's build so it gets the right module names
- msgpack uses Cython -> C++ while borg uses Cython->C.

Comment 32 Felix Schwarz 2019-05-09 21:34:50 UTC
Here is a COPR which you can try https://copr.fedorainfracloud.org/coprs/fschwarz/bundled-borgbackup/

I DID NOT TEST the built binaries. DO NOT USE on production data. As you can see in the RPM I did not really change the code but "msgpack" is really at the heart of borg's data storage so any screw up here could cause loss of (backup) data.

Also I got errors in COPR when running on ppc64le (https://copr.fedorainfracloud.org/coprs/fschwarz/bundled-borgbackup/build/906877/). If someone more knowledgeable could provide some insight that would be appreciated.

Comment 33 Felix Schwarz 2019-05-10 06:55:07 UTC
The COPR failure for ppc64le seems to be just COPR related as a koji build succeeded (https://koji.fedoraproject.org/koji/taskinfo?taskID=34754384). s390x seems to be hanging.

I did a quick test on F30 and the new RPM worked for me (create repo, do a backup, restore). However I want to have at least one or two users trying out the RPM before I think about pushing it to Fedora testing.

Comment 34 Laurent Meunier 2019-05-10 07:50:39 UTC
I also did a quick test on F30 wit the new RPM from COPR fschwarz/bundled-borgbackup (only a backup to an existing repo). No more errors, and the backup finished successfully.

Comment 35 Felix Schwarz 2019-05-10 10:53:16 UTC
Thanks, Laurent. Ideally we'll one or two more testers and then we'll push the new code to testing.

Comment 36 Albert Vernon 2019-05-10 19:05:12 UTC
The following operations worked for me using fschwarz/bundled-borgbackup:

borg info
borg check
borg list
borg create
borg prune
borg mount

Comment 37 Felix Schwarz 2019-05-10 20:10:13 UTC
Thank you Albert. I submitted a Koji build for rawhide. s390x seems to be hanging in my scratch builds, let's hope that the real build will be fine (https://koji.fedoraproject.org/koji/taskinfo?taskID=34771060). I'll submit a build for F30 once rawhide is fine.

Comment 38 Thomas Moschny 2019-05-11 11:10:46 UTC
(In reply to Felix Schwarz from comment #37)
> s390x seems to be hanging in my scratch builds

A local, qemu-powered build for s390x does hang, but shows a test failure:

# fedpkg switch-branch master
# MOCKARGS="--forcearch s390x" fedpkg mockbuild --mock-config fedora-rawhide-s390x

[...]
build/lib.linux-s390x-3.7/borg/testsuite/archiver.py::RemoteArchiverTestCase::test_compression_auto_compressible FAILED [ 27%]

=================================== FAILURES ===================================
__________ RemoteArchiverTestCase.test_compression_auto_compressible ___________

self = <borg.testsuite.archiver.RemoteArchiverTestCase testMethod=test_compression_auto_compressible>

    def test_compression_auto_compressible(self):
        size, csize = self._get_sizes('auto,zlib', compressible=True)
>       assert csize < size * 0.1
E       AssertionError: assert 0 < (0 * 0.1)

build/lib.linux-s390x-3.7/borg/testsuite/archiver.py:2078: AssertionError
= 1 failed, 203 passed, 151 skipped, 71 deselected, 1 xfailed in 325.34 seconds =
error: Bad exit status from /var/tmp/rpm-tmp.ZBQpzF (%check)

Maybe the koji builds simply fail to detect the failure...

Comment 39 Felix Schwarz 2019-05-11 13:05:29 UTC
Thank you Thomas for looking into this. Can you verify that the msgpack Cython extension (_packer/_unpacker) is built correctly? (warnings about language level are expected)

I saw this test failure also on x86_64 when using the pure-python msgpack with borg's test suite.

Comment 40 Thomas Moschny 2019-05-11 14:19:42 UTC
Taking that back. What seems to happen is that qemu emits 'Unsupported setsockopt level=270 optname=3' messages to stderr which wrongly get interpreted by the testsuite as the output of the 'borg list' command.

FTR, _packer.cpython-37m-s390x-linux-gnu.so and _unpacker.cpython-37m-s390x-linux-gnu.so binaries are present.

(And btw, the corresponding .cpp files probably should not be installed, but that's really a minor thing.)

Comment 41 Felix Schwarz 2019-05-13 07:46:03 UTC
Update: upstream added some commits which might prevent hangs in the test suite. I hope to get access to a s390x machine soon so I can try adding a few patches. If that fails I'll (temporarily) exclude s390x and push an update for all other architectures so you get a working borgbackup again.

Comment 42 Marcel Haerry 2019-05-13 09:42:16 UTC
Same happens in EPEL7 atm.

Comment 43 Felix Schwarz 2019-05-13 09:49:17 UTC
(In reply to Marcel Haerry from comment #42)
> Same happens in EPEL7 atm.

EPEL7 should still use msgpack 0.5.6 so I think you are experiencing a different issue.

Please file a new issue and provide additional info.

Comment 44 Marcel Haerry 2019-05-13 10:01:23 UTC
(In reply to Felix Schwarz from comment #43)
> (In reply to Marcel Haerry from comment #42)
> > Same happens in EPEL7 atm.
> 
> EPEL7 should still use msgpack 0.5.6 so I think you are experiencing a
> different issue.
> 
> Please file a new issue and provide additional info.

You are right and I made a wrong alarm, as I misread from which machine I got the output of the failed update. Sorry and thanks for the hint.

Comment 45 hontonoroger+redhat 2019-05-15 18:25:51 UTC
Similar problem has been detected:

I just called "borg" and it happened.

reporter:       libreport-2.10.0
cmdline:        /usr/bin/python3 /usr/bin/borg
crash_function: resolve
exception_type: pkg_resources.ContextualVersionConflict
executable:     /usr/bin/borg
interpreter:    python3-3.7.3-1.fc30.x86_64
kernel:         5.0.14-300.fc30.x86_64
package:        borgbackup-1.1.7-2.fc30
reason:         __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
runlevel:       N 5
type:           Python3
uid:            1000

Comment 46 Felix Schwarz 2019-05-16 07:27:39 UTC
upstream added some changes to fix problems I found while working on this bug. There is even a new upstream release 1.1.10 which makes packaging much easier (I hit several issues while trying to package a git snapshot of borgbackup's "1.1-maint" branch.

However the (sloooow) koji builders still see test failures on various architectures. I'll investigate as part of bug 1710715 and will push an update as soon as possible.

Comment 47 Felix Schwarz 2019-05-16 10:20:25 UTC
borgbackup 1.1.10 pushed+built for rawhide+F29. I'll keep this bug open specifically to fix F30.

Unfortunately I can't get the test suite to pass on F30 (in mock/Koji). I published my current state at https://github.com/FelixSchwarz/borg-fedora/issues/1 - any help is appreciated. I even considered just disabling the failing test but I'm hesitant to do so - even though the current situation (borgbackup being completely broken in F30) is really [censored].

Comment 48 Fedora Update System 2019-05-16 18:23:01 UTC
borgbackup-1.1.10-2.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-90776185cd

Comment 49 Felix Schwarz 2019-05-16 18:27:08 UTC
As the previous comment indicated I finally succeeded to build borgbackup 1.1.10 for Fedora 30. The update should hit updates-testing soon. Please test the new version and report your findings in bodhi (https://bodhi.fedoraproject.org/updates/FEDORA-2019-90776185cd).

I'm very sorry that we shipped Fedora 30 with a broken version of borgbackup and that it took me so long to fix the package properly.

Unfortunately this was not just a simple "minor update" but involved some coding, scripting and quite a few builds on Fedora's infrastructure (roughly a dozen builds ran on 6 supported architectures). I experienced test suite hangs on s390x and (today) failing builds on F30 (while F29 and rawhide just worked fine).

As part of fixing this I filed 3 issues in upstream bug trackers and created 2 pull requests. I have to say I was pretty frustrated at times as it seemed as if there was always another bug waiting for me whenever I made some progress.

Despite all this I'm really glad I got a lot of help: Three different Fedora developers spent time on investigating the s390x hangs. Upstream developers of borgbackup and msgpack merged my pull requests. Especially Thomas Waldmann (main developer of borgbackup) put in extra hours at night to improve my quickly hacked bundling code for msgpack, merging it upstream and publishing a new release just in time.

I'm telling you this because I feel I owe you an apology. To me Fedora is about shipping stable software very quickly - with Fedora 30 we failed to do so for borgbackup but I hope things will go smoother in the future. Thank you for your patience.

Comment 50 Fedora Update System 2019-05-17 15:38:02 UTC
borgbackup-1.1.10-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-90776185cd

Comment 51 Marcelo Moreira de Mello 2019-05-17 16:41:35 UTC
[root@41c76f715934 /]# cat /etc/fedora-release 
Fedora release 30 (Thirty)
[root@41c76f715934 /]# rpm -aq | grep -e borg -e msgpack
python3-msgpack-0.6.1-2.fc30.x86_64
borgbackup-1.1.10-2.fc30.x86_64
[root@41c76f715934 /]# borg -V
borg 1.1.10

  Awesome! I'll test it in the next few days and report it!

  Thank you!

Comment 52 Fedora Update System 2019-05-18 01:01:58 UTC
borgbackup-1.1.10-2.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 53 Tom Chiverton 2019-05-23 20:22:34 UTC
WFM after the update