Bug 1669083
Summary: | borgbackup fails in F30 due to msgpack update | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Kevin Fenzi <kevin> |
Component: | borgbackup | Assignee: | Felix Schwarz <fschwarz> |
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | high | Docs Contact: | |
Priority: | unspecified | ||
Version: | rawhide | CC: | 5jxusjgarm, 7d28c752, abo, antoine.gemis, arjunak234, bonfostar, bugzilla.redhat.com, bzf, cfergeau, c.justin88, clasohm, craig, dev, dgibson, fredcwells, frog, fschwarz, hontonoroger+redhat, jonha87, jorti, laurent, mail, mh, mmello, peter.hutterer, pilux, rafael, reklov, releng, ricardo.arguello, Simon.Gerhards, thomas.moschny, timo, tomek, tom+f, vaaghoofdharry, vic, wpinheir |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | borgbackup-1.1.10-2.fc30 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-05-18 01:01:58 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1710715 | ||
Bug Blocks: | 1674516, 1683400 |
Description
Kevin Fenzi
2019-01-24 09:52:07 UTC
Ah, that is too bad as this is likely a "real" incompatibility. msgpack 0.6 is only supported in borg's master branch [1] and upstream explicitly advises against using it in production [2]. Actually I sent an email on 2019-01-17 14:41 UTC+1 to python-msgpack-owner *exactly* because I feared this might becomea problem and asked if they could coordinate with Fedora's borgbackup maintainer if they planned an upgrade. Now it seems as if my message got lost as Fabian Affolter pushed the 0.6 update on 2019-01-18 08:46 UTC. Let's see what we can do here, adding Fabian as he might have an idea. [1] https://github.com/borgbackup/borg/issues/4251 [2] https://github.com/borgbackup/borg/issues/4251#issuecomment-453777837 For Rawhide is only borgbackup 1.1.7 and not 1.1.8 available [1]. The latest version of borgbackup 1.1.8 was build at the end of December. In 1.1.8 was support added for python-msgpack 0.6.0 thus I postponed the update of python-msgpack because I was waiting for the rawhide build. As there was no build available after 2.5 week I updated python-msgpack in rawhide because back then python-msgpack 0.6.0 was out for six weeks and included security fixes. [1] https://apps.fedoraproject.org/packages/borgbackup 1.1.8 was committed in december, but failed to build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1182746 Looks like a failure in tests. =================================== FAILURES =================================== __________________ ArchiverTestCase.test_corrupted_repository __________________ self = <borg.testsuite.archiver.ArchiverTestCase testMethod=test_corrupted_repository> def test_corrupted_repository(self): self.cmd('init', '--encryption=repokey', self.repository_location) self.create_src_archive('test') self.cmd('extract', '--dry-run', self.repository_location + '::test') output = self.cmd('check', '--show-version', self.repository_location) self.assert_in('borgbackup version', output) # implied output even without --info given > self.assert_not_in('Starting repository check', output) # --info not given for root logger E AssertionError: 'Starting repository check' unexpectedly found in 'borgbackup version 1.1.8\nStarting repository check\nStarting repository index check\nCompleted repository check, no problems found.\n' build/lib.linux-x86_64-3.7/borg/testsuite/archiver.py:1494: AssertionError 1 failed, 2233 passed, 310 skipped, 142 deselected, 4 xfailed in 909.07 seconds BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.sHksSs (%check) BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.sHksSs (%check) Seems like https://github.com/borgbackup/borg/issues/3805 but that was already committed to 1.1.8, so not sure. Sorry folks, forgot to build 1.1.8 for rawhide. The build previously failed with some buildroot issues, but now its this failing test. I saw the same Issue as Kevin and can be related to it. Not shure if we can xfail this test, better report this upstream. reported upstream: https://github.com/borgbackup/borg/issues/4287 Shot in the blue: `setup_logging` contains also a logging.clear(), removing it lets all tests pass for testsuite/archvier.py --- a/src/borg/testsuite/archiver.py 2018-12-09 03:18:19.000000000 +0100 +++ b/src/borg/testsuite/archiver.py.b 2019-01-30 21:18:48.973311799 +0100 @@ -279,7 +279,7 @@ os.chdir(self._old_wd) # note: ignore_errors=True as workaround for issue #862 shutil.rmtree(self.tmpdir, ignore_errors=True) - setup_logging() + #setup_logging() def cmd(self, *args, **kw): exit_code = kw.pop('exit_code', 0) *** Bug 1674709 has been marked as a duplicate of this bug. *** I patched out the logging check until we figured out the issue about the logging. seems to be the only test failing because of this. The Issue with msgpack persists. Only the upcoming release of borgbackup 1.2 supports msgpack > 0.5.6 (ex. 0.6) So until the release of borgbackup 1.2 there are no further updates possible for borg in f30 and rawhide. Especially the current Releases in f30 and rahwhide can be broken, due the wrong msgpack version. Is borgbackup 1.2.0 going to be released before f30 release? If not, and assuming the msgpack 0.6.0 support cannot be backported, should a python-msgpack056 compat package be introduced? *** Bug 1694426 has been marked as a duplicate of this bug. *** Fwiw, I can workaround this on my f30 system with dnf downgrade https://kojipkgs.fedoraproject.org//packages/python-msgpack/0.5.6/5.fc29/x86_64/python3-msgpack-0.5.6-5.fc29.x86_64.rpm as I have nothing else depending on this package on my install. Similar problem has been detected: Running: borg list :: reporter: libreport-2.10.0 cmdline: /usr/bin/python3 /usr/bin/borg list :: crash_function: resolve exception_type: pkg_resources.ContextualVersionConflict executable: /usr/bin/borg interpreter: python3-3.7.3-1.fc30.x86_64 kernel: 5.0.9-301.fc30.x86_64 package: borgbackup-1.1.7-2.fc30 reason: __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup runlevel: N 5 type: Python3 uid: 0 Hitting the same problem on Fedora 30 SilverBlue [root@storm ~]# rpm -qi borgbackup Name : borgbackup Version : 1.1.7 Release : 2.fc30 Architecture: x86_64 Install Date: Tue 30 Apr 2019 06:13:18 PM EDT Group : Unspecified Size : 4337397 License : BSD Signature : RSA/SHA256, Sun 23 Sep 2018 08:47:49 PM EDT, Key ID ef3c111fcfc659b9 Source RPM : borgbackup-1.1.7-2.fc30.src.rpm Build Date : Sun 23 Sep 2018 08:27:34 PM EDT Build Host : buildvm-27.phx2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://borgbackup.readthedocs.org Bug URL : https://bugz.fedoraproject.org/borgbackup Summary : A deduplicating backup program with compression and authenticated encryption Description : BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption. [root@storm ~]# borg list Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master ws.require(__requires__) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'}) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/borg", line 6, in <module> from pkg_resources import load_entry_point File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module> @_call_aside File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside f(*args, **kwargs) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master return cls._build_from_requirements(__requires__) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements dists = ws.resolve(reqs, Environment()) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup [root@storm ~]# rpm-ostree status -v State: idle AutomaticUpdates: disabled Deployments: ● ostree://fedora-workstation:fedora/30/x86_64/silverblue Version: 30.20190427.n.0 (2019-04-27T08:57:18Z) BaseCommit: a602958b1097d0790361a8b907f6f6ecbb62b7bbf826b086401eafb4ffcb845d ├─ repo-0 ((invalid timestamp)) └─ repo-1 ((invalid timestamp)) Commit: e60764a2f28e25c3adfb9479a41e8b009dec18d3d8dfa38552cb12ebea2e2039 ├─ fedora-cisco-openh264 (2019-03-21T15:16:16Z) ├─ rpmfusion-free (2019-04-16T20:46:20Z) ├─ rpmfusion-free-updates (2019-04-29T08:32:13Z) ├─ updates (2019-04-30T01:11:12Z) ├─ rpmfusion-nonfree-steam (2019-02-27T18:49:45Z) ├─ rpmfusion-nonfree-nvidia-driver (2019-03-25T17:44:44Z) ├─ rpmfusion-nonfree (2019-04-16T21:53:39Z) ├─ fedora (2019-04-27T09:42:56Z) ├─ google-chrome (2019-04-30T19:45:42Z) └─ rpmfusion-nonfree-updates (2019-04-29T09:15:15Z) Staged: no StateRoot: fedora-workstation GPGSignature: 1 signature Signature made Sat 27 Apr 2019 04:57:30 AM EDT using RSA key ID EF3C111FCFC659B9 Good signature from "Fedora <fedora-30-primary>" InactiveRequests: fedora-toolbox LayeredPackages: NetworkManager-l2tp borgbackup fedora-workstation-repositories fish fuse-exfat git gnome-tweaks google-chrome-stable knock krb5-workstation libvirt libvirt-daemon net-tools oathtool qemu-img qemu-kvm snapd synergy terminator tilix tmux vim-enhanced virt-manager xclip xdotool xl2tpd xorg-x11-drv-nvidia I was able to work around the issue as posted on c#11 (https://bugzilla.redhat.com/show_bug.cgi?id=1669083#c11) Borg breaks in Fedora 30… And downgrading is no solution, so please try to fix this soon. This is a backup system after all, if this does not work, you have a problem… ``` $ borg -V Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master ws.require(__requires__) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'}) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/borg", line 6, in <module> from pkg_resources import load_entry_point File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module> @_call_aside File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside f(*args, **kwargs) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master return cls._build_from_requirements(__requires__) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements dists = ws.resolve(reqs, Environment()) File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup ``` According to the Borgbackup release notes, version 1.1.8 or later is compatible with python3-msgpack 0.6.1 that ships with Fedora 30. Looks like building the latest from upstream will fix this problem. (In reply to Albert Vernon from comment #16) > According to the Borgbackup release notes, version 1.1.8 or later is > compatible with python3-msgpack 0.6.1 that ships with Fedora 30. Looks like > building the latest from upstream will fix this problem. Could you please provide a link where you read this? To the best of my knowledge only borgbackup 1.2 (currently alpha) supports msgpack 0.6.1. That version was whitelisted in https://github.com/borgbackup/borg/commit/0ebfaa5b61a675c22cda301bc20d0b00372dd181 and I don't see any code in borgbackup 1.1.x which supports your claim. Actually upstream is pretty clear that only borgbackup 1.2 will support msgpack 0.6+ ( https://github.com/borgbackup/borg/issues/3753#issuecomment-454011810 ). I fear there is no elegant solution to this. Anyways Fedora 30 has been released and borgbackup is broken. We need to deal with this somehow: 1. retire borgbackup in F30 until borgbackup 1.2 is released (current version is 1.2.0 alpha6). 2. create a new package python3-msgpack05 with conflicts python3-msgpack and make Fedora's borgbackup dependent on the new package 3. vendorize a copy of msgpack 0.5.6, add a bit of import magic in borg and ship that. @bpereto: Any preference/ideas how to proceed? Similar problem has been detected: I tried running "borg list <backup-location>" to see if my backup solution works after upgrading to fedora 30. (Note: I installed borgmatic from a fc29 rpm because installing it via the official repos gives a dependency error of python3-ruamel-yaml) reporter: libreport-2.10.0 cmdline: /usr/bin/python3 /usr/bin/borg list /mnt/hd/linux/backups/pc/borg-backup crash_function: resolve exception_type: pkg_resources.ContextualVersionConflict executable: /usr/bin/borg interpreter: python3-3.7.3-1.fc30.x86_64 kernel: 5.0.9-301.fc30.x86_64 package: borgbackup-1.1.7-2.fc30 reason: __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup runlevel: N 5 type: Python3 uid: 1000 *** Bug 1705487 has been marked as a duplicate of this bug. *** I was mistaken. Borg commit 0ebfaa5 for ticket #4308 is in version 1.2 like you said and not 1.18 as I stated. Thank you for the clarification. Similar problem has been detected: Just install borgbackup on F30 and try to use it. This problem happens with every borg command. reporter: libreport-2.10.0 cmdline: /usr/bin/python3 /usr/bin/borg init --encryption=repokey Fedora-Backups/ crash_function: resolve exception_type: pkg_resources.ContextualVersionConflict executable: /usr/bin/borg interpreter: python3-3.7.3-1.fc30.x86_64 kernel: 5.0.9-301.fc30.x86_64 package: borgbackup-1.1.7-2.fc30 reason: __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup runlevel: N 5 type: Python3 uid: 1000 (In reply to Felix Schwarz from comment #17) > I fear there is no elegant solution to this. Anyways Fedora 30 has been > released and borgbackup is broken. We need to deal with this somehow: > 1. retire borgbackup in F30 until borgbackup 1.2 is released (current > version is 1.2.0 alpha6). > 2. create a new package python3-msgpack05 with conflicts python3-msgpack and > make Fedora's borgbackup dependent on the new package > 3. vendorize a copy of msgpack 0.5.6, add a bit of import magic in borg and > ship that. To me, (3) looks like a good solution. Another possibility would be to put 1.2.0 alpha into F30. (In reply to Thomas Moschny from comment #22) > Another possibility would be to put 1.2.0 alpha into F30. Upstream says: "borgbackup 1.2.0a6 alpha release - do not use this on production backup repositories." (https://github.com/borgbackup/borg/releases). I think upstream is just cautious but on the other hand users entrust Fedora/borgbackup to provide a safety net and corrupt backup data can cause significant losses. > To me, (3) looks like a good solution. That's my preferred solution as well but it needs someone to fiddle with the code. Also neither is a really good solution because upstream has no support/infrastructure to vendorize msgpack. This means we need to code something ourself which is a delicate thing to do in a backup software. (Also it costs time and I don't know if I have enough of that in the next days to implement something like that.) After downgrading to python3-msgpack 0.5.6 using Christophe's suggestion, I added the following line to the [main] section of /etc/dnf/dnf.conf to keep `dnf upgrade` from re-installing 0.6.1: excludepkgs=python3-msgpack So, a python3-msgpack 0.5.6 downgrade is IMO a suitable workaround for the many users currently stranded after F30 upgrade, assuming there is a fc30 build of same. Following Albert's lead, I also downgraded (to the F29 version @ 0.5.6-5.fc29.x86_64) and is working again. (In reply to Felix Schwarz from comment #17) > I fear there is no elegant solution to this. Anyways Fedora 30 has been > released and borgbackup is broken. We need to deal with this somehow: > 1. retire borgbackup in F30 until borgbackup 1.2 is released (current > version is 1.2.0 alpha6). > 2. create a new package python3-msgpack05 with conflicts python3-msgpack and > make Fedora's borgbackup dependent on the new package > 3. vendorize a copy of msgpack 0.5.6, add a bit of import magic in borg and > ship that. 4. downgrade the python3-msgpack package in fedora 30 to 0.5.6 (through the use of an epoch) imo this is better than 2, unless something else really needs 0.6.0 (In reply to Christophe Fergeau from comment #27) > 4. downgrade the python3-msgpack package in fedora 30 to 0.5.6 (through the > use of an epoch) imo this is better than 2, unless something else really needs 0.6.0 I omitted that option because I think Fabian deliberately updated python3-msgpack in January (see comment 2): > I updated python-msgpack in rawhide because back then python-msgpack 0.6.0 was out for six weeks and included security fixes. Yesterday I tried (very briefly) to bundle a pure python version of msgpack (slow) but so far without much success. If there is someone who wants to invest some time I can upload a git repo (basic Python skills needed). Raising priority, since it essentially makes borgbackup wholly unusable. Similar problem has been detected: I've tried to run borgbackup, just as I was doing with Fedora 29. There seems to be an issue with Fedora 30 and the msgpack package: https://github.com/borgbackup/borg/issues/3889 reporter: libreport-2.10.0 cmdline: /usr/bin/python3 /usr/bin/borg create -v --stats --compression lz4 --show-rc --show-version borgbackup.net:/data/backup/spring.local::{hostname}-{now} /home/laurent/.ssh/ /home/laurent/.thunderbird/ /home/laurent/Devel/ /home/laurent/Documents/ /home/laurent/Pictures/ /home/laurent/_tmp/ crash_function: resolve exception_type: pkg_resources.ContextualVersionConflict executable: /usr/bin/borg interpreter: python3-3.7.3-1.fc30.x86_64 kernel: 5.0.10-300.fc30.x86_64 package: borgbackup-1.1.7-2.fc30 reason: __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup runlevel: N 5 type: Python3 uid: 1000 Ok, I got borgbackup to pass all the unit tests with an bundled copy of msgpack 0.5.6. I'll try to get this packaged in a COPR this evening. It will needs quite a bit of user testing before even submitting it to the testing repo. Just a few notes on why this was more complicated than expected: - tests for borgbackup 1.1.9 fail with the pure-python msgpack - need to compile msgpack's Cython as part of borg's build so it gets the right module names - msgpack uses Cython -> C++ while borg uses Cython->C. Here is a COPR which you can try https://copr.fedorainfracloud.org/coprs/fschwarz/bundled-borgbackup/ I DID NOT TEST the built binaries. DO NOT USE on production data. As you can see in the RPM I did not really change the code but "msgpack" is really at the heart of borg's data storage so any screw up here could cause loss of (backup) data. Also I got errors in COPR when running on ppc64le (https://copr.fedorainfracloud.org/coprs/fschwarz/bundled-borgbackup/build/906877/). If someone more knowledgeable could provide some insight that would be appreciated. The COPR failure for ppc64le seems to be just COPR related as a koji build succeeded (https://koji.fedoraproject.org/koji/taskinfo?taskID=34754384). s390x seems to be hanging. I did a quick test on F30 and the new RPM worked for me (create repo, do a backup, restore). However I want to have at least one or two users trying out the RPM before I think about pushing it to Fedora testing. I also did a quick test on F30 wit the new RPM from COPR fschwarz/bundled-borgbackup (only a backup to an existing repo). No more errors, and the backup finished successfully. Thanks, Laurent. Ideally we'll one or two more testers and then we'll push the new code to testing. The following operations worked for me using fschwarz/bundled-borgbackup: borg info borg check borg list borg create borg prune borg mount Thank you Albert. I submitted a Koji build for rawhide. s390x seems to be hanging in my scratch builds, let's hope that the real build will be fine (https://koji.fedoraproject.org/koji/taskinfo?taskID=34771060). I'll submit a build for F30 once rawhide is fine. (In reply to Felix Schwarz from comment #37) > s390x seems to be hanging in my scratch builds A local, qemu-powered build for s390x does hang, but shows a test failure: # fedpkg switch-branch master # MOCKARGS="--forcearch s390x" fedpkg mockbuild --mock-config fedora-rawhide-s390x [...] build/lib.linux-s390x-3.7/borg/testsuite/archiver.py::RemoteArchiverTestCase::test_compression_auto_compressible FAILED [ 27%] =================================== FAILURES =================================== __________ RemoteArchiverTestCase.test_compression_auto_compressible ___________ self = <borg.testsuite.archiver.RemoteArchiverTestCase testMethod=test_compression_auto_compressible> def test_compression_auto_compressible(self): size, csize = self._get_sizes('auto,zlib', compressible=True) > assert csize < size * 0.1 E AssertionError: assert 0 < (0 * 0.1) build/lib.linux-s390x-3.7/borg/testsuite/archiver.py:2078: AssertionError = 1 failed, 203 passed, 151 skipped, 71 deselected, 1 xfailed in 325.34 seconds = error: Bad exit status from /var/tmp/rpm-tmp.ZBQpzF (%check) Maybe the koji builds simply fail to detect the failure... Thank you Thomas for looking into this. Can you verify that the msgpack Cython extension (_packer/_unpacker) is built correctly? (warnings about language level are expected) I saw this test failure also on x86_64 when using the pure-python msgpack with borg's test suite. Taking that back. What seems to happen is that qemu emits 'Unsupported setsockopt level=270 optname=3' messages to stderr which wrongly get interpreted by the testsuite as the output of the 'borg list' command. FTR, _packer.cpython-37m-s390x-linux-gnu.so and _unpacker.cpython-37m-s390x-linux-gnu.so binaries are present. (And btw, the corresponding .cpp files probably should not be installed, but that's really a minor thing.) Update: upstream added some commits which might prevent hangs in the test suite. I hope to get access to a s390x machine soon so I can try adding a few patches. If that fails I'll (temporarily) exclude s390x and push an update for all other architectures so you get a working borgbackup again. Same happens in EPEL7 atm. (In reply to Marcel Haerry from comment #42) > Same happens in EPEL7 atm. EPEL7 should still use msgpack 0.5.6 so I think you are experiencing a different issue. Please file a new issue and provide additional info. (In reply to Felix Schwarz from comment #43) > (In reply to Marcel Haerry from comment #42) > > Same happens in EPEL7 atm. > > EPEL7 should still use msgpack 0.5.6 so I think you are experiencing a > different issue. > > Please file a new issue and provide additional info. You are right and I made a wrong alarm, as I misread from which machine I got the output of the failed update. Sorry and thanks for the hint. Similar problem has been detected: I just called "borg" and it happened. reporter: libreport-2.10.0 cmdline: /usr/bin/python3 /usr/bin/borg crash_function: resolve exception_type: pkg_resources.ContextualVersionConflict executable: /usr/bin/borg interpreter: python3-3.7.3-1.fc30.x86_64 kernel: 5.0.14-300.fc30.x86_64 package: borgbackup-1.1.7-2.fc30 reason: __init__.py:786:resolve:pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup runlevel: N 5 type: Python3 uid: 1000 upstream added some changes to fix problems I found while working on this bug. There is even a new upstream release 1.1.10 which makes packaging much easier (I hit several issues while trying to package a git snapshot of borgbackup's "1.1-maint" branch. However the (sloooow) koji builders still see test failures on various architectures. I'll investigate as part of bug 1710715 and will push an update as soon as possible. borgbackup 1.1.10 pushed+built for rawhide+F29. I'll keep this bug open specifically to fix F30. Unfortunately I can't get the test suite to pass on F30 (in mock/Koji). I published my current state at https://github.com/FelixSchwarz/borg-fedora/issues/1 - any help is appreciated. I even considered just disabling the failing test but I'm hesitant to do so - even though the current situation (borgbackup being completely broken in F30) is really [censored]. borgbackup-1.1.10-2.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-90776185cd As the previous comment indicated I finally succeeded to build borgbackup 1.1.10 for Fedora 30. The update should hit updates-testing soon. Please test the new version and report your findings in bodhi (https://bodhi.fedoraproject.org/updates/FEDORA-2019-90776185cd). I'm very sorry that we shipped Fedora 30 with a broken version of borgbackup and that it took me so long to fix the package properly. Unfortunately this was not just a simple "minor update" but involved some coding, scripting and quite a few builds on Fedora's infrastructure (roughly a dozen builds ran on 6 supported architectures). I experienced test suite hangs on s390x and (today) failing builds on F30 (while F29 and rawhide just worked fine). As part of fixing this I filed 3 issues in upstream bug trackers and created 2 pull requests. I have to say I was pretty frustrated at times as it seemed as if there was always another bug waiting for me whenever I made some progress. Despite all this I'm really glad I got a lot of help: Three different Fedora developers spent time on investigating the s390x hangs. Upstream developers of borgbackup and msgpack merged my pull requests. Especially Thomas Waldmann (main developer of borgbackup) put in extra hours at night to improve my quickly hacked bundling code for msgpack, merging it upstream and publishing a new release just in time. I'm telling you this because I feel I owe you an apology. To me Fedora is about shipping stable software very quickly - with Fedora 30 we failed to do so for borgbackup but I hope things will go smoother in the future. Thank you for your patience. borgbackup-1.1.10-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-90776185cd [root@41c76f715934 /]# cat /etc/fedora-release Fedora release 30 (Thirty) [root@41c76f715934 /]# rpm -aq | grep -e borg -e msgpack python3-msgpack-0.6.1-2.fc30.x86_64 borgbackup-1.1.10-2.fc30.x86_64 [root@41c76f715934 /]# borg -V borg 1.1.10 Awesome! I'll test it in the next few days and report it! Thank you! borgbackup-1.1.10-2.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report. WFM after the update |