Bug 2048816 - borgbackup unneeded dependency on python36-msgpack
Summary: borgbackup unneeded dependency on python36-msgpack
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: borgbackup
Version: epel7
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Felix Schwarz
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-31 22:16 UTC by Tom K
Modified: 2022-02-03 19:32 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Tom K 2022-01-31 22:16:00 UTC
Description of problem:
borgbackup-1.1.17-2 has a dependency on python36-msgpack <= 0.5.6, despite msgpack being bundled with borgbackup as of version (I think) 1.1.10

In /usr/lib64/python3.6/site-packages/borg/helpers.py we can find some messaging around this change, and the following variable:

prefer_system_msgpack = False

This causes a problem for any package that requires python36-msgpack > 0.5.6, such as salt-3004


Version-Release number of selected component (if applicable):
borgbackup-1.1.17-2


How reproducible: Always



Steps to Reproduce:
Try to install any package requiring python36-msgpack > 0.5.6; example:
Error: Package: borgbackup-1.1.17-2.el7.x86_64 (@prod-epel-7)
           Requires: python36-msgpack <= 0.5.6
           Removing: python36-msgpack-0.5.6-5.el7.x86_64 (@prod-epel-7)
               python36-msgpack = 0.5.6-5.el7
           Updated By: python36-msgpack-0.6.2-2.el7.x86_64 (prod-salt-3004-el7)
               python36-msgpack = 0.6.2-2.el7

Expected results:
No dependency on python36-msgpack since it is already bundled with borgbackup and configured as the default msgpack source

Comment 1 Felix Schwarz 2022-02-01 08:49:48 UTC
> borgbackup-1.1.17-2 has a dependency on python36-msgpack <= 0.5.6, despite msgpack being bundled with borgbackup as of version (I think) 1.1.10

borgbackup for EPEL 7 does not bundle msgpack.

%if 0%{?fedora} || 0%{?rhel} == 8
  %global bundle_msgpack 1
%else
  %global bundle_msgpack 0
%endif

Which repo has "salt-3004"? EPEL 7 does not ship python36-msgpack > 0.5.6 so clearly this package is incompatible with EPEL.

Comment 2 Tom K 2022-02-02 18:46:56 UTC
My apologies - I was going by the borgbackup release notes, along with the following inside of borg/helpers.py:

> ...
> # MSGPACK =====================================================================
> # we are rather picky about msgpack versions, because a good working msgpack is
> # very important for borg, see https://github.com/borgbackup/borg/issues/3753
> #
> # because some linux distributions didn't get their dependency management right
> # and broke borgbackup by upgrading msgpack to incompatible versions, we now
> # bundle msgpack-python 0.5.6, which is the latest and best msgpack that is
> # still compatible with borg 1.1.x and we use the bundled version by default.
> #
> # if you are a package maintainer and don't like bundled library code, feel
> # free to not use the bundled code:
> # - set prefer_system_msgpack = True
> # - make sure that an external msgpack-python gets installed
> # - make sure the external msgpack-python always stays at supported versions.
> # - best versions seem to be 0.4.6, 0.4.7, 0.4.8 and 0.5.6.
> # - if you can't satisfy the above requirement, these are versions that might
> #   also work ok, IF you make sure to use the COMPILED version of
> #   msgpack-python NOT the PURE PYTHON fallback implementation: 0.5.1 and 0.5.4
> #
> # Please note:
> # - using any other version is not supported by borg development and
> #   any feedback related to issues caused by this will be ignored.
> # - especially, it is known that msgpack 0.6.x does NOT work for borg 1.1.x.

> prefer_system_msgpack = False
> ...

Looking at it further, the logic that follows then falls back to trying a system-provided msgpack upon a ImportError exception, so that makes sense now.

salt-3004 comes from https://repo.saltproject.io , in particular the following: https://repo.saltproject.io/py3/redhat/7/x86_64/3004/

They provide their own (newer) version of python36-msgpack (0.6.2-2) as of their 3001 release. Does this effectively mean that I can't have salt-minion >= 3001 + borgbackup installed on the same system?

Comment 3 Tom K 2022-02-02 18:51:31 UTC
I suppose I could manually supplement /usr/lib64/python3.6/site-packages/borg/algorithms/msgpack and call it a day

Comment 4 Tom K 2022-02-02 19:04:20 UTC
Okay, thinking about it some more now, even that won't be enough. I'll still be stuck with the incompatible RPM dependencies, requiring me to either force-install-overwrite the newer msgpack over top of the old msgpack, or to install the Salt packages while ignoring dependencies, or some variation of such hacks. None of that is ideal. 

In a world where EPEL is not the only repo on a 7 system, and where Salt is a very popular automation / config management technology, the ideal solution would be for the RPM to continue supplying the built-in msgpack.

Comment 5 Felix Schwarz 2022-02-03 07:57:36 UTC
> They provide their own (newer) version of python36-msgpack (0.6.2-2) as of their 3001 release. Does this effectively mean that I can't have salt-minion >= 3001 + borgbackup installed on the same system?

Yes, I'm afraid that is correct.

> In a world where EPEL is not the only repo on a 7 system, and where Salt is a very popular automation / config management technology, the ideal solution would be for the RPM to continue supplying the built-in msgpack.

Well, historically it's the other way around: borgbackup did not bundle msgpack until Fedora+other distros updated their msgpack to an incompatible version (https://github.com/borgbackup/borg/pull/4560). We kept the previous method of using system dependencies for EPEL 7 so the RPM (and borg 1.1.x at its initial release) always relied on the system-wide msgpack :-)
Though that is really just nitpicking and does not solve your problem.

Honestly I'm a bit hesitant to change EPEL 7 at this point in time:
- RHEL 7 (+ CentOS, Oracle Linux, ...) is pretty late in its life cycle and not expected to get major updates.
- We can not keep EPEL compatible with other repos - there are just too many of them and often they have conflicting update policies (I'm seeing this regularly with certbot).
- So far there were no requests to make borgbackup/EPEL compatible with the salt repo.
- Fedora/EPEL's policies strongly suggest using system dependencies.

On the plus side:
- We are already bundling msgpack for Fedora/EPEL 8 so this is a tested configuration and enabling it would be trivial.

I need to work on a different task for $DAYJOB right now but maybe you could also think about using the precompiled borg from github? The single-file executable should bundle msgpack and you could use borgbackup+salt that way.

Comment 6 Tom K 2022-02-03 19:32:20 UTC
Those are all very good points. I understand the position you're in. Thanks for the response!

In my particular case, I think my best option is to roll my own RPM with bundle_msgpack 1 (company policy re: binaries) - at least for the time being.


Note You need to log in before you can comment on or make changes to this bug.