Bug 531543 - Add python 3 bindings for librpm
Summary: Add python 3 bindings for librpm
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Fedora Packaging Toolset Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: Python3F13
TreeView+ depends on / blocked
 
Reported: 2009-10-28 17:30 UTC by Dave Malcolm
Modified: 2013-08-21 15:10 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-08-21 11:03:08 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Patch to rpm.spec to add a python3- subpackage (2.95 KB, patch)
2009-11-09 19:21 UTC, Dave Malcolm
no flags Details | Diff
Patch to add python3 configuration detection (3.34 KB, patch)
2009-11-09 19:25 UTC, Dave Malcolm
no flags Details | Diff
Updated version of patch to rpm.spec, adding an "rpm-python3" subpackage (5.08 KB, patch)
2010-02-22 03:20 UTC, Dave Malcolm
no flags Details | Diff
Add configuration for Python 3 to build (3.34 KB, patch)
2010-02-22 03:21 UTC, Dave Malcolm
no flags Details | Diff
Definition of an AM_PATH_PYTHON3 macro, for automake (7.91 KB, text/plain)
2010-02-22 03:23 UTC, Dave Malcolm
no flags Details
Add rpm-python3 subpackage (this time adding autotool build requirements) (5.13 KB, patch)
2010-02-22 03:35 UTC, Dave Malcolm
no flags Details | Diff
Adds an rpm-python3 subpackage (4.66 KB, patch)
2012-05-25 02:22 UTC, Dave Malcolm
no flags Details | Diff
Adds rpm-python3 subpackage with cleaner spec (7.02 KB, patch)
2013-04-03 12:30 UTC, Bohuslav "Slavek" Kabrda
no flags Details | Diff

Description Dave Malcolm 2009-10-28 17:30:16 UTC
I need to finish up the work done here:
http://lists.rpm.org/pipermail/rpm-maint/2009-October/002528.html

and get this built as a separate subpackage, ideally for Fedora 13.

Comment 1 Panu Matilainen 2009-10-29 12:57:09 UTC
Note that this is going to require a new major version of rpm to be included in Fedora 13 (that is the plan anyway so it shouldn't be an issue). The python bindings in current rpm.org HEAD require all sorts of things that are simply not available in rpm 4.7.x.

Comment 2 Dave Malcolm 2009-11-09 19:21:03 UTC
Created attachment 368275 [details]
Patch to rpm.spec to add a python3- subpackage

Patch to specfile:
  - adds a patch to configure.ac and Makefile.am
  - at the end of %prep, takes a copy of the "python" subdir as "python3", the same, except using PYTHON3 versions of the autotooled config macros
  - reruns autotools since Makefile.am files and configure.ac have changed
  - adds "--enable-python3" to configuration
  - adds "python3-rpm" subpackage
(missing a %changelog entry for now)

Comment 3 Dave Malcolm 2009-11-09 19:25:01 UTC
Created attachment 368276 [details]
Patch to add python3 configuration detection

Patch:
  - uses AM_PATH_PYTHON3 (from new automake "python3.m4" file, see bug 533920) to add a parallel configuration of a build for python 3.  Note that I had to manually clear autoconf cache checks in two places here, to avoid autoconf from erroneously using incorrect cached values when include paths have changed from under it.
  - add "python3" subdirectory to the build

Comment 4 Bug Zapper 2009-11-16 14:30:47 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 5 Dave Malcolm 2010-02-22 03:20:32 UTC
Created attachment 395412 [details]
Updated version of patch to rpm.spec, adding an "rpm-python3" subpackage

Comment 6 Dave Malcolm 2010-02-22 03:21:54 UTC
Created attachment 395413 [details]
Add configuration for Python 3 to build

Comment 7 Dave Malcolm 2010-02-22 03:23:05 UTC
Created attachment 395414 [details]
Definition of an AM_PATH_PYTHON3 macro, for automake

Comment 8 Dave Malcolm 2010-02-22 03:35:37 UTC
Created attachment 395415 [details]
Add rpm-python3 subpackage (this time adding autotool  build requirements)

Comment 9 Dave Malcolm 2010-02-22 03:38:54 UTC
Successful scratch build in Koji here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=2004177

This is with the patch to rpm.spec from attachment 395415 [details], with new source python3.m4 (attachment 395414 [details]) and new patch (attachment 395413 [details]).

The new source (python3.m4) embedded here eliminates the need for a new automake, so I'm removing bug 533920 as a blocker for this.

How does this look?

Comment 10 Dave Malcolm 2010-03-02 17:38:39 UTC
Ping?  I'm still keen to get this into Fedora 13.

Comment 11 Dave Malcolm 2010-03-05 20:34:59 UTC
Doh, was assigned to me.

Reassigning to pmatilai; Panu: how does this look?

Comment 12 Panu Matilainen 2010-03-08 12:24:58 UTC
In one word: ugly :)

I could live with it if it was going to be just a short-lived workaround but as I suspect we (and everybody else so much as thinking of python 2 -> 3 transition) are going to need parallel python 2 and 3 stacks for a long long time, I think we need to figure out something that's sanely upstreamable.

Which probably means the python module build needs to be separated from the rest of the rpm, so that for a setup like this you can
1) configure rpm at top level tree to build without python
2) build + install rpm itself
3) do 'cd python' and then build + install with whatever python versions are wanted

Comment 13 Fedora Admin XMLRPC Client 2012-04-13 23:09:18 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 14 Fedora Admin XMLRPC Client 2012-04-13 23:12:14 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 15 Dave Malcolm 2012-05-24 19:24:42 UTC
4.10.0 is now in rawhide, and I see in http://rpm.org/wiki/Releases/4.10.0:
> Python bindings can now be alternatively built with Python distutils,
> making it simpler to build the bindings for different Python versions,
> notably 2.x vs 3.x (RhBug:531543)

Comment 16 Dave Malcolm 2012-05-25 02:22:25 UTC
Created attachment 586747 [details]
Adds an rpm-python3 subpackage

The attached patch adds a python3 subpackage.  It removes the --with-python configuration option, instead using setup.py to build the bindings twice.

I had to pass in various env vars to get the bindings to build (to locate the not-yet-installed .pc file and the libs).

Successful scratch build here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4099767

Comment 17 Panu Matilainen 2013-02-08 13:13:38 UTC
Sorry I had completely forgotten about this thing :-/

The setup.py variant is less ugly than the previous ones, but as you note the setup.py itself is flawed for the purposes of in-tree building. And the otherwise "clean" option of splitting rpm-python out of rpm itself would make rpm version upgrades absolutely hideous.

It'd be better to fix up setup.py to properly support in-tree building instead of hacking around it. Somebody more experienced with python distutils than me might have a better chance of getting it right.

Comment 18 Bohuslav "Slavek" Kabrda 2013-04-03 12:30:49 UTC
Created attachment 731128 [details]
Adds rpm-python3 subpackage with cleaner spec

Hey guys,
seems I'll be taking this over from Dave.
I'm attaching a patch that makes the spec look much cleaner (with some modifications in setup.py.in, which can hopefully be merged upstream if you like them).

Comment 19 Panu Matilainen 2013-04-03 14:11:49 UTC
The spec-side looks a whole lot nicer now sure, but the py3-fixes and setup.py.in patch seem to have gotten truncated / garbled somehow so can't comment on those yet. Please (re)submit the python-side fixes separately.

Comment 20 Bohuslav "Slavek" Kabrda 2013-04-04 12:43:50 UTC
(In reply to comment #19)
> The spec-side looks a whole lot nicer now sure, but the py3-fixes and
> setup.py.in patch seem to have gotten truncated / garbled somehow so can't
> comment on those yet. Please (re)submit the python-side fixes separately.

It seems that BZ diff displaying is buggy for diffs containing new files (it doesn't display new files even in Dave's previous patch...). If you look at the raw diff [1], everything is there - I hope that that's enough.

[1] https://bugzilla.redhat.com/attachment.cgi?id=731128

Comment 21 Panu Matilainen 2013-04-10 07:05:33 UTC
Hum, I could've sworn I already commented on this but apparently I've missed hitting submit :-/

Anyway... yeah, now that you mention it I've seen the bugzilla diff bug before but had forgotten all about it. It's all there alright. The PyString -> PyBytes fix is upstreamed now, that was my mistake to begin with (old habits die hard :)

The latest patch certainly looks nicer and is something I can live with in Fedora for now. For upstreaming, there would seem to be two issues left:

The setup.py.in patch changes it to always assume being built within rpm source tree, which is fine for Fedora purposes but the idea was to technically permit distributing and building the bindings separately too. So for upstreaming that, it'd need some kind of detection whether its being built in rpm source tree or externally. Shouldn't be that big a deal.

The other issue is that this breaks the python tests in the self-test suite. A couple of pesky little details here: the self-test suite (incorrectly) assumes rpm is being built with --enable-python which it shouldn't do as the bindings are an optional add-on thing. The other question is how to enable the tests when the bindings are built outside automake control, optimally testing for both python2 and python3.

Comment 22 Bohuslav "Slavek" Kabrda 2013-04-11 10:08:56 UTC
(In reply to comment #21)
> Hum, I could've sworn I already commented on this but apparently I've missed
> hitting submit :-/
> 
> Anyway... yeah, now that you mention it I've seen the bugzilla diff bug
> before but had forgotten all about it. It's all there alright. The PyString
> -> PyBytes fix is upstreamed now, that was my mistake to begin with (old
> habits die hard :)
> 
> The latest patch certainly looks nicer and is something I can live with in
> Fedora for now. For upstreaming, there would seem to be two issues left:
> 
> The setup.py.in patch changes it to always assume being built within rpm
> source tree, which is fine for Fedora purposes but the idea was to
> technically permit distributing and building the bindings separately too. So
> for upstreaming that, it'd need some kind of detection whether its being
> built in rpm source tree or externally. Shouldn't be that big a deal.
> 

Good point, I'll work on that.

> The other issue is that this breaks the python tests in the self-test suite.
> A couple of pesky little details here: the self-test suite (incorrectly)
> assumes rpm is being built with --enable-python which it shouldn't do as the
> bindings are an optional add-on thing. The other question is how to enable
> the tests when the bindings are built outside automake control, optimally
> testing for both python2 and python3.

I'll try to look into that, too.

Comment 23 Bohuslav "Slavek" Kabrda 2013-07-15 12:16:51 UTC
Ping @panu could you please build with the patch in Fedora? It'd be great if everyone had the opportunity to test the bindings with Python 3. Thanks.

Comment 24 Panu Matilainen 2013-08-01 12:07:06 UTC
The "show-stopper" with the current patch is breaking the self-test suite, I'm not very enthusiastic about adding the patch as long as that is the case.

Comment 25 Bohuslav "Slavek" Kabrda 2013-08-01 12:59:01 UTC
(In reply to Panu Matilainen from comment #24)
> The "show-stopper" with the current patch is breaking the self-test suite,
> I'm not very enthusiastic about adding the patch as long as that is the case.

Hmm, it doesn't break self test suite in my test builds... Would you at least consider accepting the patch downstream for Fedora as you said in comment 21? I'll work on getting the patch acceptable for upstream in the meanwhile.

Comment 26 Panu Matilainen 2013-08-21 10:27:58 UTC
Heh, I just tried it and noticed the test-suite actually still works. Which is supposed to be impossible because the python bindings dont get installed into the test-root when the bindings are not built through automake.

The trick is that unlike everything else, the python tests dont run through fakechroot as arranging a full python environment inside the root is rather tricky. What happens is that it just falls back to using system rpm-python bindings, which "works" as long as the system bindings are reasonably similar as the one currently being built. So the test-suite *appears* to work when building on eg F19, but would fail on F17.

There doesn't seem to be any sane way to hook the setup.py stuff into automake test-environment creation, but what we can do is leave --enable-python into configure for the test-suite, but nuke the automake-installed bindings from buildroot and then install the bindings with setup.py, ie

make DESTDIR="$RPM_BUILD_ROOT" install

rm -rf $RPM_BUILD_ROOT/%{python_sitearch}
pushd python
%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT
%{__python3} setup.py install --skip-build --root $RPM_BUILD_ROOT
popd

Of course this isn't optimal since the python2 bindings are getting built twice and the test-suite is now strictly speaking testing a different build than what gets packaged, but meh...

Comment 27 Panu Matilainen 2013-08-21 11:03:08 UTC
Only took four years to get there *cough* but we finally have python3 bindings in rawhide:
http://koji.fedoraproject.org/koji/taskinfo?taskID=5836322

Oh and thanks to both of you for making it happen.

Comment 28 Bohuslav "Slavek" Kabrda 2013-08-21 11:24:58 UTC
(In reply to Panu Matilainen from comment #27)
> Only took four years to get there *cough* but we finally have python3
> bindings in rawhide:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=5836322
> 
> Oh and thanks to both of you for making it happen.

Thank you so much, this is great :)

Comment 29 Dave Malcolm 2013-08-21 15:10:23 UTC
 _______ 
< woot! >
 ------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||


Note You need to log in before you can comment on or make changes to this bug.