Bug 531895 - RFE: emit python3-setuptools subpackage as part of python-setuptools build
RFE: emit python3-setuptools subpackage as part of python-setuptools build
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: python-setuptools (Show other bugs)
rawhide
All Linux
low Severity medium
: ---
: ---
Assigned To: Toshio Ernie Kuratomi
Fedora Extras Quality Assurance
:
: 531648 (view as bug list)
Depends On:
Blocks: Python3F13 533290 536948
  Show dependency treegraph
 
Reported: 2009-10-29 15:06 EDT by Dave Malcolm
Modified: 2010-02-04 17:45 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-02-04 17:45:26 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Hacked-up specfile (7.80 KB, text/plain)
2009-10-29 15:06 EDT, Dave Malcolm
no flags Details
Adds python3 subpackage to the python-setuptools.spec file (3.88 KB, patch)
2010-01-14 15:54 EST, Dave Malcolm
no flags Details | Diff
modify python3 package, to not build on other dists than >fc12 (3.95 KB, patch)
2010-01-14 16:34 EST, Thomas Spura
no flags Details | Diff
Alternate spec building py2 and py3 (8.17 KB, application/octet-stream)
2010-01-16 16:29 EST, Toshio Ernie Kuratomi
no flags Details
New version of the patch to python-setuptools.spec (3.85 KB, patch)
2010-01-25 10:24 EST, Dave Malcolm
no flags Details | Diff

  None (edit)
Description Dave Malcolm 2009-10-29 15:06:31 EDT
Created attachment 366691 [details]
Hacked-up specfile

Our python-setuptools rpm uses the "distribute" fork of setuptools.  I had a go at changing the specfile to build against /usr/bin/python3 rather than /usr/bin/python, and it "just works".

This approach gives us an almost-identical specfile.  For the sake of experimentation I filed a review request for it as bug 531648

This is the approach I advocated in my proposal here: https://www.redhat.com/archives/fedora-devel-list/2009-October/msg00054.html

Advantages of this approach:
  - if the python-foo maintainer doesn't care about python 3, he/she doesn't need to
  - the two specfiles can evolve separately; if 2 and 3 need to have different versions, they can

Disadvantages of this approach:
  - the two specfiles have to be maintained separately
  - when upstream release e.g. security fixes, they have to be tracked in two places

An alternative approach would be to support both 2 and 3 from one srpm build.

It struck me that these specfiles are almost identical, so as an experiment, I had a go at hacking up the specfile to generate both 2 and 3 subpackages from one build.

Attached is a version of the specfile that does so.

Advantages of this approach:
  - single srpm and build; avoid having to update multiple packages when things change.

Disadvantages of this approach:
  - Fedora maintainer needs to care about python 3.  By adding python 3 to the mix, I'm giving them extra work.
  - 2 and 3 versions are in lockstep.  Requires upstream to case about Python 3 as well.

(Talking about setuptools, maybe it's possible to use setuptools to transparently do some of this work for other packages?)

Thoughts?
Comment 1 Andrew McNabb 2009-10-29 15:30:04 EDT
This looks really cool.  One question: how does the 2to3 tool work into this?  Do source files have to have __future__ imports for Python 3 syntax, or can 2to3 be run just for the Python 3 version?
Comment 2 Toshio Ernie Kuratomi 2009-10-29 19:22:30 EDT
* I distrust creating the packages automatically with 2to3... not sure if that's a reasonable distrust or not, though.

* When upstream doesn't have a unified source tree, we will need to have two separate packages at the srpm level.  That means that not all packages will be the same there (but for the end user, it would look a lot alike).

* Also requires upstream to care about python2.  At some point this will become the issue instead of caring about python3.  It will show up as foo-1.x supports python2 and foo-2.x supports python3.

I find your solving of "when upstream release e.g. security fixes, they have to be tracked in two places" to be a great advantage, though....
Comment 3 Toshio Ernie Kuratomi 2009-10-29 20:40:37 EDT
Reviewed implementation -- that part looks good.
Comment 4 Toshio Ernie Kuratomi 2009-11-07 11:23:46 EST
Another Con: Makes it harder to figure out what to checkout from cvs or what to file a bug against in bugzilla.  For any python3-* package you first have to try: python3-foo.  If that fails, then you try python-foo.  (bugzilla is a worse problem than cvs as that touches end users).
Comment 5 Bug Zapper 2009-11-16 09:37:47 EST
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 6 Dave Malcolm 2010-01-14 15:52:24 EST
(adding package name to bug title to make the dependency tree easier to read; fixing version)
Comment 7 Dave Malcolm 2010-01-14 15:53:19 EST
Python 3 is in rawhide now (for more information see http://lists.fedoraproject.org/pipermail/devel/2010-January/129185.html )

I'm attaching a patch against the current CVS devel/python-setuptools.spec which adds a python3-setuptools subpackage.

Note that the python3-devel subpackage defines __python3, python3_sitelib and python3_sitearch, so there's no need for them in this specfile.

Successful scratch build into dist-f13 here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=1921992

I'm testing with a local build (on my F12 box) and it seems to work.
Comment 8 Dave Malcolm 2010-01-14 15:54:12 EST
Created attachment 384449 [details]
Adds python3 subpackage to the python-setuptools.spec file
Comment 9 Thomas Spura 2010-01-14 16:34:17 EST
Created attachment 384460 [details]
modify python3 package, to not build on other dists than >fc12

It would be nice to be able to use the same spec for rhel ant other branches than devel. But I couldn'd disable building the python3 subpackage...

With:
  3 %if 0%{?fedora} > 12
  4 %global with_python3 1
  5 %endif


and instead with_python3 -> 0%{?with_python3}

this works.

Koji builds:
F-12 (without python3-*):
http://koji.fedoraproject.org/koji/taskinfo?taskID=1922191

F-13 (with python3-*):
http://koji.fedoraproject.org/koji/taskinfo?taskID=1922190
Comment 10 Toshio Ernie Kuratomi 2010-01-16 16:02:25 EST
There's a bug in the proposed spec file.  easy_install is the python3 version but placed in the python2 package.
Comment 11 Toshio Ernie Kuratomi 2010-01-16 16:06:03 EST
dmalcolm: python question: Should %{_libdir}/python{VER}/lib2to3 be in the main python package instead of the tools package? It's placement makes it a public library that anyone can import and it's not in the Tools subdirectory.  I can open a bug for both py2 and py3 if you'd like to consider that for a while before deciding.
Comment 12 Toshio Ernie Kuratomi 2010-01-16 16:29:00 EST
Created attachment 384853 [details]
Alternate spec building py2 and py3

Here's an alternate take on how this would look if you do not copy the source tree, just the build output.  This is similar to the way other packages that are built twice work (like vim-enhanced/-minimal/-X11).
Comment 13 Toshio Ernie Kuratomi 2010-01-16 16:46:34 EST
http://koji.fedoraproject.org/koji/taskinfo?taskID=1927333

dmalcolm a few questions, what's the command to verify that the proper byte code is in this file?  I ran rpmlint on the generated rpms but I don't know if F-12's rpmlint will detect improper byte code.

Also, I tried this experiment:
$ rpmdev-extract python3-setuptool-*.rpm
$ cd python3-setuptools*noarch/usr/lib/python3.1/site-packages
$ python3 -c 'import pkg_resources; print(pkg_resources.__file__)'
pkg_resources.py
$ mv pkg_resources.py ../
$ python3 -c 'import pkg_resources; print(pkg_resources.__file__)'
pkg_resources.pyc

(This is with an older build of python3.1.1 on F-12)

This happens with both tomspur's build and my build.  When I cd to the python2 rpm's directory, python2 -c 'import pkg_resources ; print pkg_resources.__file__'  prints pkg_resources.pyc

Could be a misconfiguration in my old python3 build but also might be a product of how we're building these.  Want to check that out?
Comment 14 Dave Malcolm 2010-01-18 18:56:03 EST
(In reply to comment #11)
> dmalcolm: python question: Should %{_libdir}/python{VER}/lib2to3 be in the main
> python package instead of the tools package? It's placement makes it a public
Yes, please file a bug.
Comment 15 Dave Malcolm 2010-01-18 19:02:26 EST
(In reply to comment #12)
> Created an attachment (id=384853) [details]
> Alternate spec building py2 and py3
> 
> Here's an alternate take on how this would look if you do not copy the source
> tree, just the build output.  This is similar to the way other packages that
> are built twice work (like vim-enhanced/-minimal/-X11).    

This approach makes the assumption that the %build phase doesn't have side-effects on the source tree.  For a well-behave source tree that shouldn't matter, but I'm sure we'll encounter awkward cases where "%build" does something to the source tree.  For these, the original approach of copying the source tree will work, whereas the suggested approach will fail.

By way of example, if a %build does a "configure" without changing to a build subdirectory, you'll have "droppings" from the configure run, along with a "Makefile" in the root directory of the build, and so the two builds could start confusing each other.
Comment 16 Dave Malcolm 2010-01-18 19:04:29 EST
(In reply to comment #12)
> Created an attachment (id=384853) [details]
> Alternate spec building py2 and py3
> 
> Here's an alternate take on how this would look if you do not copy the source
> tree, just the build output.  This is similar to the way other packages that
> are built twice work (like vim-enhanced/-minimal/-X11).    

BTW, this doesn't seem to define the "py2dir" that it references in a few places.
Comment 17 Dave Malcolm 2010-01-18 19:11:37 EST
(In reply to comment #13)
> http://koji.fedoraproject.org/koji/taskinfo?taskID=1927333
> 
> dmalcolm a few questions, what's the command to verify that the proper byte
> code is in this file?  I ran rpmlint on the generated rpms but I don't know if
> F-12's rpmlint will detect improper byte code.
Bug 531102 is tracking the new rpmlint test; I'm not sure what its current status is (it's in the upstream rpmlint).

I patched "file" to detect this, see bug 531082.  You can test this manually using hexdump on a .pyo file, for 3.1, the first four bytes should be:
 4f 0c 0d 0a

(BTW I suspect that F-12's rpmbuild doesn't gets this right; this is bug 531117, which I believe is only fixed for F-13)

[snip]
Comment 18 Thomas Spura 2010-01-23 07:48:32 EST
What's the state on this?

With Toshios' new example spec from comment 13 python-minimock builds great with this:
SRPM: http://tomspur.fedorapeople.org/python3/python-minimock-1.2.5-2.fc12.src.rpm

I also tested the little example from upstream at:
http://pypi.python.org/pypi/MiniMock#introduction

Works as excepted.


(Didn't test it with the spec file from comment 9.)
Comment 19 Thomas Spura 2010-01-23 08:52:31 EST
(Sorry, don't want to seem pushy...

I just want to further test, before the python3 guidelines are approved, and something went wrong)
Comment 20 Dave Malcolm 2010-01-25 10:24:34 EST
Created attachment 386647 [details]
New version of the patch to python-setuptools.spec

Hi Thomas!  Thanks for looking at this.

Toshio and I have been working on the guidelines on IRC.

The biggest change has been to arrange for %{py3dir} to be automatically supplied by the python3-devel macros.

I'm attaching an updated version of patch.

I've tried to update this to reflect the latest state of https://fedoraproject.org/wiki/PackagingDrafts/Python3, as a sanity check of both the guidelines and this package.

This version uses the py3dir definition from the latest build of python3-devel.

Scratch build into dist-f13: http://koji.fedoraproject.org/koji/taskinfo?taskID=1943288
Comment 21 Toshio Ernie Kuratomi 2010-01-28 23:02:32 EST
I've committed what we have so far to the development repo but with with_python3 disabled until we get the Guidelines finalized and the brp_python_bytecompile script fixed.  here's a build from that spec file:

http://koji.fedoraproject.org/koji/taskinfo?taskID=1951161
Comment 22 Toshio Ernie Kuratomi 2010-01-28 23:43:31 EST
I've been looking at rewriting the last portions of the Python3 page to be in the style of Guidelines and see one thing -- The current page talks about the packager running 2to3 on the sources.  This isn't really something that I was anticipating when I wrote the bit about deciding on split vs single srpm.  That portion only anticipates upstreams actively distributing both python2 and python3 modules.  Do we want packagers to run 2to3 on python2 code when upstream is not actively supporting python3?  If we do, do we want to require packagers to make separate srpms for these?
Comment 23 Thomas Spura 2010-01-31 20:43:30 EST
*** Bug 531648 has been marked as a duplicate of this bug. ***
Comment 24 Dave Malcolm 2010-02-04 17:45:26 EST
Toshio enabled the subpackage and built it into dist-f13 as python-setuptools-0.6.10-3.fc13:
http://koji.fedoraproject.org/koji/taskinfo?taskID=1963883

The buildArch subtask is here
http://koji.fedoraproject.org/koji/taskinfo?taskID=1963890
and does indeed contain python3-setuptools-0.6.10-3.fc13.noarch.rpm ; I quickly reviewed the build.log and it appeared sane.

Closing; thanks!

Note You need to log in before you can comment on or make changes to this bug.