This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 251645 - python-setuptools drags in python-devel on normal installs
python-setuptools drags in python-devel on normal installs
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: python-setuptools (Show other bugs)
rawhide
All Linux
low Severity low
: ---
: ---
Assigned To: Konstantin Ryabitsev
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-08-10 04:54 EDT by Hans de Goede
Modified: 2007-11-30 17:12 EST (History)
4 users (show)

See Also:
Fixed In Version: 0.6c7-2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-11-18 12:52:06 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Patch to implement utils subpackage (2.29 KB, patch)
2007-08-18 19:12 EDT, Toshio Ernie Kuratomi
no flags Details | Diff
New patch that moves site-packages/easy_install* (2.61 KB, patch)
2007-08-22 11:53 EDT, Toshio Ernie Kuratomi
no flags Details | Diff
Patch implementing comment #34 (2.04 KB, patch)
2007-09-24 14:44 EDT, Toshio Ernie Kuratomi
no flags Details | Diff

  None (edit)
Description Hans de Goede 2007-08-10 04:54:37 EDT
Description of problem:
gnome-games now comes with glchess which requires PyOpenGL for its 3d rendering.
PyOpenGL in turn requires python-setuptools (it uses it runtime, see below),
which drags in python-devel.

To see the use of setuptools in PyOpenGL, install PyOpenGL and then open:
/usr/lib/python2.5/site-packages/OpenGL/arrays/formathandler.py

And then at line 8:
import pkg_resources

And line 37-39:
                for entrypoint in pkg_resources.iter_entry_points(
                        "OpenGL.arrays.formathandler"
                ):

So all PyOpenGL needs from python-setuptools is pkg_resources.py, I've verified
this be rpm -e python-setuptools and then manually restoring pkg_resources.py

I see 2 possible solutions:
1) Since most of python-setuptools is meant for development put all of
   python-setuptools execept for pkg_resources.py in python-setuptools-devel,
   this esp. makes sense because then python-setuptools-devel would require
   python-devel, instead of a "normal" package requiring a -devel package.

   And then have the main python-setuptools package contain pkg_resources.py

   To me this is the prefered solution, disadvantage anything BuildRequiring
   python-setuptools needs to update its BR's, but this can be coordinated with
   a mail to the maintainers list and is OK to do in Rawhide

2) put pkg_resources.py in python-setuptools-pkg_resources, and make the main 
   package require it. This feels wrong though. Esp the make the main 
   package require it part says to me that we are doing things the wrong way 
   around then.

I've also looked at other possible runtime use candidates in python-setuptools,
and I couldn't find any.


p.s.

Jeremy I've put you in the CC because the draggin of python-setuptools caused
size problems for the livecd, once this bug is resolved that should no longer be
a problem as then a package with only one .py file will be required upon by 
PyOpenGL.
Comment 1 Konstantin Ryabitsev 2007-08-10 06:07:26 EDT
Requirement on python-devel was added because of #240707. I don't see an easy
way to reconcile what you're asking and a requirement to be able to run
easy_install.
Comment 2 Hans de Goede 2007-08-10 06:26:36 EDT
(In reply to comment #1)
> Requirement on python-devel was added because of #240707. I don't see an easy
> way to reconcile what you're asking and a requirement to be able to run
> easy_install.

??

All I'm asking for is putting pkg_resources.py in its own sub-package and making
the rest of it require it, this will mean no differences what so ever for normal
setuptools users, but then PyOpenGL can just require the pkg_resources.py
containing package instead of needing to require the complete python-setuptools
and through that python-devel.

In other words I'm asking for a package split to stop python-devel ending up on
the live cd because of:
gnome-games -> PyOpenGL -> python-setuptools -> python-devel

And since there are packages which have runtime requires on parts of
python-setuptools I'm also suggestion to put pkg_resources.py in the main
package and the rest in a -devel subpackage. But if you prefer to keep most of
it in python-setuptools and put pkg_resources.py in
python-setuptools-pkg_resources that will work too. (its ugly as we now have a
main package depending on one of its own sub packages and on a -devel package,
but it will work).
Comment 3 Konstantin Ryabitsev 2007-08-10 08:28:29 EDT
Please ask for an RFC on fedora-devel. What you are requesting is a very drastic
change to a very fundamental package and thus must be discussed with the larger
community to see how other packages will be potentially affected.
Comment 4 Jeremy Katz 2007-08-10 10:23:34 EDT
I think that trying to split python-setuptools is probably not the right
approach here.  It is intended to just be a development package.  

I wonder how hard it would be to get PyOpenGL to not use pkg_resources and if
the upstream would be amenable to it.  Especially as there seems to be some
concern from the setuptools upstream on the stability of using pkg_resources
(the pkg_resources stuff is just based on a draft PEP and not anything more).
Comment 5 Hans de Goede 2007-08-10 13:48:35 EDT
(In reply to comment #4)
> I think that trying to split python-setuptools is probably not the right
> approach here.  It is intended to just be a development package.  
> 
> I wonder how hard it would be to get PyOpenGL to not use pkg_resources and if
> the upstream would be amenable to it.  Especially as there seems to be some
> concern from the setuptools upstream on the stability of using pkg_resources
> (the pkg_resources stuff is just based on a draft PEP and not anything more).

I have taken a look but fixing PyOpenGL is beyond my skills, I would be happy to
take a patch for this.

But I fail to see the problem with putting pkg_resources.py in its own
python-setuptools-pkg_resources subpackage and make the main package require the
subpackage, this changes _nothing_ for other uses of python-setuptools, is
without any chance for regressions, and fixes the problem at hand. To me
pkg_resources seems clearly designed for runtime use, and thus should be
seperated from the devel stuff.
Comment 6 Toshio Kuratomi 2007-08-10 14:31:56 EDT
(In reply to comment #4)
> I think that trying to split python-setuptools is probably not the right
> approach here.  It is intended to just be a development package.  
> 
> I wonder how hard it would be to get PyOpenGL to not use pkg_resources and if
> the upstream would be amenable to it.  Especially as there seems to be some
> concern from the setuptools upstream on the stability of using pkg_resources
> (the pkg_resources stuff is just based on a draft PEP and not anything more).

No.  pkg_resources definitely provides runtime services for python packages 
(plugins/modules with version information).

I think Hans is on the right track:  pkg_resource.py needs to be in a separate
subpackage from easy-install.  It's just a question of how the packages should
be broken up.

I think breaking the easy_install command into its own subpackage that Requires:
python-devel makes the most sense.

Note that some of the functionality of the setuptools module will be broken in
this scenario *but* python's builtin distutils is broken in the same manner.

(Try running /usr/lib64/python2.5/distutils/tests/test_install.py on a system
without python-devel to see what I mean.)
Comment 7 Toshio Ernie Kuratomi 2007-08-18 19:12:27 EDT
Created attachment 161810 [details]
Patch to implement utils subpackage

Patch to implement a python-setuptools-utils subpackage to contain easy_install
and the python-devel Require.
Comment 8 Hans de Goede 2007-08-19 02:07:52 EDT
(In reply to comment #7)
> Created an attachment (id=161810) [edit]
> Patch to implement utils subpackage
> 
> Patch to implement a python-setuptools-utils subpackage to contain easy_install
> and the python-devel Require.

Thanks for working on this!

Wouldn't it be better to call it python-setuptools-devel?

Also this fixes the dep, but still leaves a lot of python-setuptools in one
package for packages which only need pkg_resources.py

Atleast /usr/lib/python2.5/site-packages/easy_install.py* should also be in the
-utils package and not in the main package, but what about all the stuff under
/usr/lib/python2.5/site-packages/setuptools, can that be used runtime by other
packages too, or is it on;y used by easy_install.

Also while looking at this, I noticed a
/usr/lib/python2.5/site-packages/setuptools/tests dir, I don't think that we
want to be shipping this.
Comment 9 Toshio Ernie Kuratomi 2007-08-20 14:40:13 EDT
(In reply to comment #8)
> Wouldn't it be better to call it python-setuptools-devel?
> 
It contains an application which installs packages rather than anything to make
building easier so I lean towards -util.

> Also this fixes the dep, but still leaves a lot of python-setuptools in one
> package for packages which only need pkg_resources.py
> 
True but it doesn't seem to make all that much sense to break a single file
(with no extra dependencies) into its own subpackage.  I'd wait for upstream to
decide they have reason to break pkg_resources and setuptools apart personally,
before going to this level.

> Atleast /usr/lib/python2.5/site-packages/easy_install.py* should also be in the
> -utils package and not in the main package,

Agreed.  I researched that but forgot about it when I wrote the spec to generate
the patch.

> but what about all the stuff under
> /usr/lib/python2.5/site-packages/setuptools, can that be used runtime by other
> packages too, or is it on;y used by easy_install.
>
That module is equivalent to the distutils module in python so I think it should
remain.
 
> Also while looking at this, I noticed a
> /usr/lib/python2.5/site-packages/setuptools/tests dir, I don't think that we
> want to be shipping this.

I'll defer to icon on this.
Comment 10 Ignacio Vazquez-Abrams 2007-08-21 18:00:25 EDT
And now for my crazy idea. What about merging python-setuptools into python?
We'd leave pkg_resources and setuptools in python and move easy_install into
python-devel.
Comment 11 Jeremy Katz 2007-08-22 09:45:19 EDT
(In reply to comment #10)
> And now for my crazy idea. What about merging python-setuptools into python?
> We'd leave pkg_resources and setuptools in python and move easy_install into
> python-devel.

Given that it has a separate upstream with a separate release schedule, I'd be
somewhat against merging like that.

A -util subpackage is looking like it's going to be the best answer for now
Comment 12 Konstantin Ryabitsev 2007-08-22 10:04:59 EDT
Well, just excluding /usr/bin/easy_install* is not going to be enough, since
you'll notice that it's just a fancy way of calling
/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py. Plus we
also have /usr/lib/python2.5/site-packages/easy_install.py. Eventually, someone
will find a way to break things.

If we're going to be splitting the package, then the cleanest would be to stick
the contents of site-packages/setuptools/command into python-setuptools-devel,
but that would break the world due to changed dependencies. Alternatively, we
can create a separate package called python-setuptools-runtime and stick
site-packages/pkg_resources.py* into it, which would be least invasive. Then
python-setuptools would just require python-setuptools-runtime.

Thoughts?
Comment 13 Hans de Goede 2007-08-22 10:38:10 EDT
I would like to see the python-setuptools and python-setuptools-devel split,
this is how all other Fedora packages are splitting of a -runtime, because we
started with the devel files in the main package and packages depend on this is
just plain wrong.

Also since most (all but PyOpenGL?) packages only need python-setuptools, when
building we are breaking BR's here not normal Requires, and the packagers should
be able to handle this when a package build fails because of this.

So I say do a python-setuptools and python-setuptools-devel split and send a
mail to the devel list announcing this.
Comment 14 Konstantin Ryabitsev 2007-08-22 11:00:25 EDT
It is, perhaps, a bit late to do this in the F8 cycle?
Comment 15 Hans de Goede 2007-08-22 11:04:04 EDT
(In reply to comment #14)
> It is, perhaps, a bit late to do this in the F8 cycle?

Why, many packages still need to be rebuild, if you change it now, those will
get caught now. Also Matt Domsch will probably do some rawhide massrebuilds
before test3, those will catch the others.

The alternative is to do the -runtime now, and then later switch, that way lies
madness. As said, this is not going to break anything for endusers, only
rebuilds. Unless other packages Require python-setuptools, which can easily be
queried with repoquery, and then fixed.
Comment 16 Toshio Ernie Kuratomi 2007-08-22 11:53:29 EDT
Created attachment 162071 [details]
New patch that moves site-packages/easy_install*

This patch includes /usr/lib/python2.5/site-packages/easy_install.py* in the
-utils subpackage.
Comment 17 Toshio Ernie Kuratomi 2007-08-22 11:55:22 EDT
(In reply to comment #12)
> Well, just excluding /usr/bin/easy_install* is not going to be enough, since
> you'll notice that it's just a fancy way of calling
> /usr/lib/python2.5/site-packages/setuptools/command/easy_install.py. 
[snip]
> If we're going to be splitting the package, then the cleanest would be to stick
> the contents of site-packages/setuptools/command into python-setuptools-devel,
> but that would break the world due to changed dependencies.

 /usr/lib/python2.5/site-packages/easy_install.py can safely be moved into the
util package.  Nothing under site-packages/setuptools can be moved without
breaking things.

> Alternatively, we
> can create a separate package called python-setuptools-runtime and stick
> site-packages/pkg_resources.py* into it, which would be least invasive. Then
> python-setuptools would just require python-setuptools-runtime.
> 
I could go for this but I will note that distutils is broken in the same way
that setuptools is.  ie: running certain distutils commands without python-devel
installed will raise an exception.  The only difference is that distutils
doesn't have an easy_install script to make this obvious.  So putting
easy_install in a subpackage makes sense to me for that reason.
Comment 18 Toshio Ernie Kuratomi 2007-08-22 12:03:08 EDT
(In reply to comment #15)
> (In reply to comment #14)
> > It is, perhaps, a bit late to do this in the F8 cycle?
> 
> Why, many packages still need to be rebuild, if you change it now, those will
> get caught now. Also Matt Domsch will probably do some rawhide massrebuilds
> before test3, those will catch the others.
>
If I'm reading the initial problem right, not fixing this for F8 means that some
livecd spins will be impossible because of size constraints.  So this is an
important fix.

> The alternative is to do the -runtime now, and then later switch, that way lies
> madness. As said, this is not going to break anything for endusers, only
> rebuilds. Unless other packages Require python-setuptools, which can easily be
> queried with repoquery, and then fixed.
> 
python-setuptools-runtime will break Requires (have to update python-setuptools
=> python-setuptools-runtime)

python-setuptools-devel will break BuildRequires.

python-setuptools-utils shouldn't break any rpm dependencies but it will cause
people using easy_install to have to install another package.

Jeremy:  An alternative is to move these files:
  %{_includedir}/python2.5/pyconfig-*.h
  %{_python_sitearch}/config/Makefile

from python-devel to python.  That will fix distutils and setuptools without
having to Require: python-devel.
Comment 19 Toshio Ernie Kuratomi 2007-08-24 13:57:08 EDT
Conversation with jeremy::

(10:34:43 AM) abadger1999: jeremy: What do you think of adding the Makefile and
pyconfig-*.h files to python?
(10:34:52 AM) abadger1999: In regards to the setuptools bug
(10:35:54 AM) jeremy: abadger1999: I'd rather not
(10:36:45 AM) jeremy: abadger1999: since it feels a bit hacky....  but I might
grudgingly do it
(10:36:46 AM) abadger1999: k.  I'll go ahead and ask icon if it's okay to go
with my last patch to split setuptools then.
(10:37:45 AM) abadger1999: jeremy: The only advantage I can see is that it's
technically broken in both setuptools and distutils.
(10:38:27 AM) abadger1999: jeremy: But I don't think people run across it unless
they're running easy_install from setuptools.  So splitting the package is fine
with me.
(10:38:40 AM) jeremy: abadger1999: really, distutils should be in a separate
package.  but mistake of the past

So it seems we're all agreed that splitting setuptools is the way to go.  Icon,
you're the maintainer, so I guess it comes down to your choice now :-)

* python-setuptools-devel with the bulk of code, python-setuptools with
pkg_resources.  python-setuptools-devel Requires: python-setuptools, python-devel

* python-setuptools with the bulk of code, python-setuptools-runtime with
pkg_resources.  python-setuptools Requires: python-setuptools-runtime, python-devel

* python-setuptools with the builk of code, python-setuptools-utils with
easy_install.  python-setuptools-utils Requires: python-setuptools, python-devel
Comment 20 Jeremy Katz 2007-08-27 15:18:26 EDT
FWIW, I'd vaguely lean towards the middle option just for the lack of breaking
lots of things building.  But no strong opinion.  It'd be really good if we
could get this in the next day or so as I'm having "fun" getting live images to
fit for test2 
Comment 21 Konstantin Ryabitsev 2007-08-27 19:29:57 EDT
OK, my final executive decision was to "do the right thing" and split off
python-setuptools and python-setuptools-devel. The changes have been committed
in devel, but I'm not building it yet until everyone agrees that this resolves
the issue and is the right way forward.
Comment 22 Jeremy Katz 2007-08-27 20:25:51 EDT
ACK from me -- send a heads-up to fedora-devel-announce and build on :)
Comment 23 Toshio Ernie Kuratomi 2007-08-27 23:06:55 EDT
The changes in cvs look good to me too.
Comment 24 Hans de Goede 2007-08-28 03:24:52 EDT
And the changes in CVS look fine to me too, build away.
Comment 25 Konstantin Ryabitsev 2007-08-28 08:37:06 EDT
Package built, the world notified.
Let the moaning begin! :)
Comment 26 Toshio Ernie Kuratomi 2007-09-05 15:51:52 EDT
Well, I've found a problem with this strategy.  Some packages need to use
setuptools in order to use setuptools' plugin architecture.  Inside their
setup.py, they create a dependency on setuptools>=VERSION.

When you attempt to use the package, you get the following traceback:
Traceback (most recent call last):
  File "./start-tester.py", line 3, in <module>
    pkg_resources.require("TurboGears")
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 524, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: setuptools>=0.6c2

This is from pkg_resources attempting to find the relevant setuptools module and
egg.

So we need to move the setuptools module from python-setuptools-devel to
python-setuptools.
Comment 27 Hans de Goede 2007-09-05 16:00:24 EDT
(In reply to comment #26)
> Well, I've found a problem with this strategy.  Some packages need to use
> setuptools in order to use setuptools' plugin architecture.  Inside their
> setup.py, they create a dependency on setuptools>=VERSION.
> 
> When you attempt to use the package, you get the following traceback:
> Traceback (most recent call last):
>   File "./start-tester.py", line 3, in <module>
>     pkg_resources.require("TurboGears")
>   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626, in require
>     needed = self.resolve(parse_requirements(requirements))
>   File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 524, in resolve
>     raise DistributionNotFound(req)  # XXX put more info here
> pkg_resources.DistributionNotFound: setuptools>=0.6c2
> 
> This is from pkg_resources attempting to find the relevant setuptools module and
> egg.
> 
> So we need to move the setuptools module from python-setuptools-devel to
> python-setuptools.

I'm not a python expert (far from), but couldn't one argue, that if all they
need form setup-tools is pkg_resources that they shouldn't be requiring
setuptools, and that the proper fix is to remove the setuptools>=0.6c2 fromt
here requirements in their setup.py?

Comment 28 Toshio Ernie Kuratomi 2007-09-05 16:18:48 EDT
(In reply to comment #27)

> I'm not a python expert (far from), but couldn't one argue, that if all they
> need form setup-tools is pkg_resources that they shouldn't be requiring
> setuptools, and that the proper fix is to remove the setuptools>=0.6c2 fromt
> here requirements in their setup.py?

One could but TurboGears actually requires setuptools, not just pkg_resources
Comment 29 Ignacio Vazquez-Abrams 2007-09-05 16:54:19 EDT
Then it should either require setuptools-devel or there needs to be a
TurboGears-devel subpackage.
Comment 30 Toshio Ernie Kuratomi 2007-09-05 17:46:19 EDT
I disagree.  That puts us back where we started with a non-devel package pulling
in python-devel.  I think the reasoning should be:

It's legal to use the setuptools module at runtime, therefore the setuptools
module should not be in the -devel package which brings in python-devel.
Comment 31 Matt Domsch 2007-09-09 12:38:33 EDT
at this point, TurboGears fails to work unless python-setuptools-devel is
installed.  It took me a while to figure this out too.  Of course, it only bit
me while I was working offline and couldn't yum install anything. :-(
Comment 32 Toshio Ernie Kuratomi 2007-09-11 15:33:48 EDT
Put another way: the setuptools module is used by a tool at runtime.  The tools
that are built from it are commonly tools needed at buildtime but that doesn't
limit the usefulness of the library in other situations.  We wouldn't allow
"make" to have a dependency on a -devel package even though it's a buildtime.
Comment 33 Konstantin Ryabitsev 2007-09-14 15:50:27 EDT
OK, I've tried to intelligently separate things, but I don't have a moment to
test it with a package that requires it at runtime. It's committed in CVS --
Toshio, could you check it out and tell me if that resolves your problem?
Comment 34 Toshio Ernie Kuratomi 2007-09-14 16:19:21 EDT
We probably don't want to split the setuptools/command module from the main
setuptools package as things in setuptools import things from setuptools.command.

Only %{python_sitelib}/easy_install* and %{_bindir}/* can be safely moved into
the -devel package.
Comment 35 Konstantin Ryabitsev 2007-09-14 16:30:07 EDT
Ugh... well, that puts us back to where we started from, as some of the things
in setuptools/command will want stuff from python-devel. E.g. witness
setuptools/command/easy_install.py.

I'm afraid this is proving to be an exercise in futility. :/
Comment 36 Konstantin Ryabitsev 2007-09-18 06:01:57 EDT
Well, we need to decide what to do before F8 freezes over. I guess, ultimately
we can go with what's suggested in #34, since it's less likely that someone
would need to use easy_install on an RPM system.
Comment 37 Toshio Ernie Kuratomi 2007-09-18 12:19:10 EDT
comment 34 gets my vote ;-).
Comment 38 Toshio Ernie Kuratomi 2007-09-24 14:44:43 EDT
Created attachment 204411 [details]
Patch implementing comment #34

Here's a patch to implement comment #34.  Icon, I'll apply this today and build
it unless someone comes up with something better since freeze will be here
tomorrow.
Comment 39 Konstantin Ryabitsev 2007-09-24 14:51:25 EDT
I'm ready to commit to CVS and build in a moment.
Comment 40 Konstantin Ryabitsev 2007-09-24 14:58:36 EDT
Build is on the way http://koji.fedoraproject.org/koji/taskinfo?taskID=172375
Comment 41 Karsten Hopp 2007-09-26 10:08:34 EDT
Turbogears still won't start with 
python-setuptools-0.6c7-2.fc8
TurboGears-1.0.3.2-1.fc8
python-setuptools-devel-0.6c7-2.fc8

It's the same error as mentioned in comment #26
Comment 42 Konstantin Ryabitsev 2007-09-26 10:24:24 EDT
Well, if it's actually trying to import easy_install, then it's incorrect
behaviour and should be fixed in turbogears.
Comment 43 Konstantin Ryabitsev 2007-11-18 12:52:06 EST
Please reopen if there are any remaining concerns.

Note You need to log in before you can comment on or make changes to this bug.