Description of problem: gnome-games now comes with glchess which requires PyOpenGL for its 3d rendering. PyOpenGL in turn requires python-setuptools (it uses it runtime, see below), which drags in python-devel. To see the use of setuptools in PyOpenGL, install PyOpenGL and then open: /usr/lib/python2.5/site-packages/OpenGL/arrays/formathandler.py And then at line 8: import pkg_resources And line 37-39: for entrypoint in pkg_resources.iter_entry_points( "OpenGL.arrays.formathandler" ): So all PyOpenGL needs from python-setuptools is pkg_resources.py, I've verified this be rpm -e python-setuptools and then manually restoring pkg_resources.py I see 2 possible solutions: 1) Since most of python-setuptools is meant for development put all of python-setuptools execept for pkg_resources.py in python-setuptools-devel, this esp. makes sense because then python-setuptools-devel would require python-devel, instead of a "normal" package requiring a -devel package. And then have the main python-setuptools package contain pkg_resources.py To me this is the prefered solution, disadvantage anything BuildRequiring python-setuptools needs to update its BR's, but this can be coordinated with a mail to the maintainers list and is OK to do in Rawhide 2) put pkg_resources.py in python-setuptools-pkg_resources, and make the main package require it. This feels wrong though. Esp the make the main package require it part says to me that we are doing things the wrong way around then. I've also looked at other possible runtime use candidates in python-setuptools, and I couldn't find any. p.s. Jeremy I've put you in the CC because the draggin of python-setuptools caused size problems for the livecd, once this bug is resolved that should no longer be a problem as then a package with only one .py file will be required upon by PyOpenGL.
Requirement on python-devel was added because of #240707. I don't see an easy way to reconcile what you're asking and a requirement to be able to run easy_install.
(In reply to comment #1) > Requirement on python-devel was added because of #240707. I don't see an easy > way to reconcile what you're asking and a requirement to be able to run > easy_install. ?? All I'm asking for is putting pkg_resources.py in its own sub-package and making the rest of it require it, this will mean no differences what so ever for normal setuptools users, but then PyOpenGL can just require the pkg_resources.py containing package instead of needing to require the complete python-setuptools and through that python-devel. In other words I'm asking for a package split to stop python-devel ending up on the live cd because of: gnome-games -> PyOpenGL -> python-setuptools -> python-devel And since there are packages which have runtime requires on parts of python-setuptools I'm also suggestion to put pkg_resources.py in the main package and the rest in a -devel subpackage. But if you prefer to keep most of it in python-setuptools and put pkg_resources.py in python-setuptools-pkg_resources that will work too. (its ugly as we now have a main package depending on one of its own sub packages and on a -devel package, but it will work).
Please ask for an RFC on fedora-devel. What you are requesting is a very drastic change to a very fundamental package and thus must be discussed with the larger community to see how other packages will be potentially affected.
I think that trying to split python-setuptools is probably not the right approach here. It is intended to just be a development package. I wonder how hard it would be to get PyOpenGL to not use pkg_resources and if the upstream would be amenable to it. Especially as there seems to be some concern from the setuptools upstream on the stability of using pkg_resources (the pkg_resources stuff is just based on a draft PEP and not anything more).
(In reply to comment #4) > I think that trying to split python-setuptools is probably not the right > approach here. It is intended to just be a development package. > > I wonder how hard it would be to get PyOpenGL to not use pkg_resources and if > the upstream would be amenable to it. Especially as there seems to be some > concern from the setuptools upstream on the stability of using pkg_resources > (the pkg_resources stuff is just based on a draft PEP and not anything more). I have taken a look but fixing PyOpenGL is beyond my skills, I would be happy to take a patch for this. But I fail to see the problem with putting pkg_resources.py in its own python-setuptools-pkg_resources subpackage and make the main package require the subpackage, this changes _nothing_ for other uses of python-setuptools, is without any chance for regressions, and fixes the problem at hand. To me pkg_resources seems clearly designed for runtime use, and thus should be seperated from the devel stuff.
(In reply to comment #4) > I think that trying to split python-setuptools is probably not the right > approach here. It is intended to just be a development package. > > I wonder how hard it would be to get PyOpenGL to not use pkg_resources and if > the upstream would be amenable to it. Especially as there seems to be some > concern from the setuptools upstream on the stability of using pkg_resources > (the pkg_resources stuff is just based on a draft PEP and not anything more). No. pkg_resources definitely provides runtime services for python packages (plugins/modules with version information). I think Hans is on the right track: pkg_resource.py needs to be in a separate subpackage from easy-install. It's just a question of how the packages should be broken up. I think breaking the easy_install command into its own subpackage that Requires: python-devel makes the most sense. Note that some of the functionality of the setuptools module will be broken in this scenario *but* python's builtin distutils is broken in the same manner. (Try running /usr/lib64/python2.5/distutils/tests/test_install.py on a system without python-devel to see what I mean.)
Created attachment 161810 [details] Patch to implement utils subpackage Patch to implement a python-setuptools-utils subpackage to contain easy_install and the python-devel Require.
(In reply to comment #7) > Created an attachment (id=161810) [edit] > Patch to implement utils subpackage > > Patch to implement a python-setuptools-utils subpackage to contain easy_install > and the python-devel Require. Thanks for working on this! Wouldn't it be better to call it python-setuptools-devel? Also this fixes the dep, but still leaves a lot of python-setuptools in one package for packages which only need pkg_resources.py Atleast /usr/lib/python2.5/site-packages/easy_install.py* should also be in the -utils package and not in the main package, but what about all the stuff under /usr/lib/python2.5/site-packages/setuptools, can that be used runtime by other packages too, or is it on;y used by easy_install. Also while looking at this, I noticed a /usr/lib/python2.5/site-packages/setuptools/tests dir, I don't think that we want to be shipping this.
(In reply to comment #8) > Wouldn't it be better to call it python-setuptools-devel? > It contains an application which installs packages rather than anything to make building easier so I lean towards -util. > Also this fixes the dep, but still leaves a lot of python-setuptools in one > package for packages which only need pkg_resources.py > True but it doesn't seem to make all that much sense to break a single file (with no extra dependencies) into its own subpackage. I'd wait for upstream to decide they have reason to break pkg_resources and setuptools apart personally, before going to this level. > Atleast /usr/lib/python2.5/site-packages/easy_install.py* should also be in the > -utils package and not in the main package, Agreed. I researched that but forgot about it when I wrote the spec to generate the patch. > but what about all the stuff under > /usr/lib/python2.5/site-packages/setuptools, can that be used runtime by other > packages too, or is it on;y used by easy_install. > That module is equivalent to the distutils module in python so I think it should remain. > Also while looking at this, I noticed a > /usr/lib/python2.5/site-packages/setuptools/tests dir, I don't think that we > want to be shipping this. I'll defer to icon on this.
And now for my crazy idea. What about merging python-setuptools into python? We'd leave pkg_resources and setuptools in python and move easy_install into python-devel.
(In reply to comment #10) > And now for my crazy idea. What about merging python-setuptools into python? > We'd leave pkg_resources and setuptools in python and move easy_install into > python-devel. Given that it has a separate upstream with a separate release schedule, I'd be somewhat against merging like that. A -util subpackage is looking like it's going to be the best answer for now
Well, just excluding /usr/bin/easy_install* is not going to be enough, since you'll notice that it's just a fancy way of calling /usr/lib/python2.5/site-packages/setuptools/command/easy_install.py. Plus we also have /usr/lib/python2.5/site-packages/easy_install.py. Eventually, someone will find a way to break things. If we're going to be splitting the package, then the cleanest would be to stick the contents of site-packages/setuptools/command into python-setuptools-devel, but that would break the world due to changed dependencies. Alternatively, we can create a separate package called python-setuptools-runtime and stick site-packages/pkg_resources.py* into it, which would be least invasive. Then python-setuptools would just require python-setuptools-runtime. Thoughts?
I would like to see the python-setuptools and python-setuptools-devel split, this is how all other Fedora packages are splitting of a -runtime, because we started with the devel files in the main package and packages depend on this is just plain wrong. Also since most (all but PyOpenGL?) packages only need python-setuptools, when building we are breaking BR's here not normal Requires, and the packagers should be able to handle this when a package build fails because of this. So I say do a python-setuptools and python-setuptools-devel split and send a mail to the devel list announcing this.
It is, perhaps, a bit late to do this in the F8 cycle?
(In reply to comment #14) > It is, perhaps, a bit late to do this in the F8 cycle? Why, many packages still need to be rebuild, if you change it now, those will get caught now. Also Matt Domsch will probably do some rawhide massrebuilds before test3, those will catch the others. The alternative is to do the -runtime now, and then later switch, that way lies madness. As said, this is not going to break anything for endusers, only rebuilds. Unless other packages Require python-setuptools, which can easily be queried with repoquery, and then fixed.
Created attachment 162071 [details] New patch that moves site-packages/easy_install* This patch includes /usr/lib/python2.5/site-packages/easy_install.py* in the -utils subpackage.
(In reply to comment #12) > Well, just excluding /usr/bin/easy_install* is not going to be enough, since > you'll notice that it's just a fancy way of calling > /usr/lib/python2.5/site-packages/setuptools/command/easy_install.py. [snip] > If we're going to be splitting the package, then the cleanest would be to stick > the contents of site-packages/setuptools/command into python-setuptools-devel, > but that would break the world due to changed dependencies. /usr/lib/python2.5/site-packages/easy_install.py can safely be moved into the util package. Nothing under site-packages/setuptools can be moved without breaking things. > Alternatively, we > can create a separate package called python-setuptools-runtime and stick > site-packages/pkg_resources.py* into it, which would be least invasive. Then > python-setuptools would just require python-setuptools-runtime. > I could go for this but I will note that distutils is broken in the same way that setuptools is. ie: running certain distutils commands without python-devel installed will raise an exception. The only difference is that distutils doesn't have an easy_install script to make this obvious. So putting easy_install in a subpackage makes sense to me for that reason.
(In reply to comment #15) > (In reply to comment #14) > > It is, perhaps, a bit late to do this in the F8 cycle? > > Why, many packages still need to be rebuild, if you change it now, those will > get caught now. Also Matt Domsch will probably do some rawhide massrebuilds > before test3, those will catch the others. > If I'm reading the initial problem right, not fixing this for F8 means that some livecd spins will be impossible because of size constraints. So this is an important fix. > The alternative is to do the -runtime now, and then later switch, that way lies > madness. As said, this is not going to break anything for endusers, only > rebuilds. Unless other packages Require python-setuptools, which can easily be > queried with repoquery, and then fixed. > python-setuptools-runtime will break Requires (have to update python-setuptools => python-setuptools-runtime) python-setuptools-devel will break BuildRequires. python-setuptools-utils shouldn't break any rpm dependencies but it will cause people using easy_install to have to install another package. Jeremy: An alternative is to move these files: %{_includedir}/python2.5/pyconfig-*.h %{_python_sitearch}/config/Makefile from python-devel to python. That will fix distutils and setuptools without having to Require: python-devel.
Conversation with jeremy:: (10:34:43 AM) abadger1999: jeremy: What do you think of adding the Makefile and pyconfig-*.h files to python? (10:34:52 AM) abadger1999: In regards to the setuptools bug (10:35:54 AM) jeremy: abadger1999: I'd rather not (10:36:45 AM) jeremy: abadger1999: since it feels a bit hacky.... but I might grudgingly do it (10:36:46 AM) abadger1999: k. I'll go ahead and ask icon if it's okay to go with my last patch to split setuptools then. (10:37:45 AM) abadger1999: jeremy: The only advantage I can see is that it's technically broken in both setuptools and distutils. (10:38:27 AM) abadger1999: jeremy: But I don't think people run across it unless they're running easy_install from setuptools. So splitting the package is fine with me. (10:38:40 AM) jeremy: abadger1999: really, distutils should be in a separate package. but mistake of the past So it seems we're all agreed that splitting setuptools is the way to go. Icon, you're the maintainer, so I guess it comes down to your choice now :-) * python-setuptools-devel with the bulk of code, python-setuptools with pkg_resources. python-setuptools-devel Requires: python-setuptools, python-devel * python-setuptools with the bulk of code, python-setuptools-runtime with pkg_resources. python-setuptools Requires: python-setuptools-runtime, python-devel * python-setuptools with the builk of code, python-setuptools-utils with easy_install. python-setuptools-utils Requires: python-setuptools, python-devel
FWIW, I'd vaguely lean towards the middle option just for the lack of breaking lots of things building. But no strong opinion. It'd be really good if we could get this in the next day or so as I'm having "fun" getting live images to fit for test2
OK, my final executive decision was to "do the right thing" and split off python-setuptools and python-setuptools-devel. The changes have been committed in devel, but I'm not building it yet until everyone agrees that this resolves the issue and is the right way forward.
ACK from me -- send a heads-up to fedora-devel-announce and build on :)
The changes in cvs look good to me too.
And the changes in CVS look fine to me too, build away.
Package built, the world notified. Let the moaning begin! :)
Well, I've found a problem with this strategy. Some packages need to use setuptools in order to use setuptools' plugin architecture. Inside their setup.py, they create a dependency on setuptools>=VERSION. When you attempt to use the package, you get the following traceback: Traceback (most recent call last): File "./start-tester.py", line 3, in <module> pkg_resources.require("TurboGears") File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 524, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: setuptools>=0.6c2 This is from pkg_resources attempting to find the relevant setuptools module and egg. So we need to move the setuptools module from python-setuptools-devel to python-setuptools.
(In reply to comment #26) > Well, I've found a problem with this strategy. Some packages need to use > setuptools in order to use setuptools' plugin architecture. Inside their > setup.py, they create a dependency on setuptools>=VERSION. > > When you attempt to use the package, you get the following traceback: > Traceback (most recent call last): > File "./start-tester.py", line 3, in <module> > pkg_resources.require("TurboGears") > File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626, in require > needed = self.resolve(parse_requirements(requirements)) > File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 524, in resolve > raise DistributionNotFound(req) # XXX put more info here > pkg_resources.DistributionNotFound: setuptools>=0.6c2 > > This is from pkg_resources attempting to find the relevant setuptools module and > egg. > > So we need to move the setuptools module from python-setuptools-devel to > python-setuptools. I'm not a python expert (far from), but couldn't one argue, that if all they need form setup-tools is pkg_resources that they shouldn't be requiring setuptools, and that the proper fix is to remove the setuptools>=0.6c2 fromt here requirements in their setup.py?
(In reply to comment #27) > I'm not a python expert (far from), but couldn't one argue, that if all they > need form setup-tools is pkg_resources that they shouldn't be requiring > setuptools, and that the proper fix is to remove the setuptools>=0.6c2 fromt > here requirements in their setup.py? One could but TurboGears actually requires setuptools, not just pkg_resources
Then it should either require setuptools-devel or there needs to be a TurboGears-devel subpackage.
I disagree. That puts us back where we started with a non-devel package pulling in python-devel. I think the reasoning should be: It's legal to use the setuptools module at runtime, therefore the setuptools module should not be in the -devel package which brings in python-devel.
at this point, TurboGears fails to work unless python-setuptools-devel is installed. It took me a while to figure this out too. Of course, it only bit me while I was working offline and couldn't yum install anything. :-(
Put another way: the setuptools module is used by a tool at runtime. The tools that are built from it are commonly tools needed at buildtime but that doesn't limit the usefulness of the library in other situations. We wouldn't allow "make" to have a dependency on a -devel package even though it's a buildtime.
OK, I've tried to intelligently separate things, but I don't have a moment to test it with a package that requires it at runtime. It's committed in CVS -- Toshio, could you check it out and tell me if that resolves your problem?
We probably don't want to split the setuptools/command module from the main setuptools package as things in setuptools import things from setuptools.command. Only %{python_sitelib}/easy_install* and %{_bindir}/* can be safely moved into the -devel package.
Ugh... well, that puts us back to where we started from, as some of the things in setuptools/command will want stuff from python-devel. E.g. witness setuptools/command/easy_install.py. I'm afraid this is proving to be an exercise in futility. :/
Well, we need to decide what to do before F8 freezes over. I guess, ultimately we can go with what's suggested in #34, since it's less likely that someone would need to use easy_install on an RPM system.
comment 34 gets my vote ;-).
Created attachment 204411 [details] Patch implementing comment #34 Here's a patch to implement comment #34. Icon, I'll apply this today and build it unless someone comes up with something better since freeze will be here tomorrow.
I'm ready to commit to CVS and build in a moment.
Build is on the way http://koji.fedoraproject.org/koji/taskinfo?taskID=172375
Turbogears still won't start with python-setuptools-0.6c7-2.fc8 TurboGears-1.0.3.2-1.fc8 python-setuptools-devel-0.6c7-2.fc8 It's the same error as mentioned in comment #26
Well, if it's actually trying to import easy_install, then it's incorrect behaviour and should be fixed in turbogears.
Please reopen if there are any remaining concerns.