Bug 523210 - python-twisted doesn't provide an .egg-info file
Summary: python-twisted doesn't provide an .egg-info file
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-twisted
Version: 14
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Julian Sikorski
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-09-14 13:41 UTC by Ruben Kerkhof
Modified: 2010-11-17 23:26 UTC (History)
3 users (show)

Fixed In Version: python-twisted-10.1.0-3.fc14
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-11-17 23:26:57 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
a .egg-info directory as produced by "python setup.py install" (8.26 KB, application/x-bzip2)
2010-07-19 15:59 UTC, Zooko O'Whielacronx
no flags Details

Description Ruben Kerkhof 2009-09-14 13:41:15 UTC
Description of problem:
All the Twisted subpackages have .egg-info files, but the main Twisted package, python-twisted, doesn't. This breaks setuptools 'install_requires=' feature.


Version-Release number of selected component (if applicable):
python-twisted-8.2.0-3.fc12.noarch

How reproducible:
python -c 'import pkg_resources;print pkg_resources.require("Twisted>=2.4.0")'

Actual results:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 626, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 524, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: Twisted>=2.4.0


Expected results:
[Twisted 8.2.0 (/usr/lib64/python2.6/site-packages)]

Additional info:
More info can be found in this debian bug:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=477384

Comment 1 Bug Zapper 2009-11-16 12:21:36 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 2 Ruben Kerkhof 2010-01-31 14:48:07 UTC
Thomas, ping?

Comment 3 Bug Zapper 2010-03-15 12:49:44 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 4 Thomas Vander Stichele 2010-05-24 10:33:44 UTC
Hi Ruben,

should it be possible to do this at all ? Who defines what names are valid to use for setuptools ? How would you for example require only words ? or only the core of twisted ? Is there a canonical list of requirements that works across distros ?

Comment 5 Ruben Kerkhof 2010-06-06 11:41:23 UTC
Hi Thomas!

AFAIK there's not cross-distro list of requirements that work. I bumped into this while packaging Tahoe, which Requires twisted. It seems most other distros supply an egg-file for the whole Twisted set.

Comment 6 Zooko O'Whielacronx 2010-07-19 15:59:29 UTC
Created attachment 432933 [details]
a .egg-info directory as produced by "python setup.py install"

Comment 7 Zooko O'Whielacronx 2010-07-19 15:59:43 UTC
All Python packages (which Python calls "distributions" instead of "packages") should come with a directory or file named $PKGNAME-$VERSION.egg-info or if it is specific to a specific Python version it should be named $PKGNAME-$VERSION-$PYVERSION.egg-info.

For Twisted, PKGNAME="Twisted". (This field is defined as being evaluated case-insensitively, but it wouldn't hurt to use the correct capitalization -- "Twisted" -- in case some tools don't respect that standard.)

For example:

http://packages.debian.org/sid/all/python-twisted/filelist

Filelist of package python-twisted in sid of architecture all

/usr/share/pyshared/Twisted-10.0.0.egg-info

The actual contents of this directory or file are produced automatically by distutils when you run "python setup.py install". Here is an example which I just produced by running "python setup.py install" with the current SVN trunk of Twisted on my Mac OS X machine (attached as Twisted-10.1.0_r29874-py2.6.egg-info.tar.bz2 ).

Comment 8 Bug Zapper 2010-07-30 10:43:53 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 9 Zooko O'Whielacronx 2010-09-02 04:21:30 UTC
What can I do to help move this ticket along? I could try to learn a bit more about Fedora packaging and submit a proper patch. Could someone advise me on how to do that?

Comment 10 Zooko O'Whielacronx 2010-09-22 16:07:07 UTC
Hi folks, this bug is continuing to cause problems for me and my users. As far as I know the fix should be quite straightforward, as I described in comment 6 and comment 7. If someone could give me more information about how the Fedora package is generated then I could suggest how to patch it to generate the .egg-info file or to preserve that file and avoid deleting it (since it gets generated automatically by distutils).

What can I do to help?

Comment 11 Julian Sikorski 2010-09-23 20:32:35 UTC
Hi Zooko, I am the new twisted co-maintainer. Fedora twisted package is just a metapackage, which merely depends on the other subpackages, all of which are built from separate tarballs. I just pushed a rebase to 10.1.0, but it seems the problem is still there.

Comment 12 Zooko O'Whielacronx 2010-09-23 21:09:45 UTC
Hi Julian! Thank you for contributing your time. :-)

How is the Twisted-Core package built? As far as I know, when you run "python setup.py build" in the twisted source tree, then it should produce a .egg-info file named "Twisted". If you will include this .egg-info file under the name 'Twisted-something' not 'Twisted-Core-something' then I believe this bug would be fixed.

Comment 13 Julian Sikorski 2010-09-23 21:44:08 UTC
python-twisted-core is built from the TwistedCore tarball, and the egg-info file is not renamed as a part of the rpm build process. In other words, it is named as setup.py names it.

Comment 14 Zooko O'Whielacronx 2010-09-23 21:55:46 UTC
Oh I see, you are using the "TwistedCore" and other sub-package tarballs. That explains it. For the long term, I think you might consider moving back to the monolithic Twisted tarball, which seems to be the primary distribution method.

But anyway, for the short term, I think the right thing is *not* for the Fedora TwistedCore package to provide the Twisted .egg-info, like I said earlier, but instead for the Fedora Twisted meta-package to provide the Twisted .egg-info file.

Could you provide such a file as I posted in comment 6?

Thanks.

Comment 15 Julian Sikorski 2010-09-23 22:02:23 UTC
Yeah, I just had a closer look on how Debian do it, and it seems they ship their own egg-info file. I'll add such fix once the dust after the massive upgrade to 10.1.0 settles.

Comment 16 Julian Sikorski 2010-09-30 19:26:41 UTC
Hmm, I'm not sure how to handle python version (the latter one) in a clean way:
/usr/lib64/python2.6/site-packages/Twisted_Core-10.1.0-py2.6.egg-info
vs
/usr/lib/python2.7/site-packages/Twisted_Core-10.1.0-py2.7.egg-info
Any suggestions?

Comment 17 Zooko O'Whielacronx 2010-09-30 21:13:32 UTC
You can test what will happen with this command-line:

python -c "import pkg_resources;print pkg_resources.require('Twisted')"

Replace "python" with "python2.6" or "python2.7" as appropriate.

Note that the .egg-info file needs to be in a directory which is in the list of "site dirs". See this Ubuntu bug for more detail about this "site dirs" issue:

https://bugs.launchpad.net/ubuntu/+source/python-support/+bug/254035

Comment 18 Julian Sikorski 2010-09-30 21:21:57 UTC
The folder is OK, it is handled via python_sitearch macro. The problem is the py2.6 or py2.7 in the egg-info filename.
As python-twisted metapackage is currently built without any source at all, I need to figure out a way to manually give this filename in the .spec. Some bash wizardry would definitely be useful.

Comment 19 Zooko O'Whielacronx 2010-09-30 22:13:53 UTC
Try just leaving that part off! Make a file named Twisted-10.1.0.egg-info . That might work. :-)

Comment 20 Zooko O'Whielacronx 2010-09-30 22:16:04 UTC
Oh wait, Twisted does actually have a compiled extension module in it (for optimization purposes). So if you have Twisted for python 2.6 installed, you won't be able to load and use it with python 2.7, right? In that case, it would be a mistake to leave off the Python version number and allow the following code to succeed on python 2.7:

python -c 'import pkg_resources;print pkg_resources.require("Twisted")'

Comment 21 Zooko O'Whielacronx 2010-10-02 21:42:32 UTC
Hm, I went and looked, and I don't see where you are getting the separate tarballs like TwistedCore. As far as I know, the upstream Twisted project distributes only one unified tarball, e.g.: http://tmrc.mit.edu/mirror/twisted/Twisted/10.1/ . Where are you getting a "TwistedCore" tarball?

Comment 22 Julian Sikorski 2010-10-02 21:49:35 UTC
http://tmrc.mit.edu/mirror/twisted/Core/10.1/TwistedCore-10.1.0.tar.bz2
On an unrelated note, the backlight in my laptop screen has just failed, so my packaging work will be on hold for now.

Comment 23 Julian Sikorski 2010-10-03 19:30:53 UTC
Luckily it seems to have fixed itself on it's own. Strange.

Comment 24 Julian Sikorski 2010-10-03 19:33:14 UTC
Furthermore: we might consider using the big tarball to build the packages, but there are several questions: first of all, are individual egg-info files built as well when you do it?
Also, we would need Thomas's approval; I just joined as a co-maintainer and I am not going to push such a huge change without his agreement. A new review would probably be needed too.

Comment 25 Zooko O'Whielacronx 2010-10-30 08:23:48 UTC
This problem continues to bother the Tahoe-LAFS project. Here is a note on an issue ticket mentioning this bug: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1190#comment:34

Here is a buildbot report showing how running the Tahoe-LAFS tests from an installed version of Tahoe-LAFS on Fedora fails because Tahoe-LAFS checks (when it is run) that the "Twisted" python distribution (package) is present, and even though the "twisted" module is present, we can't tell that the "Twisted" python distribution (package) is present, due to the missing .egg-info:

http://tahoe-lafs.org/buildbot/builders/Ruben%20Fedora/builds/665/steps/test-from-prefixdir/logs/stdio

If we got this issue fixed in Fedora and then Ruben upgraded his buildslave to this fixed version of Fedora then I believe that test would go from red to green.

Comment 26 Fedora Update System 2010-11-07 09:03:36 UTC
python-twisted-10.1.0-2.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/python-twisted-10.1.0-2.fc14

Comment 27 Fedora Update System 2010-11-07 21:27:20 UTC
python-twisted-10.1.0-2.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update python-twisted'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/python-twisted-10.1.0-2.fc14

Comment 28 Fedora Update System 2010-11-10 01:16:11 UTC
python-twisted-10.1.0-3.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update python-twisted'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/python-twisted-10.1.0-3.fc14

Comment 29 Julian Sikorski 2010-11-14 13:20:29 UTC
Zooko, does the package from testing help?

Comment 30 Zooko O'Whielacronx 2010-11-14 20:26:24 UTC
The person who could test that would be Ruben Kerkhof who operates the Fedora buildslave for Tahoe-LAFS:

http://tahoe-lafs.org/buildbot/builders/Ruben%20Fedora

He could install the new twisted package and perform the simple manual test he write in the initial description:

https://bugzilla.redhat.com/show_bug.cgi?id=523210#c0

However, there's no reason to believe that this has been fixed, right? There is a new version of Twisted packaged, but nobody has changed it to include a .egg-info file like we need, have they?

Comment 31 Julian Sikorski 2010-11-14 20:31:51 UTC
Wrong. I pushed an updated package to testing and it could fix the problem.

Comment 32 Zooko O'Whielacronx 2010-11-15 05:54:52 UTC
(In reply to comment #31)
> Wrong. I pushed an updated package to testing and it could fix the problem.

Oh, great! I hope it works. Anyone who has Fedora can try it by executing the command-line written in the initial description:

https://bugzilla.redhat.com/show_bug.cgi?id=523210#c0

Comment 33 Julian Sikorski 2010-11-15 07:23:53 UTC
$ python -c 'import pkg_resources;print pkg_resources.require("Twisted>=2.4.0")'
[Twisted 10.1.0 (/usr/lib/python2.7/site-packages)]

This seems to work. The only question is whether it is ok that it points to python_sitelib, while twisted subpackages reside in python_sitearch.

Comment 34 Ruben Kerkhof 2010-11-15 08:55:27 UTC
> This seems to work. The only question is whether it is ok that it points to
> python_sitelib, while twisted subpackages reside in python_sitearch.

Yes, it works fine for me as well. thanks.
I don't think its a problem for it pointing to python_sitelib instead of pointing to python_sitearch, they're both in sys.path.

Zooko, you can see that the proper Twisted is selected at build-time now:
http://tahoe-lafs.org/buildbot/builders/Ruben%20Fedora/builds/697/steps/test-with-fake-dists/logs/stdio

Comment 35 Julian Sikorski 2010-11-15 09:37:05 UTC
Cool. If you guys have FAS accounts, please give karma to the update.

Comment 36 Fedora Update System 2010-11-17 23:26:49 UTC
python-twisted-10.1.0-3.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.