Spec URL: http://dmalcolm.fedorapeople.org/python-packaging/pypy.spec SRPM URL: http://dmalcolm.fedorapeople.org/python-packaging/pypy-1.2-1.fc14.src.rpm Description: pypy is a reimplementation of the Python language in Python itself, which is then compiled into implementations with particular properties. Successful scratch build in Koji: http://koji.fedoraproject.org/koji/taskinfo?taskID=2147291 Notes: Getting this to build was quite a challenge (i) A build takes at least an hour, and likely 3-5 hours, depending on how many variant runtimes we want to build (ii) It currently only builds on Fedora 13, not on Fedora 14 (rawhide). I get this failure when I try to build on f14: [platform:execute] gcc /tmp/usession-0/platcheck_27.o -pthread -lrt -o /tmp/usession-0/platcheck_27 [geninterp:WARNING] skipped classmethod: <bound method classobj.fromkeys of <class UserDict.UserDict at 0x7f47e2690ef0>> [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 259, in main [translation:ERROR] default_goal='compile') [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/translator/driver.py", line 763, in from_targetspec [translation:ERROR] spec = target(driver, args) [translation:ERROR] File "targetpypystandalone.py", line 223, in target [translation:ERROR] return self.get_entry_point(config) [translation:ERROR] File "targetpypystandalone.py", line 239, in get_entry_point [translation:ERROR] entry_point = create_entry_point(space, w_dict) [translation:ERROR] File "targetpypystandalone.py", line 31, in create_entry_point [translation:ERROR] w_os = setup_nanos(space) [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/translator/goal/nanos.py", line 50, in setup_nanos [translation:ERROR] w_os_path = space.wrap(app_os_path.buildmodule(space, 'path')) [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/interpreter/gateway.py", line 855, in buildmodule [translation:ERROR] return Module(space, space.wrap(name), self.getwdict(space)) [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/interpreter/gateway.py", line 851, in getwdict [translation:ERROR] return space.fromcache(ApplevelCache).getorbuild(self) [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/tool/cache.py", line 51, in getorbuild [translation:ERROR] result = self._build(key) [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/interpreter/baseobjspace.py", line 195, in _build [translation:ERROR] return self.build(key) [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/interpreter/gateway.py", line 910, in build [translation:ERROR] return PyPyCacheDir.build_applevelinterp_dict(app, self.space) [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/interpreter/gateway.py", line 984, in build_applevelinterp_dict [translation:ERROR] w_glob = initfunc(space) [translation:ERROR] File "<codegen /home/david/rpmbuild/BUILD/pypy-1.2/pypy/translator/geninterplevel.py:1540>", line 647, in init__builtin__ [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/objspace/std/objspace.py", line 671, in getattr [translation:ERROR] raiseattrerror(self, w_obj, name) [translation:ERROR] File "/home/david/rpmbuild/BUILD/pypy-1.2/pypy/objspace/descroperation.py", line 29, in raiseattrerror [translation:ERROR] typename, name) [translation:ERROR] OpErrFmt: [<W_TypeObject(AttributeError)>: 'module' object has no attribute 'getcwdu'] [translation] batch mode, not calling interactive helpers It appears that this is a python-2.6.4 -> python-2.6.5 change, I believe relating to a change within Python's os.path.abspath implementation on Posix: /home/david/rpmbuild/BUILD/pypy-1.2/pypy/_cache/nanos_d3cc75d994d0742917950b586d00aaa8.py has: gs_getcwdu = space.new_interned_str('getcwdu') gbltin_getcwdu = space.getattr(mod_posix, gs_getcwdu) and this is used by the implementation of abspath from /usr/lib64/python2.6/posixpath.py and this has indeed changed, and started using getcwdu between 2.6.4 and 2.6.5: http://svn.python.org/view/python/trunk/Lib/posixpath.py?r1=78037&r2=78247 (iii) Don't seem to be getting meaningful debuginfo packages. I tried to fix this by changing the tmpdir location it uses when autogenerating sources, but that didn't seem to help.
Issue (ii) above has been fixed in upstream SVN after the 1.2 tarball, as: https://codespeak.net/viewvc/?view=rev&revision=72073
Trying a scratch build into dist-f14, with r72073 patch cherrypicked: http://koji.fedoraproject.org/koji/taskinfo?taskID=2162634
The build successfully completed (took 4.5 hours) Updated specfile: http://dmalcolm.fedorapeople.org/python-packaging/pypy.spec Change since comment #0: http://dmalcolm.fedorapeople.org/python-packaging/pypy-from-1.2-1-to-1.2-2.diff Updated SRPM rpm: http://dmalcolm.fedorapeople.org/python-packaging/pypy-1.2-2.fc14.src.rpm
I'll look at reviewing this soon. ;)
OK - Package meets naming and packaging guidelines OK - Spec file matches base package name. OK - Spec has consistant macro usage. OK - Meets Packaging Guidelines. See below - License OK - License field in spec matches OK - License file included in package OK - Spec in American English OK - Spec is legible. See below - Sources match upstream md5sum: d423bd7dfbfbcfebb8435e55bcb36516 pypy-1.2-src.tar.bz2 d423bd7dfbfbcfebb8435e55bcb36516 pypy-1.2-src.tar.bz2.orig OK - BuildRequires correct See below - Package has %defattr and permissions on files is good. OK - Package has a correct %clean section. OK - Package has correct buildroot OK - Package is code or permissible content. OK - Packages %doc files don't affect runtime. OK - Package has rm -rf RPM_BUILD_ROOT at top of %install OK - Package compiles and builds on at least one arch. OK - Package has no duplicate files in %files. OK - Package doesn't own any directories other packages own. OK - Package owns all the directories it creates. OK - Package obey's FHS standard (except for 2 exceptions) See below - No rpmlint output. See below - final provides and requires are sane. SHOULD Items: OK - Should build in mock. OK - Should build on all supported archs OK - Should function as described. OK - Should have sane scriptlets. See below - Should have subpackages require base package with fully versioned depend. OK - Should have dist tag OK - Should package latest version OK - Should not use file requires outside of /etc, /bin, /sbin, /usr/bin, or /usr/sbin Issues: 1. Can you use " and " in the License tag between Licenses, not commas? (This is an and I think, not an or) 2. Can you fix the Source0 to use the full upstream url? http://pypy.org/download/pypy-1.2-src.tar.bz2 3. Should the Requires: pypy-libs be versioned? 4. I assume there's not any parallel make support here? Would sure be nice to speed up the build. 5. rpmlint says: a) Fixed by using 'and': pypy.src: W: invalid-license MIT, PSL, LGPL, Distributable pypy-debuginfo.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable pypy-libs.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable pypy-nojit.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable pypy-sandbox.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable pypy-stackless.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable b) Fixed by using full url: pypy.src: W: invalid-url Source0: pypy-1.2-src.tar.bz2 c) I think these are bogus, just looking at -libs name, but it's not really a libraries file. Perhaps it should be something like pypy-library ? Or pypy-lib-python? I guess it's not a big deal. pypy-stackless.x86_64: E: explicit-lib-dependency pypy-libs pypy-nojit.x86_64: E: explicit-lib-dependency pypy-libs pypy-sandbox.x86_64: E: explicit-lib-dependency pypy-libs d) I can see why this is, but not sure on a solution. Supress debuginfo? pypy-debuginfo.x86_64: E: debuginfo-without-sources e) What is this link for? pypy-libs.x86_64: W: dangling-relative-symlink /usr/share/pypy-1.2/pypy/lib/py ../../py f) Might be worth fixing the non executable py files to not have the python shebang? pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_bz2.py 0644L /usr/bin/python pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/test/test_optparse.py 0644L /usr/bin/python pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/plat-freebsd7/regen 0644L /bin/sh pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/test/test_sets.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecmaps_jp.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_tcl.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-sunos5/SUNAUDIODEV.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/IN.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/unittest.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/pypy/lib/md5.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_anydbm.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/difflib.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/AL.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/CD.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_urllib2net.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/FL.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/readcd.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-mac/Carbon/CoreGraphics.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/GET.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/cdplayer.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_with.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecencodings_jp.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/WAIT.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_random.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_multibytecodec.py 0644L /usr/bin/env pypy-libs.x86_64: W: devel-file-in-non-devel-package /usr/share/pypy-1.2/pypy/lib/app_test/ctypes_tests/_ctypes_test.c pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/pypy/lib/app_test/test_binascii.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_whichdb.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/test/test_aepack.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/FILE.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecencodings_hk.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-mac/Carbon/CG.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecmaps_tw.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-mac/Carbon/CarbonEvents.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_marshal.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/test/test_htmlparser.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/encodings/rot_13.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/lib-tk/Tix.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/tarfile.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/IOCTL.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/jpeg.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_urllibnet.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/plat-atheos/regen 0644L /bin/sh pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/CL.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/GLWS.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/idlelib/idle.bat pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/timeit.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecmaps_hk.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecencodings_kr.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecmaps_kr.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/trace.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/torgb.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/trace.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/test/test_socket.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-sunos5/IN.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_multibytecodec_support.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/plat-os2emx/regen 0644L /bin/sh pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/SV.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/DEVICE.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-mac/Carbon/CarbonEvt.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/idlelib/PyShell.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/panelparser.py pypy-libs.x86_64: W: spurious-executable-perm /usr/share/doc/pypy-libs-1.2/demo/bpnn.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/test/test_random.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/readcd.doc pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/Cookie.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/encodings/rot_13.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/test/test_bz2.py 0644L /usr/bin/python pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/test/test_marshal.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_socket.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/test/test_wsgiref.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecencodings_cn.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_optparse.py 0644L /usr/bin/python pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_popen.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/bsddb/dbshelve.py 0644L /bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/test/test_dumbdbm.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/cddb.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/CL_old.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/webbrowser.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_sets.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_eof.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test___future__.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_urllib2_localnet.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/modified-2.5.2/test/test_urllib2_localnet.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/flp.py pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/test/test_binascii.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/plat-freebsd4/regen 0644L /bin/sh pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/GL.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/plat-freebsd6/regen 0644L /bin/sh pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_urlparse.py 0644L /usr/bin/env pypy-libs.x86_64: E: wrong-script-interpreter /usr/share/pypy-1.2/lib-python/2.5.2/test/test_largefile.py python pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_largefile.py 0644L python pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/pypy/lib/test2/pickledtasklet.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/crashers/recursive_call.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/bsddb/test/test_dbtables.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_popen2.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/panel.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecencodings_tw.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/flp.doc pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_logging.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/test/test_grp.py pypy-libs.x86_64: E: wrong-script-interpreter /usr/share/pypy-1.2/lib-python/2.5.2/test/test_pep263.py -*- pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_pep263.py 0644L -*- pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/plat-irix5/ERRNO.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_dumbdbm.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_gzip.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/test/test_codecmaps_cn.py 0644L /usr/bin/env pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/pypy/lib/sha.py 0644L /usr/bin/env pypy-libs.x86_64: E: script-without-shebang /usr/share/pypy-1.2/lib-python/2.5.2/runpy.py pypy-libs.x86_64: E: non-executable-script /usr/share/pypy-1.2/lib-python/2.5.2/plat-freebsd5/regen 0644L /bin/sh Can be ignored: pypy.src: W: spelling-error %description -l en_US reimplementation -> re implementation, re-implementation, implementation pypy-nojit.x86_64: W: no-documentation pypy-nojit.x86_64: W: no-manual-page-for-binary pypy-nojit pypy-sandbox.x86_64: W: spelling-error Summary(en_US) Sandboxed -> Sand boxed, Sand-boxed, Sandboxes pypy-sandbox.x86_64: W: no-documentation pypy-sandbox.x86_64: W: no-manual-page-for-binary pypy-sandbox pypy-stackless.x86_64: W: no-documentation pypy-stackless.x86_64: W: no-manual-page-for-binary pypy-stackless
(In reply to comment #5) > c) I think these are bogus, just looking at -libs name, but it's not really a > libraries > file. Perhaps it should be something like pypy-library ? Or pypy-lib-python? > I guess it's not a big deal. > > pypy-stackless.x86_64: E: explicit-lib-dependency pypy-libs > pypy-nojit.x86_64: E: explicit-lib-dependency pypy-libs > pypy-sandbox.x86_64: E: explicit-lib-dependency pypy-libs > I suggest to use -common as subpackage name which means this subpackage is shared between different pypy interpreters.
(In reply to comment #5) > a) Fixed by using 'and': > > pypy.src: W: invalid-license MIT, PSL, LGPL, Distributable > pypy-debuginfo.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable > pypy-libs.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable > pypy-nojit.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable > pypy-sandbox.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable > pypy-stackless.x86_64: W: invalid-license MIT, PSL, LGPL, Distributable > A few things to note here: Distributable is not a valid license tag. You'll need to figure out what it really is. If it's some piece of code that's not modifiable, it might be that it doesn't fit the OSS Guidelines :-( If it's firmware, then it's okay but you still need to figure out the proper license. Details here: http://fedoraproject.org/wiki/Licensing And you can ask spot if you're in a grey area. As for listing of licenses... if this is an and situation, you might need to ask spot how things need to be licensed. LGPL tends to trump all the other licenses but since this package has so many parts, you might need to talk to spot to determine the actual situation.
Dave: Any news here? (No hurry, just wondering... ;)
Sorry, somewhat doomed right now with other tasks. See http://lists.fedoraproject.org/pipermail/python-devel/2010-July/000269.html I hope to address this review (and bump to 1.3) soon, but I'm fairly sure I won't get a chance to look at it for at least a week, probably two.
Any news on this?
I don't really want to be primary maintainer but I'd be happy to comaintain once this is reviewed and in the distro and I can also work on addressing at least some of the review blockers. So I've got an updated package here: http://toshio.fedorapeople.org/packages/pypy-1.4-2.fc13.src.rpm http://toshio.fedorapeople.org/packages/pypy.spec Working its way through the build system here: http://koji.fedoraproject.org/koji/taskinfo?taskID=2629927 If it fails to build in the buildsystem, I'll take another crack at it this weekend. Things fixed: 1) License tag is better. dmalcolm: I found the list of licenses that had been included in the spec file present in the LICENSE file. If that's where you got the original list from then the license tag should now be correct. I was able to remove LGPL and Distributable because those were coming from bundled jar files. I removed the bundled jars (see below) so that should now be fine. 2) The Source0 is now a URL to upstream. 3) pypy-libs Req now versioned 4) The build isn't make driven but I didn't look at parallelizing it. 5) rpmlint c): Yeah, it's because it's named -libs. Another possible name is -stdlib since it is the python stdlib(for pypy). 5e) I was waiting for the koji build to complete to see if this is still an issue with 1.4. 5f) dmalcolm, what did we decide to do with this on the python and python3 packages? Other: * Removed bundled jar files. Pre-built files are not allowed. It looks like these were present in case we were building a pypy that targets the jvm (rather than POSIX-C). Simple removal should be fine until/unless we want to enable that backend. If we do enable that, one of the jar files is packaged for Fedora (jna) but the other (jasmin) is not. * Build pypy with itself after the first build as upstream recommends this now (and it's supposed to be faster than using cpython) * Build the jit'ed version on both x86 and x86_64 (1.4 feature) * It looked like we weren't picking up the CFLAGS so I added a patch to pick up the CFLAGS environment variable * In the same patch, the code that configures for linux was mandating a static libffi. I changed that so that the shared libffi should be picked up.
Hello. I'm not sure if this is the correct place, so please write me a mail if not and where should I put it. In pypy checkout there is a file in pypy/tool/release called package.py. This one is used by us to package raw binary for the release. It's probably not directly useful, but contains information which files are actually necessary for a binary to be installed and work. I would be more than happy to improve this script to it'll also work for you (say by printing a list of files).
Many items look fixed up, thanks Toshio. :) However, the scratch build seems to have failed. ;( I'll take a look once you get it building?
Yeah, sounds good. dmalcolm or maciej, the build.log output is less than enlightening (huge log file): http://koji.fedoraproject.org/koji/getfile?taskID=2630788&name=build.log It seems that make is returning an errorcode on exit there but there isn't any output that shows what the error might be. I ran that portion of the build locally (f13, x86_64) and it appeared to build fine; pypy was present and those ERROR lines were WARNING lines instead. However, translate did give an exit code of 1. Not sure how to tell what's going wrong....
New build started with the portion that would use our CFLAGS disabled to test whether the CFLAGS have an effect on this: http://koji.fedoraproject.org/koji/taskinfo?taskID=2631079
Okay, that build didn't finish but it got past the build phase and into the install. So it does look like this is some interaction with our CFLAGS. It's too bad the build takes so long, otherwise it would be easier to debug this via trial and error. I'm going to try to get a build to finish and then we can try to debug which CFLAG option is causing the problems.
Any reason for passing --translation-verbose? That kind of explains why the build logs are so huge (we don't use it for every day for example). Can't you test your build so pypy-c is already build and then you package it?
Hello. There was extra info just added that you might find useful: http://pypy.org/download.html#building-from-source In particular, you can play around with make once you get build directory. You're passing a lot of custom flags to CFLAGS, most of them doesn't make sense in terms of PyPy (like -fstack-protector, because we have our own stack-depth checking and we generate assembler anyway), but they also confuse trackgcroot. In short, we postprocess assembler after gcc to find out where are gc roots. This does not cooperate nicely with passing custom options to gcc.
Maciej, thanks! I don't know why we using --translation-verbose, I'm guessing that dmalcolm was using that to get a feel for how the translation process works. dmalcolm, I'm going to turn that off for now. Feel free to turn it back on if there's a reason you want that. Your tips will aid in figuring out what to do with CFLAGS. Very useful! Do you happen to know if some of our CFLAGS are safe and useful to use with pypy? Do you know which CFLAGS, like -fstack-protector are not needed with the checking that pypy does itself? If you don't know I'll experiment a bit to see what we can come up with. About the files that get installed on the filesystem, I cheated and just looked at what ended up in the Linux binary package that is downloaded from pypy.org. A few questions: * I see that you have some include files in the binary distribution but no compiled shared library. What are the header files for? Should we be packaging them? * My understanding is that pypy can support compiled modules as well as pure python modules. If that's the case, where does pypy search for them? We're presently installing all of the pypy stdlib into /usr/share/pypy-1.4/{lib-python,pypy_lib} which dmalcolm says is being searched by pypy. But if there's compiled modules, at least those modules need to land in /usr/lib/pypy-1.4 and /usr/lib64/pypy-1.4 (on 64bit multilib arches like x86_64). Does pypy search these places by default similar to cpython does or will we need to do some work on that? With cpython we have a split site-packages such that /usr/lib/python2.5/site-packages is where we install arch independent files (pure python modules and their byte compiled cache) and (on x86_64), /usr/lib64/python2.5/site-packages is where we install the modules with compiled code.
(In reply to comment #19) Hi, will write down responses bit by bit :) > Maciej, thanks! > > Your tips will aid in figuring out what to do with CFLAGS. Very useful! Do > you happen to know if some of our CFLAGS are safe and useful to use with pypy? > Do you know which CFLAGS, like -fstack-protector are not needed with the > checking that pypy does itself? If you don't know I'll experiment a bit to see > what we can come up with. Essentially all -f flags that make gcc output different assembler (except those which are in Makefile by default like -fomit-frame-pointer) are likely to cause trouble. Those are two categories: * performance - likely pointless because we're generating our own assembler for hot places * security (like -fstack-protector) which I think are also pointless, because we're generating fairly straightforward C, which has additional checks for such things like stack depth. > > About the files that get installed on the filesystem, I cheated and just looked > at what ended up in the Linux binary package that is downloaded from pypy.org. > A few questions: > > * I see that you have some include files in the binary distribution but no > compiled shared library. What are the header files for? Should we be > packaging them? Those are for building your own C extensions with PyPy. They're somehow required for this to work, but the development status of this feature is alpha at best (I have a bit no clue how they'll work installed in some place). > * My understanding is that pypy can support compiled modules as well as pure > python modules. If that's the case, where does pypy search for them? We're > presently installing all of the pypy stdlib into > /usr/share/pypy-1.4/{lib-python,pypy_lib} which dmalcolm says is being > searched by pypy. But if there's compiled modules, at least those modules need > to land in /usr/lib/pypy-1.4 and /usr/lib64/pypy-1.4 (on 64bit multilib arches > like x86_64). Does pypy search these places by default similar to cpython does > or will we need to do some work on that? With cpython we have a split > site-packages such that /usr/lib/python2.5/site-packages is where we install > arch independent files (pure python modules and their byte compiled cache) and > (on x86_64), /usr/lib64/python2.5/site-packages is where we install the modules > with compiled code. PyPy has a very similar scheme where it looks for modules as CPython. Main difference is lib_pypy/lib-python, but besides it's almost identical. Note that PyPy implements PEP3149 which means that files will have unique names among versions (containing string pypy and 14), so there is no clash with other versions. That said, since this feature is anyway alpha, I find it very unlikely that someone will get a working module and install it somewhere, so for 1.4 it's a bit irrelevant. Cheers, fijal
Okay, finally, a build that completes: http://koji.fedoraproject.org/koji/taskinfo?taskID=2635254 The CFLAGS were *very* finicky. I removed these two flags last: -e 's/-march=i686//' -e 's/-mtune=atom//' which seems to be a bit of overkill (and has no effect on x86_64 where mtune and march are different) I'll try to build this again locally tonight so that tomorrow I have a local copy to play around with. Issues that I'll personally keep working on: * FHS compliance: Need to either move the %{_datadir}/pypy-1.4 directory to %{_libdir} or test that pypy finds modules in both %{_libdir}/pypy-1.4 and %{_datadir}/pypy-1.4. * Include files: The generated files are supposed to end up in the toplevel include directory after translation is done. I'll try to do an install of that directory. fijal: Would it be okay to place those in /usr/include/pypy-1.4/ ? Or /usr/include/pypy/ ? Or some other scheme? * CFLAGS: I've got to figure out precisely which CFLAGS are not okay and then exclude just those. Hopefully this will be easy to knock out tomorrow after I have a local build.
(In reply to comment #21) > Okay, finally, a build that completes: > http://koji.fedoraproject.org/koji/taskinfo?taskID=2635254 Another issue, for when I get time to work on this: The debuginfo package still doesn't seem to contain the source code, just the DWARF data (and buildid links): $ rpm -qplv /home/david/Download/pypy-debuginfo-1.4-3.fc13.x86_64.rpm drwxr-xr-x 2 root root 0 Nov 30 16:07 /usr/lib/debug drwxr-xr-x 2 root root 0 Nov 30 16:07 /usr/lib/debug/.build-id drwxr-xr-x 2 root root 0 Nov 30 16:07 /usr/lib/debug/.build-id/43 lrwxrwxrwx 1 root root 30 Nov 30 16:07 /usr/lib/debug/.build-id/43/18ae1067c663e6ed2c167ed03eed84590f6b1f -> ../../../../bin/pypy-stackless lrwxrwxrwx 1 root root 34 Nov 30 16:07 /usr/lib/debug/.build-id/43/18ae1067c663e6ed2c167ed03eed84590f6b1f.debug -> ../../usr/bin/pypy-stackless.debug drwxr-xr-x 2 root root 0 Nov 30 16:07 /usr/lib/debug/.build-id/50 lrwxrwxrwx 1 root root 26 Nov 30 16:07 /usr/lib/debug/.build-id/50/007c175ce50db3eb673f6ff5064f6b11558cdf -> ../../../../bin/pypy-nojit lrwxrwxrwx 1 root root 30 Nov 30 16:07 /usr/lib/debug/.build-id/50/007c175ce50db3eb673f6ff5064f6b11558cdf.debug -> ../../usr/bin/pypy-nojit.debug drwxr-xr-x 2 root root 0 Nov 30 16:07 /usr/lib/debug/.build-id/74 lrwxrwxrwx 1 root root 24 Nov 30 16:07 /usr/lib/debug/.build-id/74/39b07100914582035c53928cefb1301fa2e4d2 -> ../../../../bin/pypy-jit lrwxrwxrwx 1 root root 28 Nov 30 16:07 /usr/lib/debug/.build-id/74/39b07100914582035c53928cefb1301fa2e4d2.debug -> ../../usr/bin/pypy-jit.debug drwxr-xr-x 2 root root 0 Nov 30 16:07 /usr/lib/debug/.build-id/76 lrwxrwxrwx 1 root root 28 Nov 30 16:07 /usr/lib/debug/.build-id/76/f76b0a9dfd9b485fbe3cdcf52717bc752ab9d0 -> ../../../../bin/pypy-sandbox lrwxrwxrwx 1 root root 32 Nov 30 16:07 /usr/lib/debug/.build-id/76/f76b0a9dfd9b485fbe3cdcf52717bc752ab9d0.debug -> ../../usr/bin/pypy-sandbox.debug drwxr-xr-x 2 root root 0 Nov 30 16:07 /usr/lib/debug/usr drwxr-xr-x 2 root root 0 Nov 30 16:07 /usr/lib/debug/usr/bin -r--r--r-- 1 root root 14662480 Nov 30 16:07 /usr/lib/debug/usr/bin/pypy-jit.debug -r--r--r-- 1 root root 6361288 Nov 30 16:07 /usr/lib/debug/usr/bin/pypy-nojit.debug -r--r--r-- 1 root root 3860608 Nov 30 16:07 /usr/lib/debug/usr/bin/pypy-sandbox.debug -r--r--r-- 1 root root 9068224 Nov 30 16:07 /usr/lib/debug/usr/bin/pypy-stackless.debug
I've been working on improving the readability of the generated .c code; initial attempt at a patch sent upstream as: http://codespeak.net/pipermail/pypy-dev/2010q4/006532.html
(In reply to comment #21) > Okay, finally, a build that completes: > http://koji.fedoraproject.org/koji/taskinfo?taskID=2635254 Toshio: thanks for working on this. Do you have the specfile/src.rpm for this somewhere? It seems to have been garbage-collected by Koji, alas, and the latest version up on: http://toshio.fedorapeople.org/packages/ appears to not contain all of the changes that you refer to in the comments above. (I tried using the latest pypy.spec from there as a base, but it's missing a bunch of the changes discussed above).
Sorry, I've been away on vacation. Latest src.rpm and spec file are now uploaded to http://toshio.fedorapeople.org/packages/ I'll be back to work tomorrow if you want to throw more questions at me on IRC.
Toshio, just a quick note: I ran a local mock build of the SRPM you uploaded, and ended up with this result (-jit, -nojit, and -stackless all behave similarly): $ pypy-jit debug: WARNING: library path not found, using compiled-in sys.path 'import site' failed Python 2.5.2 (78826, Dec 15 2010, 01:41:24) [PyPy 1.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. debug: OperationError: debug: operror-type: ImportError debug: operror-value: No module named _pypy_interact $ pypy-jit -c 'import sys; print sys.path' debug: WARNING: library path not found, using compiled-in sys.path 'import site' failed ['', '/builddir/build/BUILD/pypy-1.4/lib_pypy', '/builddir/build/BUILD/pypy-1.4/lib-python/modified-2.5.2', '/builddir/build/BUILD/pypy-1.4/lib-python/2.5.2', '/builddir/build/BUILD/pypy-1.4/lib-python/2.5.2/plat-linux2']
Yeah, that's something I wanted to look into along with FHS compliance but haven't had time. dmalcolm has this note in the spec file: # Install the various support libraries as described at: # http://codespeak.net/pypy/dist/pypy/doc/getting-started-python.html#installation # which refers to a "PREFIX" found relative to the location of the binary. # Given that the pypy binaries will be in /usr/bin, PREFIX can be # "../share/pypy-1.2" relative to that directory, i.e. /usr/share/pypy-1.2 # # Running "strace" on a built binary indicates that it searches within # PREFIX/lib-python/modified-2.5.2 # not # PREFIX/lib-python/modified.2.5.2 # as given on the above page, i.e. it uses '-' not '.' But, at least with pypy-1.4, that doesn't work. pypy thinks the PREFIX is /usr/bin/ when it would need to be /usr/share (or %{_libdir}/pypy-%{version} in our case.) You can either cd to /usr/share (or /usr/share/pypy-1.4... I'm not sure which atm) or you can add /usr/share/pypy-1.4 to the PYTHONPATH in order to get pypy to work. Really, though, we need to patch pypy to recognize the proper directory.
Hello. I'll look into prefix thing at some point, it should work. Regarding various builds - we decided that "pypy" is the one with jit and "pypy-stackless" is one with stackless no jit. We don't ship nojit binaries at all (and I think this helps to avoid confusion). Cheers, fijal
(In reply to comment #25) > Sorry, I've been away on vacation. Latest src.rpm and spec file are now > uploaded to http://toshio.fedorapeople.org/packages/ I'll be back to work > tomorrow if you want to throw more questions at me on IRC. Thanks Toshio, Maciej, and everyone else who's been helping with this package. I've been working from Toshio's latest src.rpm and hope to post an updated package for review later today (the test builds take a while, alas... I've restricted things to just the JIT build for now)
Here's what I've got so far: Updated specfile: http://fedorapeople.org/~dmalcolm/python-packaging/pypy.spec Updated SRPM: http://fedorapeople.org/~dmalcolm/python-packaging/pypy-1.4-4.fc13.src.rpm Difference between Toshio's 1.4-3 and my 1.4-4 can be seen here: http://fedorapeople.org/~dmalcolm/python-packaging/pypy.spec-from-1.4-3-to-1.4-4.diff Scratch build currently underway here: http://koji.fedoraproject.org/koji/taskinfo?taskID=2670922 My last local build had only two rpmlint warnings, and I've (hopefully) fixed them in this version. Changes in 1.4-4 (from %changelog) - rename the jit build and subpackge to just "pypy", and remove the nojit and sandbox builds, as upstream now seems to be focussing on the JIT build (with only stackless called out in the getting-started-python docs); disable stackless for now - add a verbose_logs specfile boolean; leave it enabled for now (whilst fixing build issues) - add more comments, and update others to reflect 1.2 -> 1.4 changes - re-enable debuginfo within CFLAGS ("-g") - add the LICENSE and README to all subpackages - ensure the built binaries don't have the "I need an executable stack" flag - remove DOS batch files during %%prep (idlelib.bat) - remove shebang lines from .py files that aren't executable, and remove executability from .py files that don't have a shebang line (taken from our python3.spec) - bytecompile the .py files into .pyc files in pypy's bytecode format
The scratch build ran to completion. rpmlint on the resulting rpms (and src.rpm) shows a single warning: $ rpmlint *.rpm |sort 7 packages and 0 specfiles checked; 0 errors, 2 warnings. pypy-libs.i686: W: spurious-executable-perm /usr/share/doc/pypy-libs-1.4/demo/bpnn.py pypy-libs.x86_64: W: spurious-executable-perm /usr/share/doc/pypy-libs-1.4/demo/bpnn.py Looks like it's complaining about having an executable under /usr/share/doc $ rpm -qplv pypy-libs-1.4-4.fc15.i686.rpm | grep bpnn -rwxr-xr-x 1 root root 6001 Dec 16 18:32 /usr/share/doc/pypy-libs-1.4/demo/bpnn.py
(In reply to comment #11) ... > 1) License tag is better. dmalcolm: I found the list of licenses that had > been included in the spec file present in the LICENSE file. If that's where > you got the original list from then the license tag should now be correct. I > was able to remove LGPL and Distributable because those were coming from > bundled jar files. I removed the bundled jars (see below) so that should now > be fine. The LICENSE file is indeed where I got that list from. I haven't yet gone through the full tarball payload. > 4) The build isn't make driven but I didn't look at parallelizing it. The compilation phase invokes a Makefile; on my machine it used "-j 4", and in one of the Koji scratch builds it used "-j 10" It appears to come from this code in pypy-1.4/pypy/translator/c/genc.py:compile: > if self.config.translation.make_jobs != 1: > extra_opts += ['-j', str(self.config.translation.make_jobs)] which in turn is intialized in this code in pypy-1.4/pypy/config/translationoption.py: cmdline="--make-jobs", default=detect_number_of_processors()), Hopefully that's good enough for our build system. It's actually visible in the built binary, FWIW: > $ ./pypy --info|grep make_jobs > translation.make_jobs: 4 Unfortunately that parallelizable phase is only a small fraction of the build time (see "compile_c" below); from the x86_64 scratch build's log: [Timer] Timings: [Timer] annotate --- 780.8 s [Timer] rtype_lltype --- 491.7 s [Timer] pyjitpl_lltype --- 712.2 s [Timer] backendopt_lltype --- 202.3 s [Timer] stackcheckinsertion_lltype --- 21.6 s [Timer] database_c --- 243.8 s [Timer] source_c --- 477.3 s [Timer] compile_c --- 148.4 s [Timer] =========================================== [Timer] Total: --- 3078.2 s real 52m21.339s user 49m43.905s sys 0m3.552s so roughly 15% of that build time benefited from the 10 cores on that machine (presumably, would have been ~1480 seconds otherwise, so that at least is a significant saving). As I understand it, parallelizing the rest of the build would involve major rewrites (it's all in one big cpython process) > 5) rpmlint c): Yeah, it's because it's named -libs. Another possible name is > -stdlib since it is the python stdlib(for pypy). Did moving to versioned requirements fix this? > 5e) I was waiting for the koji build to complete to see if this is still an > issue with 1.4. Seems to have been fixed, either in 1.4, or in Toshio's work on the specfile. > 5f) dmalcolm, what did we decide to do with this on the python and python3 > packages? I've copied a fixer-upper from the python3 specfile in my 1.4-4 pypy.spec > Other: > * Removed bundled jar files. Pre-built files are not allowed. It looks like > these were present in case we were building a pypy that targets the jvm (rather > than POSIX-C). Simple removal should be fine until/unless we want to enable > that backend. If we do enable that, one of the jar files is packaged for BTW, I considered building that one. The impression I get is that the JVM backend isn't as mature as the .c backend, so it seems simplest to omit it for now. (fijal: do you have any advice/guidance on this?)
JVM backend is not mature enough to be shipped. Btw: how did you end up having those files in the first place? Only things that are needed for a binary are ones that are packaged using package.py, which includes stdlib and .h files (and that's about it). Do you put all the source code with the package? If so, why? Regarding time - it was said here before I think, but using pypy to build pypy is already saving half of the time. That said, what I did when building ubuntu packages was simply replace the translate.py option with copying of already built pypy-c, this way it was much more convinient (do you also build KDE from scratch each time you want to test something on a package building process?) Cheers, fijal
(In reply to comment #33) > JVM backend is not mature enough to be shipped. Thanks; we'll leave it out, then. > Btw: how did you end up having > those files in the first place? Only things that are needed for a binary are > ones that are packaged using package.py, which includes stdlib and .h files > (and that's about it). Do you put all the source code with the package? If so, > why? Sorry, it's not clear to me which files you're referring to in the above. The specfile is still using the scripted code from before; I haven't yet looked at the package.py you referred to. I plan to address this later today (note to self: comment #12 above). > Regarding time - it was said here before I think, but using pypy to build pypy > is already saving half of the time. That said, what I did when building ubuntu The specfile is written with the potential to build other configurations (stackless currently), and it use the ./pypy (i.e. the JIT-enabled binary) if it's already been built within this build. > packages was simply replace the translate.py option with copying of already > built pypy-c, this way it was much more convinient (do you also build KDE from > scratch each time you want to test something on a package building process?) I'm not sure I understand the analogy here. Source RPMs are the granularity we work at when doing rebuilds. So we don't rebuild GCC each time, if that makes sense (and eventually you get into "Reflections on Trusting Trust" territory). If you're referring to using a pypy-c supplied as part of your tarball: we avoid using binaries supplied by upstream as a general policy across all of Fedora: - we want to ensure that we always can regenerate the full OS from source, using a fully open-source toolchain, and using binaries from an upstream tarball make it harder to assert that property of our OS. - avoiding prebuilt binaries reduces the knock-on effect of an upstream repository getting Trojanned More information on this policy is here (if bz doesn't mangle the link): http://fedoraproject.org/wiki/PackagingGuidelines#No_inclusion_of_pre-built_binaries_or_libraries There are exceptions allowed for compilers and cross-compilers, but as I understand it, they are intended for when someone is bootstrapping the buildsystem on a new architecture. My gut feeling is that the speedup, whilst desirable, doesn't outweigh the benefits of knowing that every binary was built in the same highly locked-down environment (e.g. we have a fresh chroot for every build, which we can reconstruct if we need to go back to investigate something). Another way to do this is: once a pypy package is in Fedora, it could have a build-time requirement on itself (i.e. build a new pypy package using the old one each time). Loops in the build-time dependency graph make me nervous; we try to minimize them, since it makes doing a full rebuild of source of all of Fedora more difficult (e.g. when introducing a new version of GCC). Hope all of the above makes sense (it's too early in the morning here)
(In reply to comment #34) > (In reply to comment #33) > > JVM backend is not mature enough to be shipped. > Thanks; we'll leave it out, then. > > > Btw: how did you end up having > > those files in the first place? Only things that are needed for a binary are > > ones that are packaged using package.py, which includes stdlib and .h files > > (and that's about it). Do you put all the source code with the package? If so, > > why? > Sorry, it's not clear to me which files you're referring to in the above. > > The specfile is still using the scripted code from before; I haven't yet looked > at the package.py you referred to. I plan to address this later today (note to > self: comment #12 above). > I'm referring to prebuilt jars. They should not be there in the binary distribution at all, because the source code is not included (or a directory where they live in). How did you get a list of files that has to be included? > > > Regarding time - it was said here before I think, but using pypy to build pypy > > is already saving half of the time. That said, what I did when building ubuntu > The specfile is written with the potential to build other configurations > (stackless currently), and it use the ./pypy (i.e. the JIT-enabled binary) if > it's already been built within this build. > > > > packages was simply replace the translate.py option with copying of already > > built pypy-c, this way it was much more convinient (do you also build KDE from > > scratch each time you want to test something on a package building process?) > I'm not sure I understand the analogy here. Source RPMs are the granularity we > work at when doing rebuilds. So we don't rebuild GCC each time, if that makes > sense (and eventually you get into "Reflections on Trusting Trust" territory). > > If you're referring to using a pypy-c supplied as part of your tarball: we > avoid using binaries supplied by upstream as a general policy across all of > Fedora: > - we want to ensure that we always can regenerate the full OS from source, > using a fully open-source toolchain, and using binaries from an upstream > tarball make it harder to assert that property of our OS. > - avoiding prebuilt binaries reduces the knock-on effect of an upstream > repository getting Trojanned > More information on this policy is here (if bz doesn't mangle the link): > http://fedoraproject.org/wiki/PackagingGuidelines#No_inclusion_of_pre-built_binaries_or_libraries > > There are exceptions allowed for compilers and cross-compilers, but as I > understand it, they are intended for when someone is bootstrapping the > buildsystem on a new architecture. My gut feeling is that the speedup, whilst > desirable, doesn't outweigh the benefits of knowing that every binary was built > in the same highly locked-down environment (e.g. we have a fresh chroot for > every build, which we can reconstruct if we need to go back to investigate > something). > > Another way to do this is: once a pypy package is in Fedora, it could have a > build-time requirement on itself (i.e. build a new pypy package using the old > one each time). Loops in the build-time dependency graph make me nervous; we > try to minimize them, since it makes doing a full rebuild of source of all of > Fedora more difficult (e.g. when introducing a new version of GCC). > > Hope all of the above makes sense (it's too early in the morning here) Heh, hope your coffee tastes good :-) I'm not referring to the build process for a distribution itself (that should be fairly constrained, I admit) and while pypy would be cool to have, it's not more than "suggested" for a pypy build. What's I'm referring to is that when you work on a build system you might want to avoid the whole translation process at all, by say copying already built pypy binary and avoiding burden associated with creating it each time. you would still need to build one for the distribution though (but the length of build process is not as painful). That's just a suggestion and I guess I'm not getting the whole complexity of things associated anyway. Cheers, fijal
(In reply to comment #35) > (In reply to comment #34) > > (In reply to comment #33) > > > JVM backend is not mature enough to be shipped. > > Thanks; we'll leave it out, then. > > > > > Btw: how did you end up having > > > those files in the first place? Only things that are needed for a binary are > > > ones that are packaged using package.py, which includes stdlib and .h files > > > (and that's about it). Do you put all the source code with the package? If so, > > > why? > > Sorry, it's not clear to me which files you're referring to in the above. > > > > The specfile is still using the scripted code from before; I haven't yet looked > > at the package.py you referred to. I plan to address this later today (note to > > self: comment #12 above). > > > > I'm referring to prebuilt jars. They should not be there in the binary > distribution at all, because the source code is not included (or a directory > where they live in). How did you get a list of files that has to be included? FWIW, those jar files are present within SVN/HG; looking here: http://codespeak.net/pypy/dist/pypy/translator/jvm/src/ I see them, and they seem to be in the source tarballs you're building: $ tar -tjvf pypy-1.4-src.tar.bz2 | grep jar -rw-r--r-- fijal/fijal 207224 2007-12-06 06:37 pypy-1.4/pypy/translator/jvm/src/jna.jar -rw-r--r-- fijal/fijal 128766 2009-04-18 11:54 pypy-1.4/pypy/translator/jvm/src/jasmin.jar $ md5sum pypy-1.4-src.tar.bz2 6c7e5a3fab4b3f6357aab84927420b49 pypy-1.4-src.tar.bz2 (which matches that on http://pypy.org/download.html )
Sure they are. However, they're not needed (or even advised) for the binary build. If you're going to remove them for whatever reason (like policy), please also remove translator/jvm as it makes no longer sense. What's necessary however for a binary to run is listed by pypy/tool/release/package.py in source. If this file seems confusing, it's essentially what is present in our nightly runs (with rules how it got created listed in package.py, I can elaborate if needed). PyPy contains lots of stuff and not all of it is necessary for a C/POSIX build (but most of it is). If I may ask, is it possible to run the test suite after the build to check if stuff actually worked (If so, I can provide more information how to actually run this)? Cheers, fijal
(In reply to comment #37) > Sure they are. However, they're not needed (or even advised) for the binary > build. If you're going to remove them for whatever reason (like policy), please > also remove translator/jvm as it makes no longer sense. Sure, if you like, though that goes beyond the requirements of our "no bundled binaries" policy. A somewhat-related question about the pypy sources: it looks like the pypy module hierarchy is useful outside of building pypy itself. If so, is it reasonable to make the pypy module be available to the system CPython, by installing it to, say, /usr/lib/python2.7/site-packages/pypy ? (and shipping that module hierarchy within, say, a "python-pypy" subpackage?) That way you'd be able to "import pypy" from the system CPython, and e.g. invoke the translator on other python code (provided that code was sufficiently RPythonic, as I understand it). If we did that, it only seems fair to also install it below pypy's site-packages, and ship that within a "pypy-pypy" subpackage :) (my hope is to eventually build out a parallel-installable family of pypy-* rpms, mirroring our python-* rpms) > What's necessary however for a binary to run is listed by > pypy/tool/release/package.py in source. If this file seems confusing, it's > essentially what is present in our nightly runs (with rules how it got created > listed in package.py, I can elaborate if needed). Thanks. I looked briefly through this script: it looks like it's what you use to make the tarballs. > PyPy contains lots of stuff and not all of it is necessary for a C/POSIX build > (but most of it is). If I may ask, is it possible to run the test suite after > the build to check if stuff actually worked (If so, I can provide more > information how to actually run this)? I'd actually started doing this, and ran into some problems, so more information would be v. useful. Specifically, I tried putting the built binary through the Python test suite using: ./pypy -m test.regrtest from the build's "translator/goal" directory. I saw a hang in test_asynchat, so I switched to: ./pypy -m test.regrtest -x test_asynchat Various minor failures scroll by, but then I started running into intermittent segfaults, towards the "s" part of the alphabet (e.g. test_scope, test_shelve, test_sqlite), which go away when running the tests by themselves. This makes me think that there's some kind of state buildup over the course of running the full test suite. Unfortunately I wasn't able to get meaningful information out of gdb for these crashes. I tried without JIT, by using --jit threshold=-1 but this did not fix these segfaults. I then tried running the whole thing under valgrind, seeing lots of interesting warnings, so I switched to just launching the interactive interpreter under valgrind: $ valgrind --track-origins=yes --leak-check=full ./pypy --jit threshold=-1 and I see various warnings; I'll attach them to bugzilla (this comment is getting far too long as is). There's also the test suite for pypy's modules, is that "just" a case of running: python test_all.py from the pypy subdirectory? Thanks!
Created attachment 469463 [details] Result of running: $ valgrind --track-origins=yes --leak-check=full ./pypy --jit threshold=-1 -c "pass" 2> valgrind-of-pypy-pass.txt
(In reply to comment #38) > (In reply to comment #37) > > Sure they are. However, they're not needed (or even advised) for the binary > > build. If you're going to remove them for whatever reason (like policy), please > > also remove translator/jvm as it makes no longer sense. > Sure, if you like, though that goes beyond the requirements of our "no bundled > binaries" policy. well, it goes into "don't ship stuff in a way that does not work" policy :) > > A somewhat-related question about the pypy sources: it looks like the pypy > module hierarchy is useful outside of building pypy itself. If so, is it > reasonable to make the pypy module be available to the system CPython, by > installing it to, say, /usr/lib/python2.7/site-packages/pypy ? (and shipping > that module hierarchy within, say, a "python-pypy" subpackage?) I don't think it's reasonable. People tinkering with pypy are very likely required to have a very recent HG checkout anyway and everything works from the checkout directory (as a source code, it doesn't require to be installed). There are few tools that would use pypy as a module, but they're all fairly internal (like JIT trace viewer). > > That way you'd be able to "import pypy" from the system CPython, and e.g. > invoke the translator on other python code (provided that code was sufficiently > RPythonic, as I understand it). > > If we did that, it only seems fair to also install it below pypy's > site-packages, and ship that within a "pypy-pypy" subpackage :) (my hope is to > eventually build out a parallel-installable family of pypy-* rpms, mirroring > our python-* rpms) > > > > What's necessary however for a binary to run is listed by > > pypy/tool/release/package.py in source. If this file seems confusing, it's > > essentially what is present in our nightly runs (with rules how it got created > > listed in package.py, I can elaborate if needed). > Thanks. I looked briefly through this script: it looks like it's what you use > to make the tarballs. Yop. > > > > PyPy contains lots of stuff and not all of it is necessary for a C/POSIX build > > (but most of it is). If I may ask, is it possible to run the test suite after > > the build to check if stuff actually worked (If so, I can provide more > > information how to actually run this)? > I'd actually started doing this, and ran into some problems, so more > information would be v. useful. > > Specifically, I tried putting the built binary through the Python test suite > using: > ./pypy -m test.regrtest > from the build's "translator/goal" directory. > > I saw a hang in test_asynchat, so I switched to: > ./pypy -m test.regrtest -x test_asynchat > > Various minor failures scroll by, but then I started running into intermittent > segfaults, towards the "s" part of the alphabet (e.g. test_scope, test_shelve, > test_sqlite), which go away when running the tests by themselves. This makes > me think that there's some kind of state buildup over the course of running the > full test suite. That's really weird. We usually run this by py.test -A over the whole suite, however we have ways to invoke separate directories in separate test processes (reasons vary). I'll look into having a way to run this in a simple manner, will get back later. > > Unfortunately I wasn't able to get meaningful information out of gdb for these > crashes. > > I tried without JIT, by using > --jit threshold=-1 > but this did not fix these segfaults. > > I then tried running the whole thing under valgrind, seeing lots of interesting > warnings, so I switched to just launching the interactive interpreter under > valgrind: > $ valgrind --track-origins=yes --leak-check=full ./pypy --jit threshold=-1 > and I see various warnings; I'll attach them to bugzilla (this comment is > getting far too long as is). > > There's also the test suite for pypy's modules, is that "just" a case of > running: > python test_all.py > from the pypy subdirectory? > > Thanks!
(In reply to comment #40) > well, it goes into "don't ship stuff in a way that does not work" policy :) > Note that there might be a misunderstanding here. We're deleting the jars from the source tree prior to building as the policy is to not build with any pre-built binaries (except for bootstrapping). It seems like the translator/jvm stuff is neither a pre-built binary (so doesn't need to be deleted) nor installed into the package that end-users install onto their system (so we aren't shipping it; we just don't delete it from the source as part of building pypy). It would be different if we did ship the pypy directory to end users but you don't think that's a good idea. (On a side note, I originally got interested in pypy because of the possibility of translating a somewhat pythonic language into javascript. But I see that that's not going to be a supported use of rpython in the future.)
(In reply to comment #41) > (In reply to comment #40) > > well, it goes into "don't ship stuff in a way that does not work" policy :) > > > Note that there might be a misunderstanding here. We're deleting the jars from > the source tree prior to building as the policy is to not build with any > pre-built binaries (except for bootstrapping). It seems like the > translator/jvm stuff is neither a pre-built binary (so doesn't need to be > deleted) nor installed into the package that end-users install onto their > system (so we aren't shipping it; we just don't delete it from the source as > part of building pypy). > > It would be different if we did ship the pypy directory to end users but you > don't think that's a good idea. (On a side note, I originally got interested > in pypy because of the possibility of translating a somewhat pythonic language > into javascript. But I see that that's not going to be a supported use of > rpython in the future.) Ah ok! Now I understand it much better. Than "sure, go ahead, sorry for trouble" from my side.
pypy-1.4.1 was released earlier today, so I've updated pypy.spec to use that Updated spec file is here (as before): http://fedorapeople.org/~dmalcolm/python-packaging/pypy.spec (I also copied in a patch we apply to the 2.7 stdlib in our cpython "python" src.rpm to the 2.5 stdlib in pypy, fixing one of the selftests) SRPM: http://fedorapeople.org/~dmalcolm/python-packaging/pypy-1.4.1-1.fc13.src.rpm Changes to spec file: http://fedorapeople.org/~dmalcolm/python-packaging/pypy-from-1.4-4-to-1.4.1-1.diff Successful scratch build: http://koji.fedoraproject.org/koji/taskinfo?taskID=2681960 Some minor glitches lead to new rpmlint warnings: > pypy-libs.i686: E: backup-file-in-package /usr/share/pypy-1.4.1/lib-python/2.5.2/test/test_commands.py.orig Somehow this is coming from my new patch above > pypy-libs.i686: E: version-control-internal-file /usr/share/pypy-1.4.1/lib_pypy/pyrepl/.svn/all-wcprops and again, for numerous other .svn directories. These appeared in the 1.4.1 tarball; I've mentioned them upstream as: https://codespeak.net/issue/pypy-dev/issue612 along with the pre-existing: > W: spurious-executable-perm /usr/share/doc/pypy-libs-1.4.1/demo/bpnn.py My plan: - fix the above rpmlint issues - add a %check - fix the sys.path issue reported in comment #26 (I see this too, when using the scratch builds)
(In reply to comment #43) > My plan: > - fix the above rpmlint issues > - add a %check > - fix the sys.path issue reported in comment #26 (I see this too, when using > the scratch builds) I've fixed the rpmlint issues, and added the beginnings of a %check, although at the moment I've merely established a baseline subset of tests that pass - I had to disable quite a few tests in order to get the overall %check to pass (see the specfile). I haven't yet addressed the sys.path issue (I plan to look at this next). Updated specfile (1.4.1-2) at usual location: http://fedorapeople.org/~dmalcolm/python-packaging/pypy.spec Updated SRPM: http://fedorapeople.org/~dmalcolm/python-packaging/pypy-1.4.1-2.fc13.src.rpm Changes: http://fedorapeople.org/~dmalcolm/python-packaging/pypy-from-1.4.1-1-to-1.4.1-2.diff Successful scratch build: http://koji.fedoraproject.org/koji/taskinfo?taskID=2684874 rpmlint is clean on this build: $ rpmlint * | sort 7 packages and 0 specfiles checked; 0 errors, 0 warnings.
(In reply to comment #44) > I haven't yet addressed the sys.path issue (I plan to look at this next). I've created a fix for the library path issue, see: https://codespeak.net/issue/pypy-dev/issue614 and moved the libraries from /usr/share to /usr/lib{64} The installed binary now works: [david@surprise ~]$ pypy Python 2.5.2 (, Dec 22 2010, 23:14:02) [PyPy 1.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``PyPy is an exciting technology that lets you to write fast, portable, multi-platform interpreters with less effort'' >>>> import sys >>>> sys.path ['', '/usr/lib64/pypy-1.4.1/lib_pypy', '/usr/lib64/pypy-1.4.1/lib-python/modified-2.5.2', '/usr/lib64/pypy-1.4.1/lib-python/2.5.2', '/usr/lib64/pypy-1.4.1/lib-python/2.5.2/plat-linux2', '/usr/lib64/pypy-1.4.1/site-packages'] >>>> import collections >>>> collections.__file__ '/usr/lib64/pypy-1.4.1/lib_pypy/collections.pyc' Latest specfile is at usual place: http://fedorapeople.org/~dmalcolm/python-packaging/pypy.spec Updated SRPM: http://fedorapeople.org/~dmalcolm/python-packaging/pypy-1.4.1-3.fc13.src.rpm Changes: http://fedorapeople.org/~dmalcolm/python-packaging/pypy-from-1.4.1-2-to-1.4.1-3.diff Successful scratch build of an earlier version of this (in which only comments have since changed): http://koji.fedoraproject.org/koji/taskinfo?taskID=2685524 rpmlint gains a warning: $ rpmlint *.rpm|sort 7 packages and 0 specfiles checked; 0 errors, 2 warnings. pypy-libs.i686: W: only-non-binary-in-usr-lib pypy-libs.x86_64: W: only-non-binary-in-usr-lib but given that we're deliberately moving things to %{_libdir}, as per comment #21, I think it's reasonable to waive this. How is this looking?
I was sorta waiting for the dust to settle before doing another review pass. Is it ready for that now?
(In reply to comment #46) > I was sorta waiting for the dust to settle before doing another review pass. :) > Is it ready for that now? The built package should now actually work, so: yes please - I'd appreciate someone taking a look. Thanks!
Wow. This is much nicer now. ;) I really appreciate all the comments in the spec explaining how things are done/why they are done. All the issues from my initial review seem fixed. The source matches up from upstream. We can ignore the 2 rpmlint issues that are left. I'm not seeing any further blockers here, so this package is APPROVED. I'm sure you'll work on fixing tests and improving things over time as time permits. ;)
New Package SCM Request ======================= Package Name: pypy Short Description: Python implementation with a Just-In-Time compiler Owners: dmalcolm Branches: InitialCC:
Git done (by process-git-requests).
Thanks; I've imported the sources into git and am trying a build now: Building pypy-1.4.1-3.fc15 for dist-rawhide Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=2699335
Successfully built into rawhide as pypy-1.4.1-3.fc15 (see above link). I'm going to close this review request out as "NEXTRELEASE". Thanks everyone! (In reply to comment #48) ... > I'm sure you'll work on fixing tests and improving things over time as time > permits. ;) I've already filed 5 bugs against pypy in Fedora, to try to track the various FIXMEs and TODOs discussed in the .spec file, and in this review; I'll try to work on these over the coming days.
Package Change Request ====================== Package Name: pypy New Branches: el5 el6 Owners: dmalcolm InitialCC: tomspur