Bug 1239815

Created attachment 1047760 [details]
build.log

Created attachment 1047761 [details]
root.log

Created attachment 1047762 [details]
state.log

This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

We had a discussion with a centos person on IRC, and got a spec together that successfully builds pypy 2.6.0 on centos. I don't know enough to submit a patch that would pass tests, but the issues we encountered are:
- building PyPy 2.6.0 requires python 2.7 (or a pypy python)
- the set of import commands, which used to be required to enable building and installing cffi import libraries, is not part of the translation/complilation process
- PyPy builds the lions share of its code as libpypy-c.so, and then adds a thin executable to call into the shared object. This enables embedding PyPy in other programs. Ideally this shared object would be in some system wide library, but the build process patches a RPATH into the executable so that the so can be found if it is located in the same directory as the executable. This does not play well with soft-linking or hard-linking the executable into another directory.

FWIW, I am uploading the diff to the spec, it is very rough (disables tests) but should give someone an idea of what needs to be done

Created attachment 1059119 [details]
spec diff for PyPy 2.6.0

A rough first cut actually contributed by a CentOS user, posted here with permission.

python-devel currently requires python, so the no change to requires is necessary (for Fedora).

Are you disabling tests because they fail, or for some other reason?

sorry, that is part of the diff that I'm not so sure about. It does seem wrong, tests should all pass but are quite time consuming (could require hours)

Thanks Matti!
Michal Cyprian (CC'd) is currently working on fixing the FTBFS and using upstream package.py, but it's taking a while, since Michal is new to the giant of pypy.spec. :)

(In reply to mattip from comment #5)
> - the set of import commands, which used to be required to enable building
> and installing cffi import libraries, is not part of the
> translation/complilation process

We've been having trouble with this particular part:

File "<stdin>", line 1, in <module>
File "/usr/lib64/pypy-2.6.0/lib_pypy/syslog.py",
line 16, in <module>
from _syslog_cffi import ffi, lib
ImportError: No module named _syslog_cffi

Have you not run into similar issues?

I think it might be fixed post 2.6.0 release. When building the rpm I have:

[translation:info] copied: /builddir/build/BUILD/pypy-2.6.0-src/libpypy-c.so
[translation:info] usession directory: /builddir/build/BUILD/pypy-2.6.0-src/usession-pypy-0
[translation:info] created: /builddir/build/BUILD/pypy-2.6.0-src/pypy-c
[77918] translation-task}
[Timer] Timings:
[Timer] annotate                       ---  551.6 s
[Timer] rtype_lltype                   --- 1033.9 s
[Timer] pyjitpl_lltype                 --- 1329.5 s
[Timer] backendopt_lltype              ---  340.6 s
[Timer] stackcheckinsertion_lltype     ---  317.7 s
[Timer] database_c                     ---  452.0 s
[Timer] source_c                       ---  861.0 s
[Timer] compile_c                      ---  751.7 s
[Timer] ===========================================
[Timer] Total:                         --- 5638.0 s
real    94m54.822s
user    80m57.411s
sys     1m20.253s
+ echo --------------------------------------------------------------

but when running the same command in hg tip I have:

[translation:info] copied: /home/zbyszek/pypy/libpypy-c.so
[translation:info] usession directory: /home/zbyszek/pypy/usession-default-0
[translation:info] created: /home/zbyszek/pypy/pypy-c
[54921] translation-task}
[translation:info] Create cffi bindings for modules...
[54921] {translation-task
starting build_cffi_imports
* _audioop_build.py
* _curses_build.py
* _gdbm_build.py
* _pwdgrp_build.py
* _sqlite3_build.py
* _syslog_build.py
* _tkinter/tklib_build.py
/home/zbyszek/pypy/pypy-c successfully built, but errors while building the above modules will be ignored
[54923] translation-task}
[Timer] Timings:
[Timer] annotate                       ---  558.0 s
[Timer] rtype_lltype                   ---  762.8 s
[Timer] pyjitpl_lltype                 ---  943.0 s
[Timer] backendopt_lltype              ---  256.2 s
[Timer] stackcheckinsertion_lltype     ---  161.9 s
[Timer] database_c                     ---  351.9 s
[Timer] source_c                       ---  408.9 s
[Timer] compile_c                      ---  665.6 s
[Timer] build_cffi_imports             ---    2.8 s
[Timer] ===========================================
[Timer] Total:                         --- 4111.0 s

changeset:   78323:85bc12fb4725
branch:      py3k
parent:      78182:ae79f5787d65
parent:      78314:6d7ee832797e
user:        Manuel Jacob <me>
date:        Fri Jun 26 16:26:06 2015 +0200
summary:     hg merge default

+        # HACKHACKHACK
+        # ugly hack to modify target goal from compile_c to build_cffi_imports
+        # this should probably get cleaned up and merged with driver.create_exe
+        from rpython.translator.driver import taskdef
+        import types
+
+        class Options(object):
+            pass
+
+
+        def mkexename(name):
+            if sys.platform == 'win32':
+                name = name.new(ext='exe')
+            return name
+
+        @taskdef(['compile_c'], "Create cffi bindings for modules")
+        def task_build_cffi_imports(self):
+            from pypy.tool.build_cffi_imports import create_cffi_import_libraries
+            ''' Use cffi to compile cffi interfaces to modules'''
+            exename = mkexename(driver.compute_exe_name())
+            basedir = exename
+            while not basedir.join('include').exists():
+                _basedir = basedir.dirpath()
+                if _basedir == basedir:
+                    raise ValueError('interpreter %s not inside pypy repo', 
+                                     str(exename))
+                basedir = _basedir
+            modules = self.config.objspace.usemodules.getpaths()
+            options = Options()
+            # XXX possibly adapt options using modules
+            failures = create_cffi_import_libraries(exename, options, basedir)
+            # if failures, they were already printed
+            print  >> sys.stderr, str(exename),'successfully built, but errors while building the above modules will be ignored'
+        driver.task_build_cffi_imports = types.MethodType(task_build_cffi_imports, driver)
+        driver.tasks['build_cffi_imports'] = driver.task_build_cffi_imports, ['compile_c']
+        driver.default_goal = 'build_cffi_imports'
+        # HACKHACKHACK end

That is part of the cffi changes I alluded to above. Since cffi 1.0 (an integral part of pypy 2.6.0) there is a separation between runtime and buildtime. At buildtime, pypy builds import libraries for sqlite3, audioop, tk, curses, syslog, gdbm and pwdgrp via a script in pypy/tool/build_cffi_imports. For reference, in pre-cffi-1.0, the runtime libraries were built by importing, as root, the respective modules. That is why the new spec deleted those lines

In pypy 2.6.0 and onwards the script is run both by the translation process and by package.py, as Zbigniew pointed out.

The script creates import libraries in <pypy_sources>/lib_ppy, on my system (ubuntu, sorry) I have these files after translation:
lib_pypy/_audioop_cffi.pypy-26.so  
lib_pypy/_gdbm_cffi.pypy-26.so    
lib_pypy/_sqlite3_cffi.pypy-26.so
lib_pypy/_curses_cffi.pypy-26.so   
lib_pypy/_pwdgrp_cffi.pypy-26.so  
lib_pypy/_syslog_cffi.pypy-26.so

These must be installed as part of the packaging of pypy, probably to /usr/lib64/pypy-2.6.0/lib_pypy ?

(In reply to mattip from comment #5)

> - the set of import commands, which used to be required to enable building
> and installing cffi import libraries, is not part of the
> translation/complilation process


GACK, that should be "is now part of the translation/complilation process" 

Kind of changes the meaning, no?

I see that we actually made building the import libraries part of the translation process only after releasing 2.6.0, so for the 2.6.0 release you _must_ run package.py in order to build the import libraries. Sorry for the mess

I've finally finished spec file using package.py in %install section. It took me more time than I expected. If you have some suggestions or recommendations let me know.