Bug 732194 - pypy src.rpm requires large amount of RAM to rebuild (6GB ?)
Summary: pypy src.rpm requires large amount of RAM to rebuild (6GB ?)
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: pypy
Version: 15
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Dave Malcolm
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-20 15:03 UTC by Sergei LITVINENKO
Modified: 2011-08-22 17:03 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-08-22 17:03:59 UTC


Attachments (Terms of Use)

Description Sergei LITVINENKO 2011-08-20 15:03:39 UTC
Description of problem:
it is not possible to rebuild src.rpm


Version-Release number of selected component (if applicable):
pypy-1.5-1.fc15.src.rpm

How reproducible:
100%

Steps to Reproduce:
1.rpmbuild --rebuild pypy-1.5-1.fc15.src.rpm
2.
3.
  
Actual results:

...
[Timer] Timings:
[Timer] annotate                       --- 1284.8 s
[Timer] rtype_lltype                   ---  897.6 s
[Timer] pyjitpl_lltype                 ---  897.1 s
[Timer] backendopt_lltype              ---  235.0 s
[Timer] stackcheckinsertion_lltype     ---   47.5 s
[Timer] database_c                     ---  332.0 s
[Timer] source_c                       ---  456.3 s
[Timer] ===========================================
[Timer] Total:                         --- 4150.3 s
[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 306, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/driver.py", line 810, in proceed
[translation:ERROR]     return self._execute(goals, task_skip = self._maybe_skip())
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/tool/taskengine.py", line 116, in _execute                                                                                                                           
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/driver.py", line 286, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/driver.py", line 530, in task_source_c
[translation:ERROR]     exe_name=exe_name)
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/c/genc.py", line 270, in generate_source                                                                                                                             
[translation:ERROR]     defines = defines)
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/c/genc.py", line 961, in gen_source_standalone                                                                                                                       
[translation:ERROR]     sg.gen_readable_parts_of_source(f)
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/c/genc.py", line 820, in gen_readable_parts_of_source                                                                                                                
[translation:ERROR]     for node, impl in nodeiter:
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/c/genc.py", line 706, in subiter
[translation:ERROR]     impl = '\n'.join(list(node.implementation())).split('\n')
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/c/node.py", line 862, in implementation
[translation:ERROR]     for s in self.funcgen_implementation(funcgen):
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/c/node.py", line 871, in funcgen_implementation                                                                                                                      
[translation:ERROR]     funcgen.implementation_begin()
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/c/funcgen.py", line 157, in implementation_begin                                                                                                                     
[translation:ERROR]     self.collect_var_and_types()
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/translator/c/funcgen.py", line 139, in collect_var_and_types                                                                                                                    
[translation:ERROR]     seen[v] = True
[translation:ERROR]    File "/home/sergeil/desk/src/BUILD/pypy-1.5-src/pypy/tool/identity_dict.py", line 20, in __setitem__
[translation:ERROR]     self._keys[id(arg)] = arg
[translation:ERROR]  MemoryError
[translation] batch mode, not calling interactive helpers



Expected results:
rebuilding produce new set of rpm packages 


Additional info:
4Go of RAM + 4Go of swap

Comment 1 Sergei LITVINENKO 2011-08-20 15:09:30 UTC
Some more info about environment

[sergeil@ua-dudn00000 download]$ cat /etc/issue
Fedora release 15 (Lovelock)
Kernel \r on an \m (\l)


[sergeil@ua-dudn00000 download]$ uname -r
2.6.40.3-0.fc15.i686.PAE

[sergeil@ua-dudn00000 download]$ top
top - 18:05:18 up  4:46,  4 users,  load average: 0.01, 0.07, 0.36
Tasks: 160 total,   1 running, 159 sleeping,   0 stopped,   0 zombie
Cpu(s):  6.7%us,  0.3%sy,  0.0%ni, 92.9%id,  0.0%wa,  0.1%hi,  0.0%si,  0.0%st
Mem:   3911676k total,   815648k used,  3096028k free,    41808k buffers
Swap:  4605948k total,        0k used,  4605948k free,   353112k cached


[sergeil@ua-dudn00000 download]$ rpm -qa | grep pypy
pypy-1.5-1.fc15.i686
pypy-libs-1.5-1.fc15.i686

[sergeil@ua-dudn00000 download]$ rpm -qa | grep python
rpm-python-4.9.0-9.fc15.i686
python-lockfile-0.8-2.fc15.noarch
python-tempita-0.4-7.fc15.noarch
python-simplejson-2.1.6-1.fc15.1.i686
python-markupsafe-0.11-3.fc15.i686
python-meh-0.11-2.fc15.noarch
farsight2-python-0.0.25-1.fc15.i686
python-slip-gtk-0.2.17-1.fc15.noarch
python-iwlib-0.1-6.fc15.i686
plasma-scriptengine-python-4.6.5-2.fc15.i686
python-devel-2.7.1-7.fc15.i686
mod_python-3.3.1-16.fc15.i686
python-sexy-0.1.9-14.fc15.i686
gamin-python-0.1.10-9.fc15.i686
libsemanage-python-2.0.46-4.fc15.i686
python-chardet-2.0.1-3.fc15.noarch
at-spi-python-1.32.0-5.fc15.i686
python-2.7.1-7.fc15.i686
spice-gtk-python-0.5-6.fc15.i686
python-slip-dbus-0.2.17-1.fc15.noarch
python-libs-2.7.1-7.fc15.i686
python-pycurl-7.19.0-9.fc15.i686
python-IPy-0.75-1.fc15.noarch
python-ldap-2.3.12-2.fc15.i686
python-feedparser-5.0.1-1.fc15.noarch
python-BeautifulSoup-3.2.0-2.fc15.noarch
python-GnuPGInterface-0.3.2-8.fc15.noarch
gnome-python2-2.28.1-4.fc15.i686
python-ethtool-0.7-2.fc15.i686
python-xlib-0.15-0.5.rc1.fc15.noarch
python-imaging-1.1.7-4.fc15.i686
python-telepathy-0.15.19-3.fc15.noarch
policycoreutils-python-2.0.86-7.fc15.i686
gnome-python2-gnomevfs-2.28.1-4.fc15.i686
python-boto-1.9b-7.fc15.noarch
newt-python-0.52.12-3.fc15.i686
cracklib-python-2.8.18-2.fc15.i686
python-cups-1.9.57-1.fc15.i686
kross-python-4.6.5-1.fc15.i686
gnome-python2-libegg-2.25.3-33.fc15.i686
python-mako-0.3.6-2.fc15.noarch
libuser-python-0.57.2-1.fc15.i686
gnome-python2-gnome-2.28.1-4.fc15.i686
gnome-python2-libwnck-2.32.0-4.fc15.i686
python-httplib2-0.6.0-5.fc15.noarch
gnome-python2-extras-2.25.3-33.fc15.i686
gstreamer-python-0.10.19-2.fc15.i686
libproxy-python-0.4.7-1.fc15.noarch
setools-libs-python-3.3.7-12.fc15.i686
audit-libs-python-2.1.2-1.fc15.i686
gtk-vnc-python-0.4.3-1.fc15.i686
python-slip-0.2.17-1.fc15.noarch
python-crypto-2.3-4.fc15.i686
python-paste-1.7.5.1-2.fc15.noarch
libselinux-python-2.0.99-4.fc15.i686
gnome-python2-gconf-2.28.1-4.fc15.i686
python-twisted-core-10.2.0-2.fc15.i686
python-lxml-2.3-1.fc15.i686
python-enum-0.4.4-3.fc15.noarch
compizconfig-python-0.9.4-1.fc15.i686
python-beaker-1.5.3-8.fc15.noarch
python-cloudfiles-1.7.9.1-1.fc15.noarch
gnome-python2-gtkhtml2-2.25.3-33.fc15.i686
notify-python-0.1.1-18.fc15.i686
python-urlgrabber-3.9.1-10.fc15.noarch
python-virtkey-0.50-9.fc15.i686
python-kitchen-1.0.0-1.fc15.noarch
python-zope-interface-3.6.1-8.fc15.i686
python-smbc-1.0.11-1.fc15.i686
boost-python-1.46.0-3.fc15.i686
python-bugzilla-0.6.1-4.fc15.noarch
python-decorator-3.3.0-2.fc15.noarch
gnome-python2-gnomekeyring-2.32.0-4.fc15.i686
libxslt-python-1.1.26-8.fc15.i686
dbus-python-0.83.0-8.fc15.i686
gnome-python2-canvas-2.28.1-4.fc15.i686
python-iniparse-0.4-3.fc15.noarch
abrt-addon-python-2.0.3-1.fc15.i686
gnome-python2-desktop-2.32.0-4.fc15.i686
gnome-python2-bonobo-2.28.1-4.fc15.i686
libxml2-python-2.7.8-6.fc15.i686

Comment 2 Dave Malcolm 2011-08-22 17:03:59 UTC
The RPython to C translation needs a _lot_ of RAM.

Upstream says "As of March 2011, you will need at least 2 GB of memory on a 32-bit machine and 4GB on a 64-bit machine.":
http://doc.pypy.org/en/latest/getting-started-python.html#translating-the-pypy-python-interpreter

and it may be that that estimate is out of date.  Also, I suspect that this downstream-only patch:
  Patch4: pypy-1.5-more-readable-c-code.patch
adds to the memory requirement.

In my local builds on this x86_64 box, I've seen pypy use at least 7-8GB of RAM,(this box has 8GB, and was driven deep into swap); builds can take 2-3 hours IIRC.

AFAIK, there's no way to specify:

   BuildMemoryRequires: 6GB

or somesuch within a specfile.

It may help the RAM requirements to do the translation with PyPy, rather than CPython: see the:
  %global use_self_when_building 0
variable within the specfile (obviously you need to have already done it once).

There are some other notes on that upstream page on tweaking the build.

Other than that, reducing the build-time RAM requirements significantly would require major reworking of the translation toolchain.

If you're looking to hack on PyPy, have a look at the non-translated version upstream:
http://doc.pypy.org/en/latest/getting-started-python.html#running-the-python-interpreter-without-translation
That's what upstream do when working on the code.

Hope this is helpful.  Closing this out "CANTFIX" (sorry).


Note You need to log in before you can comment on or make changes to this bug.