Bug 472702 - Fix use of python/generator.py in parallel builds
Fix use of python/generator.py in parallel builds
Status: CLOSED CURRENTRELEASE
Product: Virtualization Tools
Classification: Community
Component: libvirt (Show other bugs)
unspecified
All Linux
medium Severity medium
: ---
: ---
Assigned To: Daniel Veillard
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-11-23 16:05 EST by Michael Marineau
Modified: 2010-03-16 13:16 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-03-12 06:54:27 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Parallel build fix (686 bytes, patch)
2008-11-23 16:05 EST, Michael Marineau
no flags Details | Diff

  None (edit)
Description Michael Marineau 2008-11-23 16:05:55 EST
Created attachment 324445 [details]
Parallel build fix

When doing a parallel build of libvirt with make -jX where X is >1 generator.py gets called several times all at once because it generates several files at the same time while make assumes each call will only generate one file. For example with make -j3:

Making all in python
make[2]: Entering directory `/var/tmp/portage/app-emulation/libvirt-0.4.6/work/libvirt-0.4.6/python'
Making all in .
make[3]: Entering directory `/var/tmp/portage/app-emulation/libvirt-0.4.6/work/libvirt-0.4.6/python'
/usr/bin/python ./generator.py .
/usr/bin/python ./generator.py .
/usr/bin/python ./generator.py .
/usr/lib64/python2.5/xmllib.py:9: DeprecationWarning: The xmllib module is obsolete.  Use xml.sax instead.
  warnings.warn("The xmllib module is obsolete.  Use xml.sax instead.", DeprecationWarning)
/usr/lib64/python2.5/xmllib.py:9: DeprecationWarning: The xmllib module is obsolete.  Use xml.sax instead.
  warnings.warn("The xmllib module is obsolete.  Use xml.sax instead.", DeprecationWarning)
/usr/lib64/python2.5/xmllib.py:9: DeprecationWarning: The xmllib module is obsolete.  Use xml.sax instead.
  warnings.warn("The xmllib module is obsolete.  Use xml.sax instead.", DeprecationWarning)
Found 134 functions in libvirt-api.xml
Found 134 functions in libvirt-api.xml
Found 1 functions in libvirt-python-api.xml
Generated 125 wrapper functions
Missing type converters: 

Found 1 functions in libvirt-python-api.xml
Generated 125 wrapper functions
Missing type converters: 

cat ./libvir.py libvirtclass.py > libvirt.py-t
mv libvirt.py-t libvirt.py
Found 134 functions in libvirt-api.xml
Found 1 functions in libvirt-python-api.xml
Generated 125 wrapper functions
Missing type converters: 

As you can see above libvirtclass.py is being read by cat and written by one of the generator processes at the same time which sometimes results in a partially truncated libvirt.py.

The attached patch fixes this issue by using a dummy stamp file to force generator.py to run only once.
Comment 1 Cole Robinson 2008-11-25 15:14:15 EST
Hmm, I think this was already addressed upstream. Are you using stock libvirt 0.4.6 or was this against an upstream checkout?

See https://bugzilla.redhat.com/show_bug.cgi?id=463733 for more info.
Comment 2 Michael Marineau 2008-11-25 20:14:54 EST
I didn't notice that other bug report, but the fix posted there only partially fixed the issue. Make is still waiting for generator.py to finish but it is still running generator.py as many times as -j allows. From the current CVS:

Making all in python
make[2]: Entering directory `/tmp/libvirt/python'
Making all in .
make[3]: Entering directory `/tmp/libvirt/python'
/usr/bin/python ./generator.py .
/usr/bin/python ./generator.py .
/usr/bin/python ./generator.py .

So, it is very unlikely to get corruption but the make process is still being dumb. :-) Touching a stamp file as my patch does is an easy way to make it only run once.
Comment 3 Cole Robinson 2008-12-05 12:16:59 EST
Hi, would you mind sending the patch to libvir-list@redhat.com? Things get better notice on the list. If not, I can send it along for you if you'd prefer.
Comment 4 Daniel Veillard 2009-02-26 08:49:36 EST
Okay, that should now be fixed upstream, as I commited the
patch in CVS,

  thanks to both !

Daniel
Comment 5 Daniel Berrange 2009-03-12 06:54:27 EDT
This patch is included in the 0.6.1 release

Note You need to log in before you can comment on or make changes to this bug.