Bug 472702 - Fix use of python/generator.py in parallel builds
Summary: Fix use of python/generator.py in parallel builds
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Daniel Veillard
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-11-23 21:05 UTC by Michael Marineau
Modified: 2010-03-16 17:16 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-03-12 10:54:27 UTC
Embargoed:


Attachments (Terms of Use)
Parallel build fix (686 bytes, patch)
2008-11-23 21:05 UTC, Michael Marineau
no flags Details | Diff

Description Michael Marineau 2008-11-23 21:05:55 UTC
Created attachment 324445 [details]
Parallel build fix

When doing a parallel build of libvirt with make -jX where X is >1 generator.py gets called several times all at once because it generates several files at the same time while make assumes each call will only generate one file. For example with make -j3:

Making all in python
make[2]: Entering directory `/var/tmp/portage/app-emulation/libvirt-0.4.6/work/libvirt-0.4.6/python'
Making all in .
make[3]: Entering directory `/var/tmp/portage/app-emulation/libvirt-0.4.6/work/libvirt-0.4.6/python'
/usr/bin/python ./generator.py .
/usr/bin/python ./generator.py .
/usr/bin/python ./generator.py .
/usr/lib64/python2.5/xmllib.py:9: DeprecationWarning: The xmllib module is obsolete.  Use xml.sax instead.
  warnings.warn("The xmllib module is obsolete.  Use xml.sax instead.", DeprecationWarning)
/usr/lib64/python2.5/xmllib.py:9: DeprecationWarning: The xmllib module is obsolete.  Use xml.sax instead.
  warnings.warn("The xmllib module is obsolete.  Use xml.sax instead.", DeprecationWarning)
/usr/lib64/python2.5/xmllib.py:9: DeprecationWarning: The xmllib module is obsolete.  Use xml.sax instead.
  warnings.warn("The xmllib module is obsolete.  Use xml.sax instead.", DeprecationWarning)
Found 134 functions in libvirt-api.xml
Found 134 functions in libvirt-api.xml
Found 1 functions in libvirt-python-api.xml
Generated 125 wrapper functions
Missing type converters: 

Found 1 functions in libvirt-python-api.xml
Generated 125 wrapper functions
Missing type converters: 

cat ./libvir.py libvirtclass.py > libvirt.py-t
mv libvirt.py-t libvirt.py
Found 134 functions in libvirt-api.xml
Found 1 functions in libvirt-python-api.xml
Generated 125 wrapper functions
Missing type converters: 

As you can see above libvirtclass.py is being read by cat and written by one of the generator processes at the same time which sometimes results in a partially truncated libvirt.py.

The attached patch fixes this issue by using a dummy stamp file to force generator.py to run only once.

Comment 1 Cole Robinson 2008-11-25 20:14:15 UTC
Hmm, I think this was already addressed upstream. Are you using stock libvirt 0.4.6 or was this against an upstream checkout?

See https://bugzilla.redhat.com/show_bug.cgi?id=463733 for more info.

Comment 2 Michael Marineau 2008-11-26 01:14:54 UTC
I didn't notice that other bug report, but the fix posted there only partially fixed the issue. Make is still waiting for generator.py to finish but it is still running generator.py as many times as -j allows. From the current CVS:

Making all in python
make[2]: Entering directory `/tmp/libvirt/python'
Making all in .
make[3]: Entering directory `/tmp/libvirt/python'
/usr/bin/python ./generator.py .
/usr/bin/python ./generator.py .
/usr/bin/python ./generator.py .

So, it is very unlikely to get corruption but the make process is still being dumb. :-) Touching a stamp file as my patch does is an easy way to make it only run once.

Comment 3 Cole Robinson 2008-12-05 17:16:59 UTC
Hi, would you mind sending the patch to libvir-list? Things get better notice on the list. If not, I can send it along for you if you'd prefer.

Comment 4 Daniel Veillard 2009-02-26 13:49:36 UTC
Okay, that should now be fixed upstream, as I commited the
patch in CVS,

  thanks to both !

Daniel

Comment 5 Daniel Berrangé 2009-03-12 10:54:27 UTC
This patch is included in the 0.6.1 release


Note You need to log in before you can comment on or make changes to this bug.