Bug 110325

This doesn't address the O(n) problem, it just tries to curb it
somewhat.  I don't see a way to make the build.xml size O(1) without
circumventing Ant's dependency mechanism.  Specicifically, I think
Ant insists on having all dependency spelled out statically via
the "depends" attribute of the "target" element.  If there was
a way to specify dependencies dynamically at runtime, then
the size of build.xml could be made O(1).

Indeed the project.xml for APLAWS containing a huge number of apps:

dan@camden$ grep ccm:application project.xml   | wc -l
     36
dan@camden$ ls -lh build.xml 
-rw-rw-r--    1 dan      ccm-deve     2.1M Nov 10 10:10 build.xml
dan@camden$ time ant init
init:
     [echo] /var/ccm-devel/dev/dan/aplaws-rickshaw/build.xml

BUILD SUCCESSFUL
Total time: 27 seconds

real    0m28.685s
user    0m31.780s
sys     0m0.480s
dan@camden$ 


I think we should have two goals:

1. Reduce the startup time
2. Reduce the time to do a complete 'ant deploy' when no files have
changed.

Point 1. is most urgently needed, since we've got a fixed 30 sec
startup overhead at the mo. Most of that time seems to be parsing &
processsing the build.xml.

Parameterized targets look like a definite candidate for lowering this
overhead. Another improvment may come from using the ant-contrib
package which provides an '<if>' conditional. This would lets us
combine many of the wrapper targets, eg _compile-core & compile-core.

http://ant-contrib.sourceforge.net/tasks/index.html


Finally, we should probably audit just what targets we have defined -
we may find that some are completely unneccessary.

Oh as a final note, we should change the dependencies we encode so
that instead of using the @buildOrder attribute  or 'position()' of
current <ccm:application> tag, it would pull in the actual
dependencies defined in the application.xml file.

For example, the 'xmlfeed' application is listed 34th in the
project.xml. So if I wan't to compile it with the current method, ant
will first  compile the previous 33 applications, however, xmlfeed
only depends on Core & CMS, so the minimal build is actually only 2
applications.

The final should probably go under bug 107191.  Let's keep this ticket
reasonably small.

On the face of it, parsing a 5MB file should not take 30 seconds.  So
I thought that maybe Ant takes a lot of time to process target
dependencies.  I set up a large number of (fake) apps like so:

|$ cat /tmp/fakeapps.sh 
|#!/bin/bash
|
|for nn in `seq 50`; do
|    app="fakeapp$nn"
|    ln -s /var/vadim/p4checkout/core-platform/dev $app
|    echo "<ccm:application name=\"$app\" prettyName=\"Fake App\" buildOrder=\"$nn\"/>"
|done

After modifying project.xml and rerunning ccm-configure.sh, the
resulting build.xml file is abut 3MB:

|$ ls -l build.xml 
|-rw-rw-r--    1 vadim    ccm-devel  3058810 Nov 18 11:25 build.xml

The time it takes to parse this file is negligible:

|$ java -classpath /tmp/xp.jar com.jclark.xml.apps.Time build.xml
|0.516

Running "ant init" produces results very similar to Dan's.

|$ grep -c depends build.xml
|1976
|
|$ time ant init
|Total time: 44 seconds
|
|real    0m45.270s
|user    0m35.410s
|sys     0m0.380s

Removing all the "depends" attributes from "target" elements changes
nothing:

|$ perl -i.bak -npe 's/depends="[^"]+"//' build.xml 
|$ grep -c depends build.xml
|0
|
|$ time ant init
|Total time: 40 seconds
|
|real    0m41.037s
|user    0m35.230s
|sys     0m0.250s

I can't find the link just now, but, although build.xml may *look*
like XML, ANT doesn't actually use a standard XML parser - it is hand
crafted. For example it complains bitterly if you add an
xmlns:foo="http://www.example.com" attribute to any tag.

Without using a profiler we can't really say with much confidence
whether it is parsing, or constructing the in memory objects while
parsing that takes the time. optmizeit would likely show us the
hotspot(s) very quickly.....

Created attachment 96319 [details]
Ant dependency graph for a single project.

The diagram was generated with the help of
//users/vadim/code/xsl/ant2dot.xsl#2.

I set up a dummy project.xml that contained a single application named "p" for
brevity.  I generated build.xml by running ccm-configure.sh and fed it to the
above-mentioned script like so: "$ ant2dot.sh build.xml init".

I removed the "init" node (and all its incident edges) to unclutter the graph.

Whether the Ant's homegrown parser is too slow or constructing
in-memory target objects takes a lot of time, it seems clear that
reducing the size of build.xml would reduce build times.

One thing we can do is write a wrapper script in Python or Perl (or in
pure XSLT for the masochistically inclined) that would trim down
build.xml by removing targets that are not required for the execution
of the requested target.

For example, as shown in attachment 96319 [details], running
"deploy-jar-classes-p" does not require any other target that starts
with "deploy-".  It only requires "build-p" and its prerequisites.
The wrapper script can compute all the required prerequisites and spit
out a temporary file trimmed-build.xml that contains only the necessary
targets.   It can then invoke ant like so:

$ ant deploy-jar-classes-p trimmed-build.xml

instead of the slower

$ ant deploy-jar-classes-p build.xml

Perhaps we are looking at the problem from totally the wrong angle.
The reason why build.xml is so large is that each application has 30
odd targets & there are 20-30 applications. Now, why do we need to
have separate targets per application ? It is because running a full
'ant deploy' is too slow, that people like to shortcut with 'ant
deploy-myapp'. However, we've added many dependancies between targets
so 'deploy-myapp' practically does the same thing about the full
'deploy'. 

So if we removed *all* per-application targets the build.xml would be
tiny. If we looked at each target & ensured it had optimal execution
then running 'deploy' would be fast enough that there is no need for
per-app targets.

The only case for a per-app target I see is 'compile-myapp' - useful
when you have written a bunch of code & you are iterating over
compile, view errors, fix, compile, etc cycles. In this use case there
is no need for dependancies on other compile targets & since we're not
deploying there's no risk of an inconsistent webapp.

Incidentally I've profiled 'ant init' and it seems the big overhead is
in constructed ant's in memory representation of the build.xml file.
We could try and optmize ant's  code for this (they use Strings
instead of StringBuffers when processing the character(byte[], int,
int) method of SAX DefaultHandler :-( ). Failing that, the only way to
reduce time is to make build.xml smaller.

I'm marking this as high since slow performance of ANT is really
impacting on development times for people working on > 10 apps at once.

And what about people with > 30 apps?  This whole build thing is on
crack.  I'm inclined to devote several days to write my own build
system -- eventually it'll make my life easier and more productive. 
On my box 'ant init' takes 40 seconds!!!

My suggestion is: scrap everything!  Split the damned things into 10
or more separate build.xml files.  I'm not interested in ant checking
for data model changes when I'M FIXING THE SINGLE TYPO IN JAVA CODE!!!
 I'm not interested in running junit test when MY CODE DOESN'T COMPILE!!!

Or better yet, scrap ant, make/make+ is the way to go.

One other thought: we could slice the number of apps by 10 if we made
all the content types one app, instead of many. 

You could install the entire CT RPM, and then use the UI to
selectively install the CTs in the UI. The disadvantage is that you
can't incrementally update each of the separate CTs, but I think
that's OK for CTs that are part of product.

> So if we removed *all* per-application targets the build.xml would
> be tiny.
...
> the big overhead is in constructed ant's in memory representation of
> the build.xml file.  We could try and optmize ant's code for this

That might the best long-term solution (provide we push our Ant patch
upstream).  As a short-term solution, the wrapper script approach
seems better, as it preserves the current behavior intact, while
likely offering a substantial speedup.

> Split the damned thing into 10 or more separate build.xml files.

That's effectively what the wrapper script approach would do.

> I'm not interested in running junit test when MY CODE DOESN'T
> COMPILE!!!

"ant compile-core" or some such does not have "runtests" as a
prerequisite.  See compile-p in attachment 96319 [details].

> Or better yet, scrap ant, make/make+ is the way to go.

This has been proposed, too.  I think Rafi toyed with this idea.

> This whole build thing is on crack.

Most targets (except for "larry", "kaboom" and maybe a few others)
were motivated by bugs at some point in the past.  Looking at
attachment 96319 [details], which targets (or dependencies) would you suggest
cutting out?

> "ant compile-core" or some such does not have "runtests" as a
> prerequisite.  See compile-p in attachment 96319 [details].

  I know that.  My point is: I spent 99% of my ant time compiling the
thing.  If I run junit tests, then I can accept 40 seconds startup
delay, because a) I'm not running them that often b) the task itself
takes substantially longer then 40 seconds.

  I vote for 2 (worded: TWO) build.xml files.  The first would contain
only compile/deploy targets.  The second one would be everything else,
actually the one we currently have.  The first should be ~50 k in
size, the other one 2MB+ (in case of 30+ applications).
 

> My point is: I spent 99% of my ant time compiling the thing.

Ah, I see what you mean.  Makes sense.  As long as we are campaigning
to reduce compilation times, I'd like to mention JavaMake again:
http://www.experimentalstuff.com/Technologies/JavaMake/ant.html

> One other thought: we could slice the number of apps by 10 if we made
> all the content types one app, instead of many.

In short, not an option. We've been through this discussion before. It
is a requirement of APLAWS-II that the content types are separated out.
In any case we would still have 20 applications & 1MB build.xml.

I've analysed just what targets are being generated in build.xml and
some definite optimizations come to mind. For APLAWS build.xml, there
are 1936 targets in general. Of out these:

 * 148 end in -hook. These are checks to see if a build-hoooks.xml
file exists. Only two applications (core&chat) currently uses this
capability. Rather than having targets that check for presence of
build-hooks.xml per run, we should have an attribute on
application.xml saying whether there is a build-hooks for this app.
The check is then a one-time event & we can remove 148 targets.

 * 224 end in -check. These are checks to see if various top level
directories (eg, pdl, sql, src, web) exist & thus whether or not to
run the corresponding compile/deploy type targets for each type of
file. Again nearly every single app has all these directories all the
time. We should add an attribute to application.xml which developers
can use to explicitly inform us if a particular directory is not used.
We can then remove 224 targets & speed up build time by not doing all
these checks. 


By removing these two items we can combine 'compile-[appname]' and
'_compile-[appname]' into one target.

 * 187 contain 'jar'. These are only used when making binary
distributionms. We can strip these all out & put them in a separate
build-dist.xml. The build scripts can then be updated to pass in '-f
build-dist.xml' when running ant. This will also elminate a large
number of other tasks deploy related tasks

* There a debugtests and runtests. Does anyone ever use debugtests ?
Can we remove it & just let them pass '-Dsomeprop=true' as an option
to 'runtests'.

Doing the first two points alone will simplify build.xml dramatically
& remove a huge number of filesystem checks, without causing any
serious change in functionality or compatability problems. This should
be top priority since it gives us a short term win for very little effort.

Subsequently we should consider the splitting of build.xml for the
build scripts.

Created attachment 96424 [details]
A list of targets in APLAWS build.xml grouped by type

implemented Dan's suggestion to eliminate unnecessary -hooks targets
@38639

change 38711 implements Dan's suggestion to do the directory checks at
ccm-configure.sh time instead of at ant time.  Then @38728 I checked
in updated application.xml files for all of the content types with
their directory info.  

I have compared the current build.xml with the build.xml before 38639.
 In my project.xml I have 21 applications - core, cms, and 19 content
types.  The number of targets has been cut in half and the startup
time reduced by 25%.

# new
04:41:10 dgregor@galileo foo4$ time ant init
PATH:
/usr/share/java/xml-commons-apis.jar:/usr/share/java/jaxp_parser_impl.jar:/usr/share/java/ant-optional.jar:/usr/share/java/ant.jar:/usr/share/java/httpunit.jar:/usr/share/java/junit.jar:/opt/oracle/product/9.2.0.1/jdbc/lib/classes12.jar::/opt/oracle/product/9.2.0.1/jdbc/lib/classes12.zip:/usr/java/j2sdk1.4.2_01/lib/tools.jar
Buildfile: build.xml

init:
     [echo] /var/ccm-devel/dev/dgregor/foo4/build.xml

BUILD SUCCESSFUL
Total time: 9 seconds

real	0m10.272s
user	0m8.710s
sys	0m0.170s
04:41:29 dgregor@galileo foo4$ ant -projecthelp | grep "^ " | wc -l
    522

#old
04:43:36 dgregor@galileo foo4$ time ant init
PATH:
/usr/share/java/xml-commons-apis.jar:/usr/share/java/jaxp_parser_impl.jar:/usr/share/java/ant-optional.jar:/usr/share/java/ant.jar:/usr/share/java/httpunit.jar:/usr/share/java/junit.jar:/opt/oracle/product/9.2.0.1/jdbc/lib/classes12.jar::/opt/oracle/product/9.2.0.1/jdbc/lib/classes12.zip:/usr/java/j2sdk1.4.2_01/lib/tools.jar
Buildfile: build.xml

init:
     [echo] /var/ccm-devel/dev/dgregor/foo4/build.xml

BUILD SUCCESSFUL
Total time: 12 seconds

real	0m12.677s
user	0m10.900s
sys	0m0.280s
04:43:55 dgregor@galileo foo4$ ant -projecthelp | grep "^ " | wc -l
   1048

Another idea is that we can turn ant into a daemon.  So, the first
time you run it, it loads build.xml into memory and sets up the
objects, etc.  Then it just sticks around, listening on some port. 
Subsequent calls use the running ant process.  This could definitely
be a cool feature to push upstream.

I thought someone has done that already.  I vaguely remember
Mike Bonnet playing with some hacked up daemon-enabled
version of Ant eons ago.

Created attachment 96498 [details]
tiny-build.xml


  Being extremely frustrated with the current situation, I decided to
write my own tiny-build.xml which would take care only of often needed
targets.  I started dropping the targets I don't use often (for them
I can always resort to autogenerated build.xml). however that approach
led me nowhere -- my total ignorance of XSL being one of the obstacles.

  So I took the opposite approach, and started writing tiny-build.xml
with only one target: compile (which I use 90% of the time).  Eventually I
came up with 3 targets: precompile, compile and deploy.

  The idea was to have single build/classes directory where all compiled
classes should go.  All */src directories are included in compile
source path. 

  The problem started when I tried to run some command tool with
'ccm-run'.  It turned out that ccm-run utilizes ccm.classpath file which
gives precedence to classes compiled in <app>/build/classes directories
of individual applications, instead of $CCM_HOME/webapps/WEB-INF/classes,
where the tiny-build.xml (and the original build.xml) deploys compiled
classes.

  It stroke me as something illogical -- the consequence being I cannot
use tiny-build.xml to compile classes run by ccm-run.  So I reduced
ccm.classpath to following:

/var/ccm-devel/web/sskracic/coventry/webapps/WEB-INF/classes
/var/ccm-devel/web/sskracic/coventry/webapps/WEB-INF/lib
/usr/share/java

  After having done that, ccm-run was happy with the classes deployed
by tiny-build.xml.

  Here's the quick comparison of autogenerated build.xml and
tiny-build.xml.  The test was run on most frequently used scenario:
single change in single class file (!).

  First, sizes:

[sskracic@tuborg coventry]$ ls -l *build.xml
-rw-rw-r--    1 sskracic ccm-devel  1272555 Dec 10 11:29 build.xml
-r--r--r--    1 sskracic ccm-devel     3726 Dec 12 17:16 tiny-build.xml

  Execution times for single change in single class file;

  build.xml:	 42 seconds
  tiny-build.xml: 6 seconds

> The idea was to have single build/classes directory where all
> compiled classes should go.  All */src directories are included in
> compile source path.

This makes it possible to introduce undesirable dependencies.  With
the above setup, you can make Core depend on CMS, and stuff will
happily compile.  May not be a showstopper, but definitely something
to keep eye an out for.

> So I reduced ccm.classpath to following:
> /var/ccm-devel/web/sskracic/coventry/webapps/WEB-INF/classes
> /var/ccm-devel/web/sskracic/coventry/webapps/WEB-INF/lib
> /usr/share/java

The reason that ccm.classpath in the development environment has that
directories that it has is because the build system separates
'building' from 'deploying', and ccm.classpath is set up to use the
build directories, not the deploy directories.  So, there are actually
two subtle issues here.

1) Should the build system separate building and deploying?  One
benefit to the separation is that you can build (and run tests)
without affecting your running server.  Also, the separation provides
functionality with little drawback as a developer can just as easily
run 'ant deploy' as 'ant build'

2) Should ccm.classpath use the build directories or the deploy
directories?  I could be convinced pretty easily to switch to using
the deploy directories.  What do other people think?

Now, switching gears in thought, I do not like the idea of splitting
build.xml into multiple files for a few reasons.  However, I am in
favor of adding a switch to ccm-configure that would let the user
specify the template file, and I could add a build-template-lite.xsl
to the ccm-devel package.

The following properties appear to be unused:

app.name
app.pretty.name
apps.core.webapp.name
build.pdl.dir
build.test.pdl.dir
ccm.apps.dist.dir
ccm.dist.conf.dir
compile.nowarn
enterprise.build.dir
javacc.home.dir
javadoc.dist.dir
javadoc.src.dir
shared.lib.dist.dir
slash
webapp.dist.dir
webinf.resources.dir

@39055 I changed build-template.xsl to better use path refs, reducing
reduncancy in classpath declarations.

before:
wc -l build.xml
  36023 build.xml

after:
wc -l build.xml
  17511 build.xml

Dennis, what about the "app.prettyname" property that was introduced
in change 39057?  Where is it supposed to be defined?

I downloaded and installed the latest ccm-devel-2.0.0-2 RPM from
park-street this morning, and here's the list of unused properties
that I get:

$ xsltproc \
 /var/vadim/p4checkout/users/vadim/code/xsl/build-properties.xsl \
 build.xml | \
 /var/vadim/p4checkout/users/vadim/code/xsl/build-properties.pl | sort
app.name
app.pretty.name
apps.article.webapp.name
apps.cms.webapp.name
apps.core.webapp.name
apps.formitem.webapp.name
apps.formsectionitem.webapp.name
build.pdl.dir
build.test.pdl.dir
doc.dir
javacc.home.dir
javadoc.dist.dir
javadoc.src.dir
shared.lib.dist.dir
webapp.dist.dir
webinf.dir

I am closing this ticket.  Since the ticket was filed, a lot of
performance improvements have been made to build.xml.  With a test
project of 41 applications, the number of lines in in build.xml is now
1/3 of what it was before, and startup time is 1/4th.

# as of Nov-18-2003
dgregor@galileo project$ wc -l build.xml
  46069 build.xml
dgregor@galileo project$ time ant init > /dev/null
                                                                     
                                                                     
                 real    0m29.828s
user    0m28.940s
sys 0m0.310s

# as of Jan-23-2004
dgregor@galileo project$ wc -l build.xml
  15847 build.xml
dgregor@galileo project$ time ant init > /dev/null
                                                                     
                                                                     
                 real    0m7.927s
user    0m7.140s
sys 0m0.270s