Created attachment 924940 [details] debug from ivy build Basically it looks like something wrong with this kind of ivy dependency declaration: <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="${hadoop-0.23.version}" conf="hadoop0.23.shim->default"> <artifact name="hadoop-hdfs" type="tests" ext="jar" m:classifier="tests"/> ... This will resolve to the hadoop-hdfs.jar, not the hadoop-hdfs-tests jar. It's as if the hadoop-hdfs jar is computed as a compat for the tests version. [ivy:retrieve] retrieving /usr/share/java/hadoop/hadoop-hdfs.jar [ivy:retrieve] to /home/pmackinn/rpmbuild/BUILD/hive-release-0.12.0/build/ivy/lib/hadoop0.23.shim/hadoop-hdfs-SYSTEM-tests.jar 25958 2523 /usr/share/java/hadoop/hadoop-hdfs-tests.jar 14226 6667 /usr/share/hadoop/hdfs/hadoop-hdfs.jar 14226 6667 ../BUILD/hive-release-0.12.0/build/ivy/lib/hadoop0.23.shim/hadoop-hdfs-SYSTEM-tests.jar
Confirmed.
Reproducer: $ cat ivy.xml <ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven"> <info organisation="test" module="test"/> <dependencies> <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="1.0"> <artifact name="hadoop-hdfs" type="tests" ext="jar" m:classifier="tests"/> </dependency> </dependencies> </ivy-module> $ cat build.xml <project xmlns:ivy="antlib:org.apache.ivy.ant"> <target name="run"> <ivy:retrieve /> </target> </project> $ xmvn-resolve org.apache.hadoop:hadoop-hdfs:jar:tests: /usr/share/java/hadoop/hadoop-hdfs-tests.jar $ cmp lib/hadoop-hdfs-SYSTEM-tests.jar /usr/share/java/hadoop/hadoop-hdfs-tests.jar lib/hadoop-hdfs-SYSTEM-tests.jar /usr/share/java/hadoop/hadoop-hdfs-tests.jar differ: byte 15, line 1
This problem is due to limitations of Maven POM format (Maven doesn't have separate POM for test JARs) and I cannot do anything about that. Adding m:classifier="tests" attribute to <dependency> node makes Ivy resolve test JAR correctly.
Sorry, more information... $ cat ivy.xml <!-- valid upstream ivy btw --> <ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven"> <info organisation="test" module="test"/> <dependencies> <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="2.4.1"> <artifact name="hadoop-hdfs" ext="jar"/> <artifact name="hadoop-hdfs" type="tests" ext="jar" m:classifier="tests"/> </dependency> </dependencies> </ivy-module> Same build.xml as reproducer... $ rm -fr lib/ && ant run && sum lib/hadoop-hdfs*.jar <snip> 56593 6670 lib/hadoop-hdfs-2.4.1.jar 11038 2504 lib/hadoop-hdfs-2.4.1-tests.jar $ rm -fr lib/ && ant -Divy.mode=local run && sum lib/hadoop-hdfs*.jar <snip> 15127 6665 lib/hadoop-hdfs-SYSTEM.jar 15127 6665 lib/hadoop-hdfs-SYSTEM-tests.jar $ sum /usr/share/java/hadoop/hadoop-hdfs.jar /usr/share/java/hadoop/hadoop-hdfs-tests.jar 15127 6665 /usr/share/java/hadoop/hadoop-hdfs.jar 54852 2521 /usr/share/java/hadoop/hadoop-hdfs-tests.jar It appears that the xmvn ivy resolver can't distinguish between the two jars (regular and tests), or thinks it's already resolved it in the model maybe?
I've debugged this before closing. What happens here is: 1) Ivy asks XMvn "give me model for hadoop-hdfs-tests" 2) because Maven POMs can't have classifiers, XMvn replies that such model doesn't exist 3) Ivy removes classifier retries: "give me model for hadoop-hdfs" 4) XMvn resolves and returns POM for hadoop-hdfs 5) Ivy asks XMvn "give me JAR for hadoop-hdfs" 6) XMvn resolves and returns JAR for hadoop-hdfs The real problem here is incompatibility between Maven and Ivy model semantics. XMvn tries to provide a bridge between these formats where possible. This usually saves packagers from having to install both pom.xml and ivy.xml files, but in some cases (like this) it may be necessary to install both to make both Ivy and Maven happy. There are at least two possible workarounds: 1) add m:classifier="tests" attribute to <dependency> node 2) install ivy.xml file for hadoop test JAR
$ cat ivy.xml <ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven"> <info organisation="test" module="test"/> <dependencies> <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="2.4.1"> <artifact name="hadoop-hdfs" ext="jar"/> </dependency> <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="2.4.1" m:classifier="tests"/> </dependencies> </ivy-module> $ cat build.xml <project xmlns:ivy="antlib:org.apache.ivy.ant"> <target name="run"> <ivy:retrieve /> </target> </project> Same issue even if they are listed as separate dep nodes instead of separate artifacts.
The respective provides: mvn(org.apache.hadoop:hadoop-hdfs) = 2.4.1 mvn(org.apache.hadoop:hadoop-hdfs::tests:) = 2.4.1
The case from comment #6 doesn't work because Ivy considers two dependency nodes as equal and it uses only the second one (only the second dependency is considered, the first one is ignored). You can swap them to see that. This has nothing to do with local resolver.
Created attachment 949969 [details] reproducer run with: # normal rm -fr lib/ && ant run && sum lib/hadoop-hdfs* # xmvn rm -fr lib/ && ant -Divy.mode=local run && sum lib/hadoop-hdfs*
Fixed upstream: https://github.com/mizdebsk/xmvn/commit/bcad3a746a5074ee6819a78b004df13a8ddd611d
Fixed in xmvn-2.1.0-5
I believe that this bug is fixed in xmvn-2.1.0-5, which is available in Fedora Rawhide, so I am closing this bug now. The build containing the fix can be found at Koji: http://koji.fedoraproject.org/koji/buildinfo?buildID=588620
xmvn-2.0.1-2.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/xmvn-2.0.1-2.fc21
xmvn-2.0.1-2.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.