Bug 1127804 - ivy artifacts with type,ext,classifier resolve incorrectly
Summary: ivy artifacts with type,ext,classifier resolve incorrectly
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: xmvn
Version: rawhide
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
Assignee: Mikolaj Izdebski
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-07 15:17 UTC by Pete MacKinnon
Modified: 2014-11-10 06:35 UTC (History)
4 users (show)

Fixed In Version: 2.1.0-5
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-28 16:44:22 UTC


Attachments (Terms of Use)
debug from ivy build (136.24 KB, application/x-gzip)
2014-08-07 15:17 UTC, Pete MacKinnon
no flags Details
reproducer (10.00 KB, application/x-tar)
2014-10-23 15:05 UTC, Pete MacKinnon
no flags Details

Description Pete MacKinnon 2014-08-07 15:17:09 UTC
Created attachment 924940 [details]
debug from ivy build

Basically it looks like something wrong with this kind of ivy dependency declaration:

<dependency org="org.apache.hadoop" name="hadoop-hdfs"
                rev="${hadoop-0.23.version}"
                conf="hadoop0.23.shim->default">
      <artifact name="hadoop-hdfs" type="tests" ext="jar" m:classifier="tests"/>
...

This will resolve to the hadoop-hdfs.jar, not the hadoop-hdfs-tests jar. It's as if the hadoop-hdfs jar is computed as a compat for the tests version. 

[ivy:retrieve] 	retrieving /usr/share/java/hadoop/hadoop-hdfs.jar
[ivy:retrieve] 		to /home/pmackinn/rpmbuild/BUILD/hive-release-0.12.0/build/ivy/lib/hadoop0.23.shim/hadoop-hdfs-SYSTEM-tests.jar

25958  2523 /usr/share/java/hadoop/hadoop-hdfs-tests.jar
14226  6667 /usr/share/hadoop/hdfs/hadoop-hdfs.jar
14226  6667 ../BUILD/hive-release-0.12.0/build/ivy/lib/hadoop0.23.shim/hadoop-hdfs-SYSTEM-tests.jar

Comment 1 Mikolaj Izdebski 2014-09-24 07:56:15 UTC
Confirmed.

Comment 2 Mikolaj Izdebski 2014-09-24 07:59:23 UTC
Reproducer:

$ cat ivy.xml
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="test" module="test"/>
  <dependencies>
    <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="1.0">
      <artifact name="hadoop-hdfs" type="tests" ext="jar" m:classifier="tests"/>
    </dependency>
  </dependencies>
</ivy-module>

$ cat build.xml
<project xmlns:ivy="antlib:org.apache.ivy.ant">
  <target name="run">
    <ivy:retrieve />
  </target>
</project>

$ xmvn-resolve org.apache.hadoop:hadoop-hdfs:jar:tests:
/usr/share/java/hadoop/hadoop-hdfs-tests.jar

$ cmp lib/hadoop-hdfs-SYSTEM-tests.jar /usr/share/java/hadoop/hadoop-hdfs-tests.jar
lib/hadoop-hdfs-SYSTEM-tests.jar /usr/share/java/hadoop/hadoop-hdfs-tests.jar differ: byte 15, line 1

Comment 3 Mikolaj Izdebski 2014-09-24 09:07:01 UTC
This problem is due to limitations of Maven POM format (Maven doesn't have separate POM for test JARs) and I cannot do anything about that.

Adding m:classifier="tests" attribute to <dependency> node makes Ivy resolve test JAR correctly.

Comment 4 Pete MacKinnon 2014-10-02 12:32:54 UTC
Sorry, more information...

$ cat ivy.xml
<!-- valid upstream ivy btw -->
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="test" module="test"/>
  <dependencies>
    <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="2.4.1">
      <artifact name="hadoop-hdfs" ext="jar"/>
      <artifact name="hadoop-hdfs" type="tests" ext="jar" m:classifier="tests"/>
    </dependency>
  </dependencies>
</ivy-module>

Same build.xml as reproducer...

$ rm -fr lib/ && ant run && sum lib/hadoop-hdfs*.jar
<snip>
56593  6670 lib/hadoop-hdfs-2.4.1.jar
11038  2504 lib/hadoop-hdfs-2.4.1-tests.jar

$ rm -fr lib/ && ant -Divy.mode=local run && sum lib/hadoop-hdfs*.jar
<snip>
15127  6665 lib/hadoop-hdfs-SYSTEM.jar
15127  6665 lib/hadoop-hdfs-SYSTEM-tests.jar

$ sum /usr/share/java/hadoop/hadoop-hdfs.jar /usr/share/java/hadoop/hadoop-hdfs-tests.jar
15127  6665 /usr/share/java/hadoop/hadoop-hdfs.jar
54852  2521 /usr/share/java/hadoop/hadoop-hdfs-tests.jar

It appears that the xmvn ivy resolver can't distinguish between the two jars (regular and tests), or thinks it's already resolved it in the model maybe?

Comment 5 Mikolaj Izdebski 2014-10-03 08:01:38 UTC
I've debugged this before closing. What happens here is:
1) Ivy asks XMvn "give me model for hadoop-hdfs-tests"
2) because Maven POMs can't have classifiers, XMvn replies that such model doesn't exist
3) Ivy removes classifier retries: "give me model for hadoop-hdfs"
4) XMvn resolves and returns POM for hadoop-hdfs
5) Ivy asks XMvn "give me JAR for hadoop-hdfs"
6) XMvn resolves and returns JAR for hadoop-hdfs

The real problem here is incompatibility between Maven and Ivy model semantics. XMvn tries to provide a bridge between these formats where possible. This usually saves packagers from having to install both pom.xml and ivy.xml files, but in some cases (like this) it may be necessary to install both to make both Ivy and Maven happy.

There are at least two possible workarounds:
1) add m:classifier="tests" attribute to <dependency> node
2) install ivy.xml file for hadoop test JAR

Comment 6 Pete MacKinnon 2014-10-03 16:53:03 UTC
$ cat ivy.xml
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="test" module="test"/>
  <dependencies>
    <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="2.4.1">
       <artifact name="hadoop-hdfs" ext="jar"/>
    </dependency>
    <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="2.4.1" m:classifier="tests"/>
  </dependencies>
</ivy-module>

$ cat build.xml
<project xmlns:ivy="antlib:org.apache.ivy.ant">
  <target name="run">
    <ivy:retrieve />
  </target>
</project>

Same issue even if they are listed as separate dep nodes instead of separate artifacts.

Comment 7 Pete MacKinnon 2014-10-03 17:24:02 UTC
The respective provides:

mvn(org.apache.hadoop:hadoop-hdfs) = 2.4.1
mvn(org.apache.hadoop:hadoop-hdfs::tests:) = 2.4.1

Comment 8 Mikolaj Izdebski 2014-10-06 07:34:44 UTC
The case from comment #6 doesn't work because Ivy considers two dependency nodes as equal and it uses only the second one (only the second dependency is considered, the first one is ignored). You can swap them to see that. This has nothing to do with local resolver.

Comment 9 Pete MacKinnon 2014-10-23 15:05:06 UTC
Created attachment 949969 [details]
reproducer

run with:

# normal
rm -fr lib/ && ant run && sum lib/hadoop-hdfs*

# xmvn
rm -fr lib/ && ant -Divy.mode=local run && sum lib/hadoop-hdfs*

Comment 10 Mikolaj Izdebski 2014-10-28 16:08:59 UTC
Fixed upstream: https://github.com/mizdebsk/xmvn/commit/bcad3a746a5074ee6819a78b004df13a8ddd611d

Comment 11 Mikolaj Izdebski 2014-10-28 16:09:59 UTC
Fixed in xmvn-2.1.0-5

Comment 12 Mikolaj Izdebski 2014-10-28 16:44:22 UTC
I believe that this bug is fixed in xmvn-2.1.0-5,
which is available in Fedora Rawhide, so I am closing this bug now.

The build containing the fix can be found at Koji:
http://koji.fedoraproject.org/koji/buildinfo?buildID=588620

Comment 13 Fedora Update System 2014-10-28 17:10:15 UTC
xmvn-2.0.1-2.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/xmvn-2.0.1-2.fc21

Comment 14 Fedora Update System 2014-11-10 06:35:30 UTC
xmvn-2.0.1-2.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.