Bug 1012612 - RFE: enhance ivy to to perform implicit local dep resolution
RFE: enhance ivy to to perform implicit local dep resolution
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: javapackages-tools (Show other bugs)
rawhide
All All
high Severity unspecified
: ---
: ---
Assigned To: Mikolaj Izdebski
Fedora Extras Quality Assurance
: FutureFeature, Upstream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-26 13:56 EDT by Pete MacKinnon
Modified: 2014-01-16 08:46 EST (History)
6 users (show)

See Also:
Fixed In Version: 3.5.0-1
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-16 08:46:39 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Pete MacKinnon 2013-09-26 13:56:46 EDT
Currently in ivy-based builds, there are typically a set of a resolvers that point to external repositories (like mvn). A packager could resort to tactics like the following:

-  <property name="repo.dir" value="${user.home}/.m2/repository" override="false"/>
+  <property name="repo.dir" value="/usr/share/java" override="false"/>
-    <url name="sourceforge" m2compatible="false" checksums="">
-      <artifact pattern="${sourceforge-repo}/[module]/files/[module]/[branch]/[module]-[revision](-[classifier]).[ext]"/>
-    </url>
+    <filesystem name="fedora1" m2compatible="false" alwaysCheckExactRevision="false">
+       <artifact pattern="${repo.dir}/[module](-[classifier]).[ext]"/>
     </filesystem>

etc. ...

However, it would be cleaner to have a local mode property that could be passed to the fedora version of ivy that would tell it to intercept the resolver request and attempt to map it to local JPP layout locations.

The challenge would be to try to infer a jar location from the original provided artifact pattern. The "organization" and "module" (and potentially "classifier" fields) would have to be used in possibly various combinations.
Comment 1 Pete MacKinnon 2013-09-26 14:00:17 EDT
By "passed to ivy", obviously I mean something like:

ant -Dlocal.mode=true ...
Comment 2 Pete MacKinnon 2013-09-26 14:04:45 EDT
Related to Bug 1005971
Comment 3 Mikolaj Izdebski 2013-09-27 02:29:19 EDT
Here my thoughts about this topic.

1) Using pattern-based repositories (like in comment #0) works only in some cases. Fedora supports more advanced features like artifact aliases, installing artifact into subdirectories, installing JARs with different names than module name (artifactId). This means that A custom Ivy resolver, which would fully understand Fedora layout, needs to be implemented.

2) Ivy is a library that does artifact resolution.  It can be used as Ant tasks, but it can also be used by other build systems. Enabling it to support local artifact resolution would not only help packages built with Ant, but also other build systems using Ivy.

3) Ivy modules could be dynamically mapped to Maven modules: organization to groupId, module to artifactId and so on. Even module dependencies can be mapped. This means that adding full support for Ivy wouldn't require infrastructure changes.

4) The only software that understands current repository layout used by Fedora is XMvn, which depends on Maven. This means that any package wanting to do local artifact resolution would need to have Maven installed. This may be not a problem at all, but I would rather try to attempt some refactoring to cut unneeded dependencies.
Comment 4 Pete MacKinnon 2013-11-13 10:24:30 EST
Suggestion; consider a command line property alias in order to avoid any project ivy setting patches.

For example, something like...

-Divy.xmvn.alias="ibiblio:xmvn"

Again, this might still necessitate a complementary property which defines a different resolver pattern to be used by the alias. Then, again the Fedora layout is somewhat well-defined...perhaps a series of Fedora artifact patterns could be tried.
Comment 5 Mikolaj Izdebski 2013-11-13 10:34:28 EST
(In reply to Pete MacKinnon from comment #4)
> Again, this might still necessitate a complementary property which defines a
> different resolver pattern to be used by the alias. Then, again the Fedora
> layout is somewhat well-defined...perhaps a series of Fedora artifact
> patterns could be tried.

Fedora repository structure is well defined, but its complexity is beyond capabilities of Ivy pattern-based resolver.

For example, how would parrern-based resolver possibly be able to resolve com.google.collections:google-collections to /usr/share/java/guava.jar?
Comment 6 Fedora Admin XMLRPC Client 2013-12-10 01:38:51 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 7 Mikolaj Izdebski 2013-12-16 06:19:03 EST
Reassigning back to myself.

Ivy Resolver was implemented in XMvn 1.5.0-SNAPSHOT.  It will be eventually integrated with Ivy somehow, but you can already try it out -- there are pre-release XMvn RPMs available at Jenkins.  You can use this repo:

[xmvn-jenkins]
name=xmvn-jenkins
baseurl=http://jenkins.cloud.fedoraproject.org/job/xmvn/ws/RPM/latest

To use it you can add this to your ivysettings.xml:

  <typedef name="xmvn" classname="org.fedoraproject.maven.connector.ivy.IvyResolver"/>
  <resolvers>
    <xmvn name="XMvn"/>
  </resolvers>

For SBT you can add this to .sbt file instead:

  externalResolvers := Seq(new sbt.RawRepository(new org.fedoraproject.maven.connector.ivy.IvyResolver))

(To use the resolver you may need to add the following extra dependencies to classpath: aether/api guava ivy maven/maven-model plexus-classworlds plexus-containers/plexus-container-default plexus/utils xbean/xbean-reflect xmvn/xmvn-connector xmvn/xmvn-core)

Any feedback is very welcome, it may speed things up.
Comment 8 Mikolaj Izdebski 2013-12-16 06:32:16 EST
My proposal of integrating this with Ivy is creating a system property (lets call it "ivy.mode") which would control in which mode Ivy works.  There would be three modes of operation:

1. Upstream mode, which mimics upstream behavior.  This mode is used when ivy.mode property is not defined, but can also be explicitly activated by setting ivy.mode to "upstream".

2. Local mode.  Activated by defining ivy.mode property to "local".  First tries local cache and then local system repository.

3. Bootstrap mode.  Activated by defining ivy.mode property to "bootstrap".  First tries local cache, then system local repository and then remote repositories (shared and public).  This mode is mostly useful in bootstrapping new software for which not all dependencies are yet available in system repository.
Comment 9 Pete MacKinnon 2013-12-18 12:18:42 EST
Hi Mikolaj,

Tried this out with my hive build. Ivy makes it difficult to inject the new resolver into the classpath so I resorted to:

  <classpath file="/usr/share/xmvn/lib/xmvn-core.jar"/>
  <classpath file="/usr/share/xmvn/lib/xmvn-connector.jar"/>
  <classpath file="/usr/share/xmvn/lib/aether_aether-api.jar"/>
  <classpath file="/usr/share/java/plexus-containers/plexus-container-default.jar"/>
  <classpath file="/usr/share/java/plexus-classworlds.jar"/>
  <classpath file="/usr/share/java/plexus/utils.jar"/>
  <classpath file="/usr/share/xmvn/lib/guava.jar"/>
  <typedef name="xmvn" classname="org.fedoraproject.maven.connector.ivy.IvyResolver"/>
  <resolvers>
    <xmvn name="XMvn"/>
  </resolvers>

But got this error:

Caused by: java.lang.IllegalArgumentException: bad method found for xmvn on class org.apache.ivy.core.settings.IvySettings
        at org.apache.ivy.util.Configurator.startCreateChild(Configurator.java:535)
        at org.apache.ivy.core.settings.XmlSettingsParser.inConfiguratorStarted(XmlSettingsParser.java:579)
        at org.apache.ivy.core.settings.XmlSettingsParser.startElement(XmlSettingsParser.java:201)
<snip>

In general, what do you see as the correct way to load the xmvn resolver and its deps into the Ivy classloader? The ivy <classpath> element only accepts a single file or url (no wildcards afaict). I suppose I could locally explode the xmvn jars and deps and pack them into one jar as part of the hive build, but that seems heinous.
Comment 10 Mikolaj Izdebski 2013-12-19 01:38:08 EST
(In reply to Pete MacKinnon from comment #9)
> Caused by: java.lang.IllegalArgumentException: bad method found for xmvn on
> class org.apache.ivy.core.settings.IvySettings

I never saw this.  I wonder if it has anything to do with incpomplete classpath.

> 
> In general, what do you see as the correct way to load the xmvn resolver and
> its deps into the Ivy classloader? The ivy <classpath> element only accepts
> a single file or url (no wildcards afaict). I suppose I could locally
> explode the xmvn jars and deps and pack them into one jar as part of the
> hive build, but that seems heinous.

For ant, this should do it (persistently): echo "aether/api guava ivy maven/maven-model plexus-classworlds plexus-containers/plexus-container-default plexus/utils xbean/xbean-reflect xmvn/xmvn-connector xmvn/xmvn-core" >/etc/ant.d/xmvn

or alternatively (for one session, or without root access): export OPT_JAR_LIST="aether/api guava ivy maven/maven-model plexus-classworlds plexus-containers/plexus-container-default plexus/utils xbean/xbean-reflect xmvn/xmvn-connector xmvn/xmvn-core"
Comment 11 Pete MacKinnon 2014-01-05 11:02:00 EST
Mikolaj, looks good so far:

# rpm -q xmvn maven-local
xmvn-1.5.0-0.9.git5b8a5e3.noarch
maven-local-3.4.2-1.fc21.noarch

<snip>
[ivy:resolve] :: loading settings :: file = /builddir/hive-0.12.0/ivy/ivysettings.xml
[ivy:resolve] :: resolving dependencies :: org.apache.hive#hive-shims;0.12.0
[ivy:resolve]   confs: [hadoop0.23.shim]
[ivy:resolve]   found org.apache.hadoop#hadoop-common;SYSTEM in XMvn
[ivy:resolve]   [SYSTEM] org.apache.hadoop#hadoop-common;2.2.0 (forced)
[ivy:resolve]   found org.apache.zookeeper#zookeeper-test;SYSTEM in XMvn
[ivy:resolve]   [SYSTEM] org.apache.zookeeper#zookeeper-test;any (forced)
[ivy:resolve]   found org.jboss.netty#netty;SYSTEM in XMvn
[ivy:resolve]   [SYSTEM] org.jboss.netty#netty;any (forced)
[ivy:resolve]   found com.jcraft#jzlib;SYSTEM in XMvn
[ivy:resolve]   [SYSTEM] com.jcraft#jzlib;any (forced)
[ivy:resolve]   found junit#junit;SYSTEM in XMvn
[ivy:resolve]   [SYSTEM] junit#junit;any (forced)
...
Comment 12 Mikolaj Izdebski 2014-01-16 08:23:20 EST
Implemented upstream in javapackages-tools 3.5.0.  To use this feature you need to install "ivy-local" package.

There are 3 modes of behaviour, controlled by setting "ivy.mode" property.

1) local mode, activated with -Divy.mode=local
First try Ivy local repository, then system repository through XMvn.

2) bootstrap mode, activated with -Divy.mode=bootstrap
First try Ivy local repository, then system repository through XMvn, finally public remote repositories (download artifacts from the Internet).  This mode is useful when building packages for which not all dependencies have yet been packaged for Fedora.

3) upstream mode, activate by default (-Divy.mode=upstream works too)
Mimic upstream behaviour (local first, then remote).  Never tries to resolve from system repos.

There is no need for custom settings or touching classpath.  Everything should work out of the box after defining "ivy.mode".

To summarize:
  BuildRequires: ivy-local
  ant -Divy.mode=local ...
Comment 13 Mikolaj Izdebski 2014-01-16 08:36:45 EST
Fixed in javapackages-tools-3.5.0-1
Comment 14 Mikolaj Izdebski 2014-01-16 08:46:39 EST
I believe that this feature is included in javapackages-tools-3.5.0-1,
which is available in Fedora Rawhide, so I am closing this bug now.

The build which includes the feature can be found at Koji:
http://koji.fedoraproject.org/koji/buildinfo?buildID=491550

Note You need to log in before you can comment on or make changes to this bug.