Bug 477076 - R CMD javareconf in %post binds Java-based R packages to particular revs of Java
R CMD javareconf in %post binds Java-based R packages to particular revs of Java
Status: CLOSED NEXTRELEASE
Product: Fedora EPEL
Classification: Fedora
Component: R (Show other bugs)
el5
x86_64 Linux
low Severity low
: ---
: ---
Assigned To: Tom "spot" Callaway
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-12-18 18:51 EST by Chloe Lopez
Modified: 2009-03-05 11:30 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-07 23:18:35 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Chloe Lopez 2008-12-18 18:51:08 EST
Description of problem:
In the %post section, the R-core rpm in EPEL for RHEL5 (x86_64) does

    R CMD javareconf > /dev/null 2>&1 || exit 0

R's javareconf command is pretty smart, and is able figure out values that work for each of the variables it sets: JAR, JAVA, JAVAC, JAVA_HOME, JAVAH, JAVA_CPPFLAGS, JAVA_LIBS, and JAVA_LD_LIBRARY_PATH. 

Unfortunately, the values it typically finds are canonical paths (no symlinks) which are specific to particular versions/installations of java. For instance JAVA_HOME might get set to /usr/lib/jvm/java-1.6.0-sun-1.6.0.11.x86_64/jre. This completely disregards the "alternatives" system extant in RHEL. 

This in itself might not be so bad, since you could run "R CMD javareconf" every time your java changes or gets updated. (This solution is not great for automated management of RPMs, though.) 

But things get more complicated if you have compiled java-based R packages against the old version of java. Suddenly these packages fail to find the java libraries, even if all that's changed is the release number, and the libraries should theoretically work. R CMD javareconf doesn't help, because the libraries that the java-based R package is looking for no longer exist.

Version-Release number of selected component (if applicable):

R-core-2.8.0-2.el5 from EPEL for RHEL5

How reproducible:
Steps to Reproduce:
1. Install an old version of some Java implementation (say, java-sun-1.6.0)
2. Install R
3. Compile and install a java-based R package like rJava
4. Update the java implementation to a newer version of the same implementation
  
Actual results:

rJava fails to find the correct libraries.

Expected results:

rJava should continue to work as long as the main java version and implemntation, as specified in /etc/alternatives, hasn't changed.

Additional info:

I built a custom R rpm that solves this problem that has the following javareconf command in %post:

R CMD javareconf \
    JAR=%{jar} \
    JAVA=%{java} \
    JAVAC=%{javac} \
    JAVA_HOME=%{java_home}/jre \
    JAVAH=%{java_home}/bin/javah \
    JAVA_CPPFLAGS='-I%{java_home}/include\ -I%{java_home}/include/linux' \
    JAVA_LIBS='-L%{java_home}/jre/lib/%{java_arch}/server\ -L%{java_home}/jre/lib/%{java_arch}\ -L%{java_home}/lib/%{java_arch}\ -L/usr/java/packages/lib/%{java_arch}\ -L/lib\ -L/usr/lib\ -l\ jvm' \
    JAVA_LD_LIBRARY_PATH=%{java_home}/jre/lib/%{java_arch}/server:%{java_home}/jre/lib/%{java_arch}:%{java_home}/lib/%{java_arch}:/usr/java/packages/lib/%{java_arch}:/lib:/usr/lib \
|| exit 0

This also has the benefit of using the right RPM macros (AFAICT) for the Java directories, except for %{java_arch}, which I defined earlier in the spec file:

%ifarch x86_64
%define java_arch amd64
%else
%define java_arch i386
%endif

This solves the problem, at least in the limited context where I encountered it. It probably does not solve every case. In additon, if you were to build an RPM for the Java-based R package, you could specify the java dependency there in a sane way, if need be.

I am especially uncertain about what happens or should happen if you have multiple java implementations installed and you install an R java-based package that was built against an R set up like this and a java implementation that is not the main one in /etc/alternatives. 

One could instead have this javareconf in the %build section of the Java-based R package, but that would require you to build it as root.

I think it's okay for a RHEL/Fedora package of R to assume that java will be located where the Fedora Java guidelines specify. This will obviously not work if the RPM is installed on a non-Fedora distro, or if some third-party java needs to be used.
Comment 1 Fedora Update System 2008-12-22 16:13:19 EST
rpy-1.0.3-6.fc9,R-2.8.1-1.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/rpy-1.0.3-6.fc9,R-2.8.1-1.fc9
Comment 2 Fedora Update System 2008-12-22 16:13:21 EST
rpy-1.0.3-6.fc10,R-2.8.1-1.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/rpy-1.0.3-6.fc10,R-2.8.1-1.fc10
Comment 3 Fedora Update System 2008-12-24 07:57:35 EST
rpy-1.0.3-6.fc9, R-2.8.1-1.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing-newkey update rpy R'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-11698
Comment 4 Fedora Update System 2008-12-24 13:43:42 EST
rpy-1.0.3-6.fc9, R-2.8.1-1.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing-newkey update rpy R'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-11698
Comment 5 Fedora Update System 2008-12-24 13:44:48 EST
rpy-1.0.3-6.fc10, R-2.8.1-1.fc10 has been pushed to the Fedora 10 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update rpy R'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F10/FEDORA-2008-11782
Comment 6 Fedora Update System 2009-01-05 14:05:15 EST
R-2.8.1-2.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/R-2.8.1-2.fc9
Comment 7 Fedora Update System 2009-01-05 14:05:19 EST
R-2.8.1-2.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/R-2.8.1-2.fc10
Comment 8 Fedora Update System 2009-01-07 04:31:17 EST
R-2.8.1-2.fc10 has been pushed to the Fedora 10 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update R'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F10/FEDORA-2009-0197
Comment 9 Fedora Update System 2009-01-07 04:32:55 EST
R-2.8.1-2.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing-newkey update R'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2009-0212
Comment 10 Fedora Update System 2009-01-07 14:52:19 EST
rpy-1.0.3-6.fc9,R-2.8.1-2.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/rpy-1.0.3-6.fc9,R-2.8.1-2.fc9
Comment 11 Fedora Update System 2009-01-07 14:52:23 EST
rpy-1.0.3-6.fc10,R-2.8.1-2.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/rpy-1.0.3-6.fc10,R-2.8.1-2.fc10
Comment 12 Fedora Update System 2009-01-07 23:18:33 EST
rpy-1.0.3-6.fc9, R-2.8.1-2.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 13 Fedora Update System 2009-01-07 23:19:19 EST
rpy-1.0.3-6.fc10, R-2.8.1-2.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 14 Elia Pinto 2009-02-27 04:21:29 EST
The described bug persist in EPEL5 and, BTW, FC10.

I have had the same problem and it is very hackery to do a rpm package for rJava.
Comment 15 Elia Pinto 2009-02-27 04:42:37 EST
Sorry, i was wrong. The problem is another.

R-core have the correct %post

but with gnu jre

 R CMD javareconf \
>     JAR=/usr/lib/jvm/jre/bin/jar \
>     JAVA=/usr/lib/jvm/jre/bin/java \
>     JAVAC=/usr/lib/jvm/jre/bin/javac \
>     JAVA_HOME=/usr/lib/jvm/jre/jre \
>     JAVAH=/usr/lib/jvm/jre/bin/javah \
>     JAVA_CPPFLAGS='-I/usr/lib/jvm/jre/include\ -I/usr/lib/jvm/jre/include/linux' \
>     JAVA_LIBS='-L/usr/lib/jvm/jre/jre/lib/amd64/server \
>     -L/usr/lib/jvm/jre/jre/lib/amd64\ -L/usr/lib/jvm/jre/lib/amd64 \
>     -L/usr/java/packages/lib/amd64\ -L/lib\ -L/usr/lib\ -ljvm' \
>     JAVA_LD_LIBRARY_PATH=/usr/lib/jvm/jre/jre/lib/amd64/server:/usr/lib/jvm/jre/jre/lib/amd64:/usr/lib/jvm/jre/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
/usr/lib64/R/bin/javareconf: line 54:  -L/usr/lib/jvm/jre/jre/lib/amd64 -L/usr/lib/jvm/jre/lib/amd64: No such file or directory
*** JAVA_HOME is not a valid path, ignoring
Java interpreter : /usr/lib/jvm/jre/bin/java
Java version     : 1.4.2
Java home path   : /usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
/usr/lib64/R/bin/javareconf: line 142: /usr/lib/jvm/jre/bin/javac: No such file or directory
Java compiler    : not functional
Java headers gen.: /usr/lib/jvm/jre/bin/javah
Java archive tool: /usr/lib/jvm/jre/bin/jar

 ** Warning: you are using custom JAVA_LD_LIBRARY_PATH without
    custom JAVA_LIBS. We cannot guarantee that this will work,
    both settings must match for a proper Java operation!
Java library path: /usr/lib/jvm/jre/jre/lib/amd64/server:/usr/lib/jvm/jre/jre/lib/amd64:/usr/lib/jvm/jre/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
JNI linker flags : -L/usr/lib64/gcj-4.1.2 -ljvm
JNI cpp flags    : -I/usr/lib/jvm/jre/include -I/usr/lib/jvm/jre/include/linux

Updating Java configuration in /usr/lib64/R

What is more

ls -l /usr/lib/jvm/jre/bin/javah
ls: /usr/lib/jvm/jre/bin/javah: No such file or directory
ls -l /usr/lib/jvm/jre/bin/javac
ls: /usr/lib/jvm/jre/bin/javac: No such file or directory

On FC10 with openjdk

 R CMD javareconf     JAR=/usr/lib/jvm/jre/bin/jar     JAVA=/usr/lib/jvm/jre/bin/java     JAVAC=/usr/lib/jvm/jre/bin/javac     JAVA_HOME=/usr/lib/jvm/jre/jre     JAVAH=/usr/lib/jvm/jre/bin/javah     JAVA_CPPFLAGS='-I/usr/lib/jvm/jre/include\ -I/usr/lib/jvm/jre/include/linux'     JAVA_LIBS='-L/usr/lib/jvm/jre/jre/lib/amd64/server \
    -L/usr/lib/jvm/jre/jre/lib/amd64\ -L/usr/lib/jvm/jre/lib/amd64 \
    -L/usr/java/packages/lib/amd64\ -L/lib\ -L/usr/lib\ -ljvm'     JAVA_LD_LIBRARY_PATH=/usr/lib/jvm/jre/jre/lib/amd64/server:/usr/lib/jvm/jre/jre/lib/amd64:/usr/lib/jvm/jre/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
/usr/lib64/R/bin/javareconf: line 54:  -L/usr/lib/jvm/jre/jre/lib/amd64 -L/usr/lib/jvm/jre/lib/amd64: No such file or directory
*** JAVA_HOME is not a valid path, ignoring
Java interpreter : /usr/lib/jvm/jre/bin/java
Java version     : 1.6.0_0
Java home path   : /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
/usr/lib64/R/bin/javareconf: line 142: /usr/lib/jvm/jre/bin/javac: No such file or directory
Java compiler    : not functional
Java headers gen.: /usr/lib/jvm/jre/bin/javah
Java archive tool: /usr/lib/jvm/jre/bin/jar

 ** Warning: you are using custom JAVA_LD_LIBRARY_PATH without
    custom JAVA_LIBS. We cannot guarantee that this will work,
    both settings must match for a proper Java operation!
Java library path: /usr/lib/jvm/jre/jre/lib/amd64/server:/usr/lib/jvm/jre/jre/lib/amd64:/usr/lib/jvm/jre/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -L$(JAVA_HOME)/lib/amd64 -L$(JAVA_HOME)/../lib/amd64 -L -L/usr/java/packages/lib/amd64 -L/usr/lib64 -L/lib64 -L/lib -L/usr/lib -ljvm
JNI cpp flags    : -I/usr/lib/jvm/jre/include -I/usr/lib/jvm/jre/include/linux

Updating Java configuration in /usr/lib64/R
Done.

 ls -l /usr/lib/jvm/jre/bin/javah
ls: cannot access /usr/lib/jvm/jre/bin/javah:

ecc.

So it is an alternatives problem ?

I have no clue. Thanks
Comment 16 Chloe Lopez 2009-02-27 15:53:49 EST
The Java macros are set (on RHEL5, at least) by the macro file in /etc/rpm/macros.jpackage (from the jpackage-utils RPM). The macros are set at build time by sourcing /usr/share/java-utils/java-functions and running shell functions defined there.

I have no idea what the build environment is like where the RPMs are being built, but it looks like the Java RPM macros are getting set to the wrong thing. My suspicion is that it has to do with the fact that java-functions sets JAVA_HOME differently depending on whether a $JVM_ROOT/java directory exists. If it does, that's the value of $JAVA_HOME. If it doesn't, $JAVA_HOME is set to $JVM_ROOT/jre. My initial investigations indicate that the weird values will happen if a JRE is installed at build time, but not a full Java SDK. Probably the easiest solution, if that's the case, is to include a -devel Java package as a BuildRequires.

But you don't need a Java SDK to run R or even to run Java-based R packages. You -do- need it to -build- Java-based R packages. I bring this up because I think there may be a valid question as to what the values of JAVAC, JAVAH, and JAVA_CPPFLAGS should be if the end user doesn't have or want an SDK installed. Or what if they didn't have one, but installed one later? Should the R.spec file be using the %{_jvmdir} macro rather than the %{java_home} macro (though that doesn't help with %{javac})?

And the whole point of javareconf is that it's supposed to do the right thing. Maybe if it gets the right value of JAVA_HOME, it will find the right JAVA, JAR, JAVAC and JAVAH. What this bug really cares about is JAVA_LIBS, JAVA_CPPFLAGS, and JAVA_LD_LIBRARY_PATH, which are set by asking the installed java where to look, and which get wrong answers because java is ignorant of /etc/alternatives. This approach would solve the problem of dealing with whether the end user has a JRE or SDK installed, but wouldn't deal with what happens when they install an SDK later on. (Should R or R-devel require java-devel?)

Javareconf is smart enough to deal with non-existent JAVAC or JAVAH, and who cares what your CPPFLAGS are set to if you don't have an SDK installed. The real problem is that JAVA_HOME should not ever be set to /usr/lib/jvm/jre/jre, a clearly bogus path. Maybe it should look more like this:

R CMD javareconf \
    JAVA_HOME=%{_jvmdir}/jre \
    JAVA_CPPFLAGS='-I%{_jvmdir}/java/include\ -I%{_jvmdir}/java/include/linux' \
    JAVA_LIBS='-L%{_jvmdir}/jre/lib/%{java_arch}/server \
    -L%{_jvmdir}/jre/lib/%{java_arch}\ -L%{_jvmdir}/java/lib/%{java_arch} \
    -L/usr/java/packages/lib/%{java_arch}\ -L/lib\ -L/usr/lib\ -ljvm' \
    JAVA_LD_LIBRARY_PATH=%{_jvmdir}/jre/lib/%{java_arch}/server:%{_jvmdir}/jre/lib/%{java_arch}:%{_jvmdir}/java/lib/%{java_arch}:/usr/java/packages/lib/%{java_arch}:/lib:/usr/lib \
    > /dev/null 2>&1 || exit 0


1) JAR, JAVA, JAVAC, and JAVAH should be left to javareconf to figure out
2) JAVA_CPPFLAGS should use %{_jvmdir}/java instead of %{java_home}
3) JAVA_LIBS and JAVA_LD_LIBRARY_PATH should 
   a) use %{_jvmdir}/jre wherever they currently have ${java_home}/jre 
   b) %{_jvmdir}/java in place of other uses of %{java_home}. 

The idea here is that %{_jvmdir}/jre will exist whether you have an SDK installed or not, so you should use it when you can. Nonexistent %{_jvmdir}/java in LD_LIBRARY_PATH and LIBS is harmless, so you specify both /jre and /java versions. So only CPPFLAGS gets a value that might break if there's no SDK. 

It's too bad that %{java_home} isn't more useful, but maybe the semantics is supposed to be "Where java is at build time" rather than "Where java is expected to be at run time."
Comment 17 Tom "spot" Callaway 2009-03-04 15:49:32 EST
I have _no_ idea how %{java_home} got evaluated to: /usr/lib/jvm/jre/jre (the double jre is wrong). With the macros in jpackage-utils, it should evaluate to /usr/lib/jvm/jre everytime. Nevertheless, Chloe's assessment is accurate, and the proposed scriptlet is much cleaner, so I'm going to use it.
Comment 18 Chloe Lopez 2009-03-04 16:58:22 EST
%{java_home} will evaluate to /usr/lib/jvm/jre if only the JRE is installed on the build host, and so adding "/jre" is not appropriate, but %{java_home} evaluates to /usr/lib/jvm if the SDK is installed, so you have to add /jre.

I think I just misunderstood the semantics of %{java_home} in the first place.
Comment 19 Fedora Update System 2009-03-04 17:01:42 EST
R-2.8.1-4.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/R-2.8.1-4.fc9
Comment 20 Fedora Update System 2009-03-04 17:01:47 EST
R-2.8.1-4.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/R-2.8.1-4.fc10
Comment 21 Tom "spot" Callaway 2009-03-04 17:05:59 EST
(In reply to comment #18)
> %{java_home} will evaluate to /usr/lib/jvm/jre if only the JRE is installed on
> the build host, and so adding "/jre" is not appropriate, but %{java_home}
> evaluates to /usr/lib/jvm if the SDK is installed, so you have to add /jre.
> 
> I think I just misunderstood the semantics of %{java_home} in the first place.

That would be two of us, thanks for the explanation!
Comment 22 Fedora Update System 2009-03-05 11:29:59 EST
R-2.8.1-4.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 23 Fedora Update System 2009-03-05 11:30:52 EST
R-2.8.1-4.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.