Bug 1009863 - Java update breaks ovirt-engine start
Java update breaks ovirt-engine start
Status: CLOSED CURRENTRELEASE
Product: oVirt
Classification: Community
Component: ovirt-engine-installer (Show other bugs)
3.3
Unspecified Unspecified
urgent Severity urgent
: ---
: 3.3.1
Assigned To: Alon Bar-Lev
integration
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-19 06:51 EDT by Alexander Ludas
Modified: 2013-11-25 07:05 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-25 07:05:47 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 19816 None None None Never
oVirt gerrit 19844 None None None Never

  None (edit)
Description Alexander Ludas 2013-09-19 06:51:23 EDT
Description of problem:
After java update from java-1.7.0-openjdk-1.7.0.60-2.4.2.0.fc19.x86_64 to java-1.7.0-openjdk-1.7.0.60-2.4.2.1.fc19.x86_64 ovirt-engine wasn't able to start.

ovirt-engine: ERROR run:485 Error: Directory '/usr/lib/jvm/jre-1.7.0-openjdk-1.7.0.60-2.4.2.0.fc19.x86_64' is required but missing
Version-Release number of selected component (if applicable):

Engine reads JAVA_HOME variable from /etc/ovirt-engine/engine.conf.d/10-setup-java.conf which is set to a different version and fails to start.

How reproducible:

Steps to Reproduce:
1. Install F19 with engine
2. Down- or upgrade java-1.7.0-openjdk java-1.7.0-openjdk-devel
3. Restart engine

Actual results:
Engine fails to start

Expected results:
Engine should start

Additional info:
Changing JAVA_HOME in /etc/ovirt-engine/engine.conf.d/10-setup-java.conf to '/etc/alternatives/jre' fixes the problem.
Comment 1 Juan Hernández 2013-09-20 09:44:33 EDT
Fedora 19 has introduced a new layout in the symlinks created in the /usr/lib/jvm directory. In previous versions of Fedora (and in RHEL and CentOS as well) it was populated with links like these:

  java-1.7.0-openjdk-1.7.0.60.x86_64
  jre -> /etc/alternatives/jre
  jre-1.7.0 -> /etc/alternatives/jre_1.7.0
  jre-1.7.0-openjdk.x86_64 -> java-1.7.0-openjdk-1.7.0.60.x86_64/jre
  jre-openjdk -> /etc/alternatives/jre_openjdk

The setup tool scans this directory, looks for links to real things (not to other links) and selects the one that contains a valid 1.7.0 OpenJDK. This resulted in a name which doesn't change when the package is updated: jre-1.7.0-openjdk.x86_64.

In Fedora 19 the directory is populated as follows (see [1] for details):

  java -> /etc/alternatives/java_sdk
  java-1.7.0 -> /etc/alternatives/java_sdk_1.7.0
  java-1.7.0-openjdk -> /etc/alternatives/java_sdk_1.7.0_openjdk
  java-1.7.0-openjdk-1.7.0.60-2.4.2.1.fc19.x86_64
  java-1.7.0-openjdk.x86_64 -> /usr/lib/jvm/java-1.7.0-openjdk
  java-openjdk -> /etc/alternatives/java_sdk_openjdk
  jre -> /etc/alternatives/jre
  jre-1.7.0 -> /etc/alternatives/jre_1.7.0
  jre-1.7.0_openjdk -> /etc/alternatives/jre_1.7.0_openjdk
  jre-1.7.0-openjdk-1.7.0.60-2.4.2.1.fc19.x86_64 -> java-1.7.0-openjdk-1.7.0.60-2.4.2.1.fc19.x86_64/jre
  jre-openjdk -> /etc/alternatives/jre_openjdk

As you can see the symlink that points to the real installation directory does now contain the release number: jre-1.7.0-openjdk-1.7.0.60-2.4.2.1.fc19.x86_64. This breaks our search algorithm.

Fortunately Fedora 19 also has included "alternatives" support for specific versions of the JRE/JDK:

  # alternatives --list | grep '^jre'
  jre_1.7.0	manual	/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.4.2.1.fc19.x86_64/jre
  jre_openjdk	manual	/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.4.2.1.fc19.x86_64/jre
  jre_1.7.0_openjdk	auto	/usr/lib/jvm/jre-1.7.0-openjdk-1.7.0.60-2.4.2.1.fc19.x86_64

So I would suggest to modify the setup tool so that (only for Fedora 19 and newer) it doesn't use the previous algorithm but hardcoders JAVA_HOME as /usr/lib/jvm/jre-1.7.0_openjdk.


[1] http://pkgs.fedoraproject.org/cgit/java-1.7.0-openjdk.git/commit/?id=0cfdf60a4a315c2e0618478c69f887638eaaf93b
Comment 2 Alon Bar-Lev 2013-09-20 13:48:29 EDT
Hi Juan,

What about using the alternatives as you implied above?

JAVA_HOME="$(alternatives --list | grep ^jre_1.7.0_openjdk | cut -f3)"

This will be somewhat like what I do in Gentoo:

JAVA_HOME="$(java-config --select-vm=icedtea-7 --jre-home)"

And avoid breakage when absolute name is modified again.

Thanks,
Alon

[1] http://gerrit.ovirt.org/#/c/7549/ [My, Aug 30, 2012]
Comment 3 Alon Bar-Lev 2013-09-20 14:18:02 EDT
Juan,

I also re-recommend not to store java home in configuration file per default and search for it during runtime unless explicitly available in configuration.

Alon
Comment 4 Juan Hernández 2013-09-23 04:59:25 EDT
Using the output of "alternatives --list" doesn't seem correct to me, as it the alternatives system is designed exactly to avoid using that information directly. Now that the alternatives system solves the version issue and provides an stable version specific JRE location I think we should use it (only for Fedora 19 or newer).

However you are the owner of the setup/startup scripts now, so do as you find appropriate.
Comment 5 Alon Bar-Lev 2013-09-23 05:19:07 EDT
Thanks.

This change of naming convention was expected, and I do not think it will be the last one.

I am going to move JAVA_HOME detection to runtime, unless explicitly specified in configuration.

There will be an option for distro specific script to return the location of java for product.
Comment 6 Itamar Heim 2013-09-23 06:01:24 EDT
didn't we at some point prefer openjdk to other JREs due to some bugs, etc.?
Comment 7 Alon Bar-Lev 2013-09-23 06:04:31 EDT
(In reply to Itamar Heim from comment #6)
> didn't we at some point prefer openjdk to other JREs due to some bugs, etc.?

yes, and it is not going to change.

the question is which openjdk should we use.

if there is an update we should use the latest.

while in current form we have the JAVA_HOME hardcoded during setup.

it used to work, but fedora changed the naming conventions of the directories.

this means that:

1. we should avoid hardcoding patches during setup.

2. we should be prepared for distro specific modifications.
Comment 8 Alon Bar-Lev 2013-10-02 20:24:01 EDT
commit e2f9b2dda8d81f5bda99d7533971cd9c3517d68a
Author: Alon Bar-Lev <alonbl@redhat.com>
Date:   Thu Oct 3 00:55:26 2013 +0300

    packaging: detect java home at runtime
    
    CURRENT IMPLEMENTATION
    
    detect java home during installation.
    
    advantage is optimize application execution post-setup.
    
    disadvantage is that java home may be altered by system updates.
    
    NEW IMPLEMENTATION
    
    detect java home during runtime using pre-defined algorithm.
    
    algorithm:
    
    execute /usr/share/ovirt-engine/bin/java-home.
    
    if java-home.local is available, execute that script. this enables
    distro specific customization.
    
    if OVIRT_ENGINE_JAVA_HOME environment is set, validate java validity
    and use/abort.
    
    search predefined selected alternatives location by order, first wins.
    
    search /usr/lib/jvm for compatible jvm.
    
    NOTES
    
    the java-home script resides with the lib package as it cannot reside
    within any version locked package since it is used by setup, and it
    cannot reside within setup since it used by post setup.
    
    Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1009863
    Change-Id: I833362d0498d534b06e0d7660f1e8a61e9b1713b
    Signed-off-by: Alon Bar-Lev <alonbl@redhat.com>
Comment 9 Pavel Stehlik 2013-10-31 15:28:06 EDT
Hi Alexander, could you please help us with testing this bug?
 Thank you, P.
Comment 10 Alexander Ludas 2013-10-31 17:57:42 EDT
No problem!

Did some basic testing based on 3.3.1 beta2 and everything works as expected.

Test procedure:
- install 3.3.1b2, run engine-setup
- downgrade openjdk(-devel) from 1.7.0.60-2.4.3.0.fc19.x86_64 to 1.7.0.25-2.3.10.3.fc19.x86_64 && restart ovirt-engine 
- engine-cleanup && engine-setup 
- upgrade java && restart ovirt-engine 

No problems at all.
Comment 11 Sandro Bonazzola 2013-11-07 03:52:17 EST
Targeting to 3.3.2, it's not in 3.3.1 tree.
Comment 12 Sandro Bonazzola 2013-11-07 03:54:34 EST
sorry, it's in 3.3.1, targeting to 3.3.1.
Comment 13 Sandro Bonazzola 2013-11-25 07:05:47 EST
oVirt 3.3.1 has been released

Note You need to log in before you can comment on or make changes to this bug.