Bug 1168405 - Error 500 from JBOSS engine API
Summary: Error 500 from JBOSS engine API
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-engine-api
Version: 3.5
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.5.0
Assignee: Juan Hernández
QA Contact: Gonza
URL:
Whiteboard: integration
Depends On:
Blocks: rhev35rcblocker rhev35gablocker
TreeView+ depends on / blocked
 
Reported: 2014-11-26 20:21 UTC by Anatoly Litovsky
Modified: 2015-02-16 08:18 UTC (History)
19 users (show)

Fixed In Version: org.ovirt.engine-root-3.5.0-23
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1181729 (view as bug list)
Environment:
Last Closed: 2015-02-16 08:18:26 UTC
oVirt Team: ---
Embargoed:


Attachments (Terms of Use)
The jboss error (11.22 KB, text/html)
2014-11-26 20:21 UTC, Anatoly Litovsky
no flags Details
hosted engine setup log (314.42 KB, text/plain)
2014-11-26 20:22 UTC, Anatoly Litovsky
no flags Details
hosted engine setup answers (2.63 KB, text/plain)
2014-11-26 20:23 UTC, Anatoly Litovsky
no flags Details
first failed engine setup (2.00 MB, text/plain)
2014-11-26 20:26 UTC, Anatoly Litovsky
no flags Details
first fail engine setup conf (1.50 KB, text/plain)
2014-11-26 20:27 UTC, Anatoly Litovsky
no flags Details
second succefull engine setup (1.50 MB, text/plain)
2014-11-26 20:27 UTC, Anatoly Litovsky
no flags Details
second succefull engine setup conf (1.54 KB, text/plain)
2014-11-26 20:28 UTC, Anatoly Litovsky
no flags Details
ovirt engine log from ceperate centos install) (22.11 KB, text/plain)
2014-11-27 06:53 UTC, Anatoly Litovsky
no flags Details
ovirt engine server log from seperate centos7 install (49.62 KB, text/plain)
2014-11-27 06:55 UTC, Anatoly Litovsky
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 35652 0 master MERGED packaging: Use correct Avalon Framework .jar files in EL7 Never
oVirt gerrit 35667 0 ovirt-engine-3.5 MERGED packaging: Use correct Avalon Framework .jar files in EL7 Never

Description Anatoly Litovsky 2014-11-26 20:21:46 UTC
Created attachment 961790 [details]
The jboss error

Description of problem:

When installing hosted engine on RHEVH3.5 over el7  with hosted engine based on Centos 7.

After running engine-setup on centos 7 , 
https://192.168.122.10/ovirt-engine/ works
https://192.168.122.10/ovirt-engine/api produces 500 error

Version-Release number of selected component (if applicable):

Ovirt node 3.5
Centos 7
ovirt-engine 3.5

How reproducible:

100% (3 out of 3)

Steps to Reproduce:

Names and definitions:

    <hosted engine ip> please choose free ip to hold the engine.
    <hosted engine iname> the DNS name that will be connected to the mentioned before ip

 
Prerequisites:

    RHEVH install image with hosted engine plugin.
    Centos 7 install image Download - CentOS Wiki
    NFS server
    HTTP server
    Be familiar with using SSH to connect the RHEVH
    Be familiar with the F2 option in RHEVH (Rescue shell)

 
Setting the RHEVH

    Install RHEVH , if installing on virtual machine make sure nested virtualization is enabled and cpu configuration is copied from the hosting machine
    Setup networking and security pages to allow ssh connection
    Login using ssh into the RHEVH machine - ip can be found in the status page.
    Create local DNS resolving for engine if you dont have real DNS server.
        edit /etc/hosts
        add the  <hosted eng ip> <hosted engine name>  (for example 192.168.122.12 my-engine.com )
    Or create real DNS entry in your DNS

 
Setting NFS server

    Install nfs server on the desired computer. (yum install nfs)
    Allow nfs , mountd , rpc-bind accss in you firewall
    start the nfs server and observe for errors.
    Using the /etc/exports file allow access for the network segment with the RHEVH and Engine will be located.
    systemctl start nfs
    in RHEVH console test the NFS server is available using showmount -e <nfs server ip>

 
Setting the http server

    Install the httpd server (  yum install httpd )
    Allow httpd in firewall.
    Upload the Centos7 iso into the httpd root directory.
    The httpd root directory can be found in /etc/httpd/conf/httpd.conf under DocumentRoot tag most likely (/var/www/html)
    systemctl start httpd
    in RHEVH console download some file from the httpd server using wget

 

 
Installing hosted engine in RHEVH

    In the hosted engine page provide the path to the iso located on the http server
    run the setup
    setup will fail on installation media
    re run setup
    using remote viewer login into <rhevh ip>:5900 , the password for the login is on the ssh console.

 

 
Setting up Centos 7 machine:

    Install the machine from iso.
    In the network tab please set the Ethernet connection , set the host name to <hosted engine ip> and the static ip to <hosted engine ip>
    reboot

 
In RHEVH setup console

    Select 1 (continue setup - VM installation is complete)
    Make new RHEVH ssh session.
        run
            hosted-engine --vm-start
            hosted-engine --add-console-password

 
Continue engine setup in Centos7 (The guest VM )

    Connect back to Centos machine like in previous step or by ssh connection
    Install net-tools
    and run ifconfig to verify <hosted engine ip>
    install ovirt repositories
        yum localinstall http://resources.ovirt.org/pub/yum-repo/ovirt-release35.rpm
        yum localinstall http://resources.ovirt.org/pub/yum-repo/ovirt-release35-snapshot.rpm
    yum install ovirt-engine
    run engine-setup as root when failed re run .
    From external pc please make sure you can login to the engine.


In RHEVH console with setup

Select 1 to complete install.


Actual results:
The hosted engine setup complains about 500 error , after some invetigation and puttin it the https://192.168.122.10/ovirt-engine/api 
Error 500 was observed all logs attached.


Expected results:
Expected the setup to finish succefully

Additional info:

Comment 1 Anatoly Litovsky 2014-11-26 20:22:36 UTC
Created attachment 961791 [details]
hosted engine setup log

Comment 2 Anatoly Litovsky 2014-11-26 20:23:16 UTC
Created attachment 961792 [details]
hosted engine setup answers

Comment 3 Anatoly Litovsky 2014-11-26 20:26:46 UTC
Created attachment 961803 [details]
first failed engine setup

Comment 4 Anatoly Litovsky 2014-11-26 20:27:16 UTC
Created attachment 961804 [details]
first fail engine setup conf

Comment 5 Anatoly Litovsky 2014-11-26 20:27:56 UTC
Created attachment 961805 [details]
second succefull engine setup

Comment 6 Anatoly Litovsky 2014-11-26 20:28:22 UTC
Created attachment 961806 [details]
second succefull engine setup conf

Comment 7 Douglas Schilling Landgraf 2014-11-26 20:57:47 UTC
Hi Anatoly,

Could you please provided the engine.log, server.log and others on /var/log/ovirt-engine/ ?

Thanks!

Comment 8 Anatoly Litovsky 2014-11-27 06:53:57 UTC
Created attachment 961889 [details]
ovirt engine log from ceperate centos install)

Comment 9 Anatoly Litovsky 2014-11-27 06:55:05 UTC
Created attachment 961890 [details]
ovirt engine server log from seperate centos7 install

Comment 10 Anatoly Litovsky 2014-11-27 06:56:48 UTC
The bug can also be reproduced by instaling ovirt-engine over plain centos7 and accesing it from http://<server>/ovirt-engine/api

If needed have one installed and can share it over net.

Comment 11 Juan Hernández 2014-11-27 10:35:16 UTC
This is the relevant error message from server.log:

Caused by: java.lang.ClassNotFoundException: org.apache.avalon.framework.configuration.ConfigurationException from [Module "org.apache.xmlgraphics.fop:main" from local module loader @516e9cbe (roots: /var/lib/ovirt-engine/jboss_runtime/modules/00-ovirt-engine-modules,/var/lib/ovirt-engine/jboss_runtime/modules/01-ovirt-engine-jboss-as-modules)]
	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
	at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
	at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
	... 31 more

It means that the Avalon Framework module isn't defined correctly. It isn't because in RHEL6 and CentOS 6 it contains a single .jar file, but in Fedora it contains two files. We tweaked the .spec to take care of this difference, but didn't take into account that this tweak doesn't work for RHEL7 or CentOS7.

To workaround the issue you can do the following:

  # cd /usr/share/ovirt-engine/modules/org/apache/avalon/framework/main
  # ln -sf /usr/share/java/avalon-framework-api.jar .
  # ln -sf /usr/share/java/avalon-framework-impl.jar .
  # systemctl restart ovirt-engine

Comment 12 Gonza 2015-01-06 00:02:37 UTC
Just to be sure, is this issue only occuring when installing a hosted engine on RHEVH3.5 over el7 with a hosted engine based on Centos 7? Has anyone tried other scenarios?

Regarding comment 10, is this also happening when installing the engine over el7?

Comment 13 Juan Hernández 2015-01-06 10:53:15 UTC
This happened when installing the engine on EL7 (RHEL7 or Centos7), no matter if it is hosted engine or plain vanilla engine.

Comment 14 Pavel Novotny 2015-01-12 14:39:38 UTC
To verify this bug properly also in RHEVM, we need RHEL7 support for it, which is planned in RHEVM 3.6, according to RFE bug 920126.

Juan, is it possible to re-target this bug also to 3.6? Because until then QA cannot verify it.

Comment 15 Juan Hernández 2015-01-12 14:42:17 UTC
This is a question for Sandro, but my understanding is that this must be targeted for 3.5.0 because we want EL7 support in ovirt-engine 3.5.1.

Comment 16 Sandro Bonazzola 2015-01-13 13:33:24 UTC
Just to clarify, oVirt is meant to support EL7 in 3.5.1 using ovirt-engine-jboss-as.

We're not supposed to verify this bug in RHEV-M 3.5. RFE bug 920126 is not yet approved so it may not make it for 3.6.

Comment 17 Gonza 2015-01-13 16:39:53 UTC
This has been verified upstream oVirt with verions:
ovirt-engine-3.5.1-0.0.master.20150107061804.git447ff70.el7.centos.noarch on rhel 7.1.

Comment 18 Pavel Novotny 2015-01-13 16:46:39 UTC
(In reply to Sandro Bonazzola from comment #16)
> Just to clarify, oVirt is meant to support EL7 in 3.5.1 using
> ovirt-engine-jboss-as.
> 
> We're not supposed to verify this bug in RHEV-M 3.5. RFE bug 920126 is not
> yet approved so it may not make it for 3.6.

Thanks for clarification. I am then removing depending bug 920126, because it's not relevant for oVirt.
And cloning this bug to RHEVM product, so we have tracker for further testing once bug 920126 is solved.


Note You need to log in before you can comment on or make changes to this bug.