Bug 1374873 - ovirt-engine fails to start due to org.apache.jasper.JasperException: JBWEB004062: Unable to compile class for JSP
Summary: ovirt-engine fails to start due to org.apache.jasper.JasperException: JBWEB00...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.0.3
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ovirt-4.0.5
: ---
Assignee: Martin Perina
QA Contact: meital avital
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-09 22:36 UTC by Robert Scheck
Modified: 2019-12-26 22:20 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-04 08:04:16 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
/var/log/ovirt-engine/server.log (7.06 KB, application/x-xz)
2016-09-09 22:36 UTC, Robert Scheck 		no flags Details
View All

Description Robert Scheck 2016-09-09 22:36:59 UTC 
Created attachment 1199668 [details]
/var/log/ovirt-engine/server.log

Description of problem:
I tried to set up RHV 4 with RHVH according to the "Red Hat Virtualization
4.0 Self-Hosted Engine Guide" - nearly completely with standard values. At
the end, the ovirt-engine fails to start due to some Java errors. If you
don't dig into that, this looks like bug #1350194 - because you simply can
not log in from cockpit.

But if you look to the Apache webserver error_log, this becomes quite clear:

[Fri Sep 09 17:24:22.052867 2016] [proxy:error] [pid 1410] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed
[Fri Sep 09 17:24:22.052922 2016] [proxy:error] [pid 1410] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 5s
[Fri Sep 09 17:24:22.052937 2016] [proxy_ajp:error] [pid 1410] [client 192.168.0.100:39952] AH00896: failed to make connection to backend: 127.0.0.1
[Fri Sep 09 17:48:13.219739 2016] [proxy:error] [pid 1410] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed
[Fri Sep 09 17:48:13.219880 2016] [proxy:error] [pid 1410] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 5s
[Fri Sep 09 17:48:13.219909 2016] [proxy_ajp:error] [pid 1410] [client 192.168.0.100:42400] AH00896: failed to make connection to backend: 127.0.0.1
[Fri Sep 09 18:02:16.433276 2016] [proxy_ajp:error] [pid 3138] (104)Connection reset by peer: AH01030: ajp_ilink_receive() can't receive header

Ouch! It feels like ovirt-engine is not working properly...yes, might be
true given the zillion error related log lines; attaching simply the whole
/var/log/ovirt-engine/server.log file for the sake of completeness.

2016-09-09 17:20:06,455 INFO  [org.quartz.core.JobRunShell] (DefaultQuartzScheduler1) Job DEFAULT.org.ovirt.engine.core.vdsbroker.monitoring.PollAllVmStatsOnlyRefresher.poll#-9223372036854775741 threw a JobExecutionException: : org.quartz.JobExecutionException: failed to execute job
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_101]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_101]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101]
        at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]

2016-09-09 17:34:09,455 ERROR [io.undertow.request] (default task-15) UT005023: Exception handling request to /ovirt-engine/: org.apache.jasper.JasperException: JBWEB004062: Unable to compile class for JSP: 

JBWEB004061: An error occurred at line: 25 in the generated java file
The blank final field _jspx_imports_packages may not have been initialized

JBWEB004061: An error occurred at line: 27 in the generated java file
The blank final field _jspx_imports_classes may not have been initialized

JBWEB004061: An error occurred at line: 53 in the generated java file
This method must return a result of type JspContext

JBWEB004061: An error occurred at line: 57 in the generated java file
This method must return a result of type Map<String,Long>

JBWEB004061: An error occurred at line: 61 in the generated java file
This method must return a result of type Set<String>

JBWEB004061: An error occurred at line: 65 in the generated java file
This method must return a result of type Set<String>

JBWEB004061: An error occurred at line: 116 in the generated java file
This method must return a result of type boolean

JBWEB004061: An error occurred at line: 145 in the generated java file
This method must return a result of type boolean

JBWEB004061: An error occurred at line: 186 in the generated java file
This method must return a result of type boolean

JBWEB004061: An error occurred at line: 217 in the generated java file
This method must return a result of type boolean

Stacktrace:
        at org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:95) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:198) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:449) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:359) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:334) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:321) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:652) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.servlet.JspServletWrapper.loadTagFile(JspServletWrapper.java:242) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.TagFileProcessor.loadTagFile(TagFileProcessor.java:587) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.TagFileProcessor.access$000(TagFileProcessor.java:52) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.TagFileProcessor$TagFileLoaderVisitor.visit(TagFileProcessor.java:662) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Node$CustomTag.accept(Node.java:1535) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Node$Nodes.visit(Node.java:2375) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Node$Visitor.visitBody(Node.java:2427) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Node$Visitor.visit(Node.java:2433) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Node$Root.accept(Node.java:464) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Node$Nodes.visit(Node.java:2375) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.TagFileProcessor.loadTagFiles(TagFileProcessor.java:680) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:230) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:354) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:334) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:321) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:652) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:358) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402) [jastow.jar:2.0.0.Final-redhat-1]
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346) [jastow.jar:2.0.0.Final-redhat-1]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec.jar:1.0.0.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32) [jastow.jar:2.0.0.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:266) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToServlet(ServletInitialHandler.java:211) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.spec.RequestDispatcherImpl.includeImpl(RequestDispatcherImpl.java:352) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.spec.RequestDispatcherImpl.include(RequestDispatcherImpl.java:265) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at org.ovirt.engine.core.WelcomeServlet.doGet(WelcomeServlet.java:152) [welcome.jar:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [jboss-servlet-api_3.1_spec.jar:1.0.0.Final-redhat-1]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec.jar:1.0.0.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) [branding.jar:]
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:66) [utils.jar:]
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:]
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175) [undertow-servlet.jar:1.3.21.Final-redhat-1]
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [undertow-core.jar:1.3.21.Final-redhat-1]
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:792) [undertow-core.jar:1.3.21.Final-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]

Version-Release number of selected component (if applicable):
rhevm-appliance-20160831.0-1.x86_64.rhevm.ova

How reproducible:
Everytime, see above

Actual results:
ovirt-engine fails to start due to org.apache.jasper.JasperException: 
JBWEB004062: Unable to compile class for JSP

Expected results:
Working ovirt-engine ;-)

Additional info:
Not sure if RHV is really a product that shall be ever sold to a customer,
as RHV evaluations (pre-sale situations) are without any support - which is
quite challenging for potential customers without existing RHEV knowledge,
if things do not work at all according to the official documentation, that
everybody is pointing to. And yes, bugs of course exist - understandable...
Comment 1 Robert Scheck 2016-09-12 11:32:47 UTC 
Cross-filed case 01702003 on the Red Hat customer portal.
Comment 2 Robert Scheck 2016-09-12 15:22:31 UTC 
The ovirt-engine behaves quite strange: While the initial setup just fails
like in the initial description, a reboot of both, RHV-H and RHV-M, brings
afterwards (at least in some cases) the ovirt-engine up as it seems. Logs
are being uploaded to the case through. Nevertheless, ovirt-engine should
IMHO not fail during the setup with errors like in the initial description
ever.
Comment 3 Martin Perina 2016-09-14 18:41:04 UTC 
I'm trying to reproduce, but no luck so far. This seems to me like some permission issue (just a guess at the moment), but I don't see any connection with BZ1350194.
Comment 4 Martin Perina 2016-09-15 15:32:17 UTC 
I haven't been able to reproduce this error, doing installation exactly as described in [1] everything works as expected, no errors detected during installation and a few minutes after installation was successfully finished in Cockpit I was able to login to webadmin without any issues. Just for reference I've used following:

redhat-release-virtualization-host-4.0-4.el7.x86_64
rhevm-appliance-20160831.0-1.x86_64.rhevm.ova

Is this issue consistent in your environment? If so, could you please try to attach complete logs from engine VM and HE installation?


[1] https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/self-hosted-engine-guide/#Deploying_Self-Hosted_Engine_on_RHVH
Comment 5 Robert Scheck 2016-09-15 17:56:11 UTC 
(In reply to Martin Perina from comment #4)
> I haven't been able to reproduce this error, doing installation exactly as
> described in [1] everything works as expected, no errors detected during
> installation and a few minutes after installation was successfully finished
> in Cockpit I was able to login to webadmin without any issues. Just for
> reference I've used following:
> 
> redhat-release-virtualization-host-4.0-4.el7.x86_64
> rhevm-appliance-20160831.0-1.x86_64.rhevm.ova

The issue here is reproducible on bare metal while using iSCSI for RHV-M
and also with VMware player (with nested virtualization) while using NFSv3
for RHV-M. Second test setup happened to exclude possible iSCSI issues.

RHVH-4.0-20160822.8-RHVH-x86_64-dvd1.iso
rhevm-appliance-20160831.0-1.x86_64.rhevm.ova

> Is this issue consistent in your environment? If so, could you please try to
> attach complete logs from engine VM and HE installation?

Could you please express what that means in absolute paths on which system?

Note, the GSS ticket 01702003 has the ovirt-log-collector file attached (of
VMware player test setup).
Comment 6 Martin Perina 2016-09-16 14:56:48 UTC 
So far I still haven't been able to reproduce the issue, everything works as expected for me.
Here are some notes on issues found in provided logs:


> But if you look to the Apache webserver error_log, this becomes quite clear:
> 
> [Fri Sep 09 17:24:22.052867 2016] [proxy:error] [pid 1410] (111)Connection
> refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1)
> failed
> [Fri Sep 09 17:24:22.052922 2016] [proxy:error] [pid 1410] AH00959:
> ap_proxy_connect_backend disabling worker for (127.0.0.1) for 5s
> [Fri Sep 09 17:24:22.052937 2016] [proxy_ajp:error] [pid 1410] [client
> 192.168.0.100:39952] AH00896: failed to make connection to backend: 127.0.0.1

EAP shutdown was executed at 2016-09-09 17:21:30 and EAP startup finished successfully at 2016-09-09 17:24:47, so above logs in Apache are not relevant.


> [Fri Sep 09 17:48:13.219739 2016] [proxy:error] [pid 1410] (111)Connection
> refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1)
> failed
> [Fri Sep 09 17:48:13.219880 2016] [proxy:error] [pid 1410] AH00959:
> ap_proxy_connect_backend disabling worker for (127.0.0.1) for 5s
> [Fri Sep 09 17:48:13.219909 2016] [proxy_ajp:error] [pid 1410] [client
> 192.168.0.100:42400] AH00896: failed to make connection to backend: 127.0.0.1

Same as above, EAP shutdown was executed at 2016-09-09 17:48:07 and EAP startup finished successfully at 2016-09-09 17:48:39


> 
> Ouch! It feels like ovirt-engine is not working properly...yes, might be
> true given the zillion error related log lines; attaching simply the whole
> /var/log/ovirt-engine/server.log file for the sake of completeness.
> 
> 2016-09-09 17:20:06,455 INFO  [org.quartz.core.JobRunShell]
> (DefaultQuartzScheduler1) Job
> DEFAULT.org.ovirt.engine.core.vdsbroker.monitoring.
> PollAllVmStatsOnlyRefresher.poll#-9223372036854775741 threw a
> JobExecutionException: : org.quartz.JobExecutionException: failed to execute
> job

This exception is "normal", it happens always, most probable issue is that HE VM is not a proper VM for engine until storage domain is configured for default data center and VMs a properly imported into engine

> 
> 2016-09-09 17:34:09,455 ERROR [io.undertow.request] (default task-15)
> UT005023: Exception handling request to /ovirt-engine/:
> org.apache.jasper.JasperException: JBWEB004062: Unable to compile class for
> JSP: 
> 
> JBWEB004061: An error occurred at line: 25 in the generated java file
> The blank final field _jspx_imports_packages may not have been initialized

I haven't been able to reproduce above errors, so no idea what caused this issue ...



Anyway I found out several interesting issues in logs:

1. DNS resolution of engine FQDN
  - I have found several errors in HA setup logs like this:

     2016-09-09 23:54:27 ERROR otopi.plugins.gr_he_common.vm.cloud_init dialog.queryEnvKey:120 Host name is not valid: rhev-mgmt.fritz.box did not resolve into an IP address

  - At the end you somehow fixed the issue:
     2016-09-09 23:58:36 DEBUG otopi.plugins.gr_he_common.vm.cloud_init hostname._validateFQDNresolvability:245 rhev-mgmt.fritz.box resolves to: set(['192.168.0.102'])

  - But here's the question: proper DNS setup for engine FQDN is crucial to access 4.0 engine, so are you sure that engine FQDN is fully resolvable on engine hosts and all clients that accesses engine?



2. Accessing engine by different name than engine FQDN
  - I have noticed following error in engine.log

    2016-09-09 17:34:08,253 ERROR [org.ovirt.engine.core.sso.utils.SsoUtils] (default task-9) [] The client is not authorized to request an authorization. It's required to access the system using FQDN.

  - It seems that engine was accessed by different name or IP address, which raised the error.
  - Aside from this attempt I have not found any other access to either webadmin or userportal in the logs, only API access most probably from HE setup
Comment 7 Robert Scheck 2016-09-16 15:13:57 UTC 
(In reply to Martin Perina from comment #6)
> > 2016-09-09 17:34:09,455 ERROR [io.undertow.request] (default task-15)
> > UT005023: Exception handling request to /ovirt-engine/:
> > org.apache.jasper.JasperException: JBWEB004062: Unable to compile class for
> > JSP: 
> > 
> > JBWEB004061: An error occurred at line: 25 in the generated java file
> > The blank final field _jspx_imports_packages may not have been initialized
> 
> I haven't been able to reproduce above errors, so no idea what caused this
> issue ...

But that error I guess is the reason why the ovirt-engine does not deliver
anything to the Apache reverse proxy...could that be caused due to some DNS
failures?

> Anyway I found out several interesting issues in logs:
> 
> 1. DNS resolution of engine FQDN
>   - I have found several errors in HA setup logs like this:
> 
>      2016-09-09 23:54:27 ERROR otopi.plugins.gr_he_common.vm.cloud_init
> dialog.queryEnvKey:120 Host name is not valid: rhev-mgmt.fritz.box did not
> resolve into an IP address
> 
>   - At the end you somehow fixed the issue:
>      2016-09-09 23:58:36 DEBUG otopi.plugins.gr_he_common.vm.cloud_init
> hostname._validateFQDNresolvability:245 rhev-mgmt.fritz.box resolves to:
> set(['192.168.0.102'])
> 
>   - But here's the question: proper DNS setup for engine FQDN is crucial to
> access 4.0 engine, so are you sure that engine FQDN is fully resolvable on
> engine hosts and all clients that accesses engine?

The DNS was definately set up and even explicitly tested before and worked
properly on any random client (RHVH, other Linux, Windows client). Honestly
I would more guess a race condition during booting the RHVM VM, given some
reboots lead to a working result (but no DHCP, but static IP addresses are
everywhere in use).

> 2. Accessing engine by different name than engine FQDN
>   - I have noticed following error in engine.log
> 
>     2016-09-09 17:34:08,253 ERROR [org.ovirt.engine.core.sso.utils.SsoUtils]
> (default task-9) [] The client is not authorized to request an
> authorization. It's required to access the system using FQDN.
> 
>   - It seems that engine was accessed by different name or IP address, which
> raised the error.
>   - Aside from this attempt I have not found any other access to either
> webadmin or userportal in the logs, only API access most probably from HE
> setup

Well, that results from some tests, because the HTTPS connection did not
return anything, but was kept open (due to the Apache not getting any reply
from ovirt-engine). Given the browser keeps spinning the wheel, I used some
command line clients in the hope to get more results through.
Comment 8 Martin Perina 2016-09-16 18:13:27 UTC 
Robert, would it be possible to attach logs also from the 2nd attempt? At the moment I'm out of ideas what could go wrong in your env.
Comment 9 Martin Perina 2016-09-16 18:15:00 UTC 
Simone, could you please take a look at logs from HE point of view? Maybe you will find anything that I missed ...
Comment 10 Robert Scheck 2016-09-19 18:48:24 UTC 
(In reply to Martin Perina from comment #8)
> Robert, would it be possible to attach logs also from the 2nd attempt? At
> the moment I'm out of ideas what could go wrong in your env.

If you mean the multiple bare metal installations before, then I have to 
disappoint you, these test systems have been reinstalled meanwhile, sorry.
Comment 11 Simone Tiraboschi 2016-09-21 16:07:03 UTC 
(In reply to Martin Perina from comment #9)
> Simone, could you please take a look at logs from HE point of view? Maybe
> you will find anything that I missed ...

Hosted-engine-setup seams absolutely fine.
Comment 12 Jiri Belka 2016-09-23 14:55:55 UTC 
It works fine with (tested on real bare metal host with NFS):

RHVH-4.0-20160907.4-RHVH-x86_64-dvd1.iso
rhevm-appliance-20160831.0-1.x86_64.rhevm.ova

And this is what is downloadable from RH customer portal as recent versions.
Comment 13 Robert Scheck 2016-09-23 16:10:02 UTC 
How can I, as a non-developer (or even as a customer), provide valuable 
information when running into this issue? Just running ovirt-log-collector
doesn't seem to do the trick and these JSP errors are a mystery, too? The
RHV systems are unfortunately not small enough to freeze them like a VM in
more common situations (not sure where and how I can provide these 100+ GB
images through).

From my point of view, if the ovirt-engine fails, there must be meaningful
errors and logs somewhere afterwards.
Comment 14 Fabian Deutsch 2016-09-23 17:39:18 UTC 
Robert, could you give the RHVH + RHEVM Appliance combination in comment 12 a try, to see if this is working in your environment?
Comment 15 Martin Perina 2016-10-04 08:04:16 UTC 
We were not able to reproduce the issue nor find any clues why this issue happened in customer's environment. I'm closing this bug as WORKSFORME, but feel free to reopen it when this issue happens again with RHVH and RHVM Appliance versions mentioned in Comment 12 or later.
Note You need to log in before you can comment on or make changes to this bug.