Bug 1379127

Summary: Engine service takes longer to come up
Product: [oVirt] ovirt-engine Reporter: Tareq Alayan <talayan>
Component: BLL.InfraAssignee: Oved Ourfali <oourfali>
Status: CLOSED NOTABUG QA Contact: Pavel Stehlik <pstehlik>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.0.4.4CC: alukiano, bugs, gklein, juan.hernandez, mperina, ncredi, oourfali, talayan
Target Milestone: ovirt-4.0.6Keywords: Automation, AutomationBlocker, Regression
Target Release: ---Flags: gklein: ovirt-4.0.z?
gklein: blocker?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-26 10:46:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
rhevm logs none

Description Tareq Alayan 2016-09-25 08:44:37 UTC 
Created attachment 1204515 [details]
rhevm logs

Description of problem:
in 4.0.4-5 engine take ~4 minutes to come up and 
2016-09-22 08:33:32,629 - MainThread - job_runner.libs.tasksmanager.install_engine - INFO - GET http://{engine-url}/ovirt-engine/services/health

comes up

Version-Release number of selected component (if applicable):
rhevm-4.0.4.4-0.1.el7ev.noarch

How reproducible:
always

Steps to Reproduce:
1. restart engine and try to access http://{engine-url}/ovirt-engine/services/health



Actual results:
takes ~4minutes 

Expected results:
it was less than 2 minutes in older version 

Additional info:
Comment 1 Yaniv Kaul 2016-09-25 09:06:50 UTC 
Please specify which jboss eap version you are using.
Comment 2 Tareq Alayan 2016-09-25 09:23:56 UTC 
JBoss EAP 7.0.2.GA (WildFly Core 2.1.8.Final-redhat-1)
Comment 3 Oved Ourfali 2016-09-25 10:15:46 UTC 
In order to try and nail down the issue, can you test 4.0.3?
Previous versions?
When testing, make sure you run with a jboss version like the one used in the original release.
Comment 4 Juan Hernández 2016-09-25 18:49:47 UTC 
This may be related to the introduction of the concept of "implicit bean archives" in CDI 1.1, thus in Wildfly 10 and EAP7. Before CDI 1.1 the system only scanned the archives (the .jar or .war) files that contain a "beans.xml" file, but now, with CDI 1.1, it has to scan all archives, in order to see if they do contain any class that is annotated with an annotation that makes it a CDI bean. For us that means that Weld has now to scan not only our .jat files, but also the dependencies, like the Spring framework, the commons utils, etc. That may consume an important amount of resources, including time. According to the documentation that can be disabled:

  https://docs.jboss.org/author/display/WFLY8/CDI+Reference

For us that would mean changing the ovirt-engine.xml.in file:

diff --git a/packaging/services/ovirt-engine/ovirt-engine.xml.in b/packaging/services/ovirt-engine/ovirt-engine.xml.in
index 8aa7741..342a429 100644
--- a/packaging/services/ovirt-engine/ovirt-engine.xml.in
+++ b/packaging/services/ovirt-engine/ovirt-engine.xml.in
@@ -536,7 +536,13 @@
       </filters>
     </subsystem>
 
-    <subsystem xmlns="urn:jboss:domain:weld:3.0"/>
+    <subsystem xmlns="urn:jboss:domain:weld:3.0">
+      <!-- Disable scanning of bean archives that don't contain a
+           beans.xml file, as that takes a long of time and all our
+           bean archives do contain that file. -->
+      <require-bean-descriptor>true</require-bean-descriptor>
+    </subsystem>
+
   </profile>
 
   <interfaces>

I didn't test it, but I think it is worth checking.

Note that you can apply the change manually, to the /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.xml.in, and then restart the engine.

Alternatively, if we really need to detect CDI beans available in dependencies, we could generate Jandex indexes during build time (like we do for our own modules) so that Weld doesn't need to scan them completely during startup.
Comment 5 Oved Ourfali 2016-09-26 06:08:48 UTC 
I was able to configure that by replacing the weld line with:

<subsystem xmlns="urn:jboss:domain:weld:3.0" require-bean-descriptor="true"/>

I restarted the engine three times since, and it shows 20-25 seconds improvement.

Tareq, can you test it as well, in addition to doing a sanity + automation testing to an engine configured with this setting?
Comment 6 Tareq Alayan 2016-09-26 08:19:37 UTC 
Oved/Juan, 

Currently I am running Tier1 and Tier2 with adding the below line to /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.xml.in 

  <property name="jboss.as.management.blocking.timeout" value="600"/>


Tier1 ran to the end and engine is still up and running. 
Tier2 is still running OK until now.
Comment 7 Sandro Bonazzola 2016-09-26 09:34:17 UTC 
Oved, are we blocking 4.0.4 on this bug? GA was planned for today.
Comment 8 Oved Ourfali 2016-09-26 09:58:14 UTC 
I don't think it should block the release.
Targeting it to 4.0.5.
Comment 9 Oved Ourfali 2016-09-26 09:58:57 UTC 
Please later also test with what I've asked on comment #5, and report back.

Thanks