Hide Forgot
Created attachment 1204515 [details] rhevm logs Description of problem: in 4.0.4-5 engine take ~4 minutes to come up and 2016-09-22 08:33:32,629 - MainThread - job_runner.libs.tasksmanager.install_engine - INFO - GET http://{engine-url}/ovirt-engine/services/health comes up Version-Release number of selected component (if applicable): rhevm-4.0.4.4-0.1.el7ev.noarch How reproducible: always Steps to Reproduce: 1. restart engine and try to access http://{engine-url}/ovirt-engine/services/health Actual results: takes ~4minutes Expected results: it was less than 2 minutes in older version Additional info:
Please specify which jboss eap version you are using.
JBoss EAP 7.0.2.GA (WildFly Core 2.1.8.Final-redhat-1)
In order to try and nail down the issue, can you test 4.0.3? Previous versions? When testing, make sure you run with a jboss version like the one used in the original release.
This may be related to the introduction of the concept of "implicit bean archives" in CDI 1.1, thus in Wildfly 10 and EAP7. Before CDI 1.1 the system only scanned the archives (the .jar or .war) files that contain a "beans.xml" file, but now, with CDI 1.1, it has to scan all archives, in order to see if they do contain any class that is annotated with an annotation that makes it a CDI bean. For us that means that Weld has now to scan not only our .jat files, but also the dependencies, like the Spring framework, the commons utils, etc. That may consume an important amount of resources, including time. According to the documentation that can be disabled: https://docs.jboss.org/author/display/WFLY8/CDI+Reference For us that would mean changing the ovirt-engine.xml.in file: diff --git a/packaging/services/ovirt-engine/ovirt-engine.xml.in b/packaging/services/ovirt-engine/ovirt-engine.xml.in index 8aa7741..342a429 100644 --- a/packaging/services/ovirt-engine/ovirt-engine.xml.in +++ b/packaging/services/ovirt-engine/ovirt-engine.xml.in @@ -536,7 +536,13 @@ </filters> </subsystem> - <subsystem xmlns="urn:jboss:domain:weld:3.0"/> + <subsystem xmlns="urn:jboss:domain:weld:3.0"> + <!-- Disable scanning of bean archives that don't contain a + beans.xml file, as that takes a long of time and all our + bean archives do contain that file. --> + <require-bean-descriptor>true</require-bean-descriptor> + </subsystem> + </profile> <interfaces> I didn't test it, but I think it is worth checking. Note that you can apply the change manually, to the /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.xml.in, and then restart the engine. Alternatively, if we really need to detect CDI beans available in dependencies, we could generate Jandex indexes during build time (like we do for our own modules) so that Weld doesn't need to scan them completely during startup.
I was able to configure that by replacing the weld line with: <subsystem xmlns="urn:jboss:domain:weld:3.0" require-bean-descriptor="true"/> I restarted the engine three times since, and it shows 20-25 seconds improvement. Tareq, can you test it as well, in addition to doing a sanity + automation testing to an engine configured with this setting?
Oved/Juan, Currently I am running Tier1 and Tier2 with adding the below line to /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.xml.in <property name="jboss.as.management.blocking.timeout" value="600"/> Tier1 ran to the end and engine is still up and running. Tier2 is still running OK until now.
Oved, are we blocking 4.0.4 on this bug? GA was planned for today.
I don't think it should block the release. Targeting it to 4.0.5.
Please later also test with what I've asked on comment #5, and report back. Thanks