Description of problem: Java EE have ready made implementation for scheduling tasks and threads. Today we use Quartz for scheduled tasks with a custom thread pool exclusivity for it and also few internal thread pools for one off tasks. All this can be replaced with EE builtins ManagedScheduledExecutorService and ManagedExecutorService. Also EJB timers can be an option. Those have few advantages: 1. configurable in runtime by existing tools - jmx/ management console on port 8706 using cli and web 2. proven, supported for long time 3. less code, we don't need the engine Scheduler module and its glue-code @OnTimer annotation and the annoying builerplate 4. Cleaner api - which is also pretty similar to quarz (the verbs are almost the same, but more consize as it lets you schedule a Runnable hence lambda makes it a beauty) 5. compatibility - the api lets you schedule with rate, delay, cancel and cancel job. ManagedExecutorService is a superset of our ThreadPoolUtils 6. Supports injection - easy to code and test TBD - Gluster services in engine is using db to store quartz triggers. This means this change is probably irrelevant for gluster services and this means we won't tottaly remove the quartz depedency
continued description: Scope: 1. Thread pools - Eliminate our own ThreadPoolUtil, eliminate redundant pools, leave only those who need bounded resources (or create managed ones that answer the need - e.g HostUpdateScheduerService) 2. Scheduled tasks - Replace quartz where we can (gluster needs more info), pass criteria: - Engine schedules threads and tasks with no regression. Same delay between jobs, jobs fired with the same rate. - No performance degradation.
CodeChange? if not, please share reprosteps, thx, P.
adding needinfo as per comment#2
CodeChange
per CodeChange moving to VERIFIED
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017. Since the problem described in this bug report should be resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.
Cutting down the doc text to appropriate release note size, so I'll include the rest of the text here: Use JBoss ManagedThreadFactory, ManagedExecutorService and ManagedScheduledExecutorService instead of Java EE ExecutorService and Quartz to schedule jobs. There are two ManagedExecutorServices in engine commandCoordinator and hostUpdatesChecker. The size of the commandCoordinator executor service is defined by config variable COMMAND_COORDINATOR_THREAD_POOL_SIZE. The default size if 10. To change the size create a new conf file 99-coco-thread-pool.conf in /etc/ovirt-engine/engine.conf.d/ The size of the hostUpdatesChecker executor service is defined by config variable HOST_CHECK_FOR_UPDATES_THREAD_POOL_SIZE. The default size if 5. To change the size create a new conf file 99-host-thread-pool.conf in /etc/ovirt-engine/engine.conf.d/ Management clients, such as the WildFly CLI, can also be used to configure ManagedExecutorService instances commandCoordinator and hostUpdatesChecker with out having to restart engine. The 'engine' executor service size is define by config variables ENGINE_THREAD_POOL_MIN_SIZE and ENGINE_THREAD_POOL_MAX_SIZE. The default value for ENGINE_THREAD_POOL_MIN_SIZE is 50 and default value for ENGINE_THREAD_POOL_MAX_SIZE is 500. To change the value permanentaly create a conf file 99-engine-thread-pool.conf in /etc/ovirt-engine/engine.conf.d/. jconsole can be used to change the size of the 'engine' executor service with out having to restart engine. 'engineScheduled' ManagedScheduledExecutorService size is define by config variable ENGINE_SCHEDULED_THREAD_POOL_SIZE. The default value is 100. To change the value permanentaly create a conf file 99-engine-scheduled-thread-pool.conf in /etc/ovirt-engine/engine.conf.d/ Management clients, such as the WildFly CLI, can also be used to configure ManagedScheduledExecutorService instance 'engineScheduled' with out having to restart engine. Engine creates a log entry of the thread pool usage as define by the config variable THREAD_POOL_MONITORING_INTERVAL_IN_SECONDS. The default value is 600 seconds. So every 10 minutes a log entry is created for each pool showing the thread pool usage. Thread pool 'engine' is using 7 threads out of 500, 1 threads waiting for tasks and 0 tasks in queue. Thread pool 'engineScheduled' is using 0 threads out of 100 and 100 tasks are waiting in the queue. Thread pool 'engineThreadMonitoring' is using 1 threads out of 1 and 0 tasks are waiting in the queue.