Bug 1426745 - beakerd wastes time counting rows before fetching them
Summary: beakerd wastes time counting rows before fetching them
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Retired
Component: general
Version: 24
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: 24.2
Assignee: Dan Callaghan
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-24 18:05 UTC by Dan Callaghan
Modified: 2017-03-30 03:23 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-03-30 03:23:03 UTC
Embargoed:


Attachments (Terms of Use)

Description Dan Callaghan 2017-02-24 18:05:50 UTC
On a heavily loaded Beaker instance, like our production one, beakerd's scheduling routines can become a bottleneck. In particular the scheduler does a lot of work on the job and system tables, and is contending for locks with HTTP requests for in-progress jobs (recording results, etc).

To help reduce latency and eliminate lock contention as much as possible, beakerd should avoid doing any unnecessary queries. But currently each scheduling loop is structured in the form:

* count if there are rows that need attention
* if there is,
   * log a message to say I'm doing work
   * loop over the rows that need attention (this is a separate query)
       * do the work
   * log a message to say I'm done

We can eliminate the first count in favour of just iterating over the actual rows. There will be zero iterations of the loop if there is nothing to do.

One neat thing about the current approach is that it means the scheduler does not log anything if it is not doing any work, which means on a quiescent Beaker system the beakerd.log file stays quiet instead of filling with pointless spew. However I think we can just tweak the logging (specifically, ensure we log *inside* each loop iteration but not outside) to achieve roughly the same effect, while still avoiding the unnecessary COUNT() queries.

Comment 1 Dan Callaghan 2017-02-24 21:29:04 UTC
https://gerrit.beaker-project.org/5651

Comment 2 Roman Joost 2017-03-06 23:56:52 UTC
We'll have to do QE on the items before we can tag a release.

Comment 3 Roman Joost 2017-03-20 07:32:01 UTC
Tested on my own beaker installation (Beaker 24.2.git.9.d4c983d). Scheduled a few jobs and with reservations:

Mar 20 07:38:13 beaker-server-lc beakerd[32477]: bkr.server.tools.beakerd DEBUG Created watchdog for recipe id: 6 and system: beaker-test-vm2
Mar 20 07:38:13 beaker-server-lc beakerd[32477]: bkr.server.tools.beakerd INFO recipe ID 6 moved from Queued to Scheduled
Mar 20 07:38:13 beaker-server-lc beakerd[32477]: bkr.server.tools.beakerd DEBUG Selecting a system for recipe 7
Mar 20 07:38:13 beaker-server-lc beakerd[32477]: bkr.server.tools.beakerd DEBUG System : beaker-test-vm1 is available for Recipe 7
Mar 20 07:38:13 beaker-server-lc beakerd[32477]: bkr.server.model.scheduler DEBUG Reserving system beaker-test-vm1 for recipe 7
Mar 20 07:38:13 beaker-server-lc beakerd[32477]: bkr.server.model.activity DEBUG Tentative SystemActivity: object_id=1L, service=u'Scheduler', field=u'User', action=u'Reserved', old=u'', new=u'admin', user=admin
Mar 20 07:38:13 beaker-server-lc beakerd[32477]: bkr.server.model.inventory DEBUG Created reservation for system beaker-test-vm1 with type u'recipe', service u'Scheduler', user admin

Comment 4 Dan Callaghan 2017-03-30 03:23:03 UTC
Beaker 24.2 has been released.


Note You need to log in before you can comment on or make changes to this bug.