Bug 1302942 - [RFE] beaker-init background mode with watcher
Summary: [RFE] beaker-init background mode with watcher
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Retired
Component: general
Version: 22
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: 23.0
Assignee: Dan Callaghan
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-29 05:09 UTC by Dan Callaghan
Modified: 2016-07-07 23:10 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-07 23:10:32 UTC
Embargoed:


Attachments (Terms of Use)

Description Dan Callaghan 2016-01-29 05:09:41 UTC
The beaker-init command performs our database upgrades. Because MySQL lacks transactional DDL, the beaker-init process is quite fragile: once it's started, it really needs to run to completion without being interrupted (for example by SIGHUPs from an SSH connection dropping) otherwise the database can be left in an inconsistent state and needs manual recovery, inspecting the database state by hand and fixing up any incomplete migration.

The migration process can also potentially take a very long time (measured in hours) if the Beaker deployment is very old and the migrations are performing ALTER TABLE statements on large tables. This is another inherent limitation of InnoDB, which has to write out a new copy of the entire table to disk during ALTER TABLE.

To deal with this fragility our Ansible playbooks for internal deployment currently have a manual step where the admin is required to run beaker-init in a screen session and monitor it for completion.

A nicer solution would be if beaker-init could detach itself from the terminal and also provide some mechanism to watch a running beaker-init and determine when it has finished. Thus the playbook can be restructured to fire off beaker-init in the background and then poll for its completion. That is more robust and not susceptible to SSH connection dropouts or ^C'ing a running playbook.

Comment 1 Dan Callaghan 2016-01-29 05:11:26 UTC
Suggested implementation:

New option --background, causes beaker-init to detach itself from the terminal before performing any database operations.

New option --check-update, which skips all database operations and instead just checks the existing version of the database. Exit with status 0 if no migrations are needed, or 1 if some migrations are needed (or if the database needs to be populated).

Comment 2 Dan Callaghan 2016-02-04 00:17:34 UTC
The --check-update option is probably useful but what we really need here is for the backgrounded beaker-init to write a pid file, and then a way to check if that pid file exists (and the process is still alive). Just running --check-update in a polling loop is not a good approach because you would never notice if the backgrounded beaker-init had crashed for any reason.

Comment 3 Dan Callaghan 2016-02-04 04:10:24 UTC
A better name for the option would be --check since it needs to work with --downgrade <xyz> as well.

So the workflow can be:

beaker-init --background
(wait for /var/run/beaker-init.pid to no longer exist)
beaker-init --check

If the final command returns 0 then everything is good. Otherwise it means the backgrounded beaker-init failed for some reason.

Ansible comes with an implementation of waiting for a file to be absent, so I don't think we need to put anything into beaker-init itself to support that.

Comment 7 Dan Callaghan 2016-07-07 23:10:32 UTC
Beaker 23.0 has been released.


Note You need to log in before you can comment on or make changes to this bug.