Bug 1302942

Summary: [RFE] beaker-init background mode with watcher
Product: [Retired] Beaker Reporter: Dan Callaghan <dcallagh>
Component: generalAssignee: Dan Callaghan <dcallagh>
Status: CLOSED CURRENTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: dcallagh, dowang, mjia, rjoost
Target Milestone: 23.0Keywords: FutureFeature, Patch
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-07 23:10:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan Callaghan 2016-01-29 05:09:41 UTC 
The beaker-init command performs our database upgrades. Because MySQL lacks transactional DDL, the beaker-init process is quite fragile: once it's started, it really needs to run to completion without being interrupted (for example by SIGHUPs from an SSH connection dropping) otherwise the database can be left in an inconsistent state and needs manual recovery, inspecting the database state by hand and fixing up any incomplete migration.

The migration process can also potentially take a very long time (measured in hours) if the Beaker deployment is very old and the migrations are performing ALTER TABLE statements on large tables. This is another inherent limitation of InnoDB, which has to write out a new copy of the entire table to disk during ALTER TABLE.

To deal with this fragility our Ansible playbooks for internal deployment currently have a manual step where the admin is required to run beaker-init in a screen session and monitor it for completion.

A nicer solution would be if beaker-init could detach itself from the terminal and also provide some mechanism to watch a running beaker-init and determine when it has finished. Thus the playbook can be restructured to fire off beaker-init in the background and then poll for its completion. That is more robust and not susceptible to SSH connection dropouts or ^C'ing a running playbook.
Comment 1 Dan Callaghan 2016-01-29 05:11:26 UTC 
Suggested implementation:

New option --background, causes beaker-init to detach itself from the terminal before performing any database operations.

New option --check-update, which skips all database operations and instead just checks the existing version of the database. Exit with status 0 if no migrations are needed, or 1 if some migrations are needed (or if the database needs to be populated).
Comment 2 Dan Callaghan 2016-02-04 00:17:34 UTC 
The --check-update option is probably useful but what we really need here is for the backgrounded beaker-init to write a pid file, and then a way to check if that pid file exists (and the process is still alive). Just running --check-update in a polling loop is not a good approach because you would never notice if the backgrounded beaker-init had crashed for any reason.
Comment 3 Dan Callaghan 2016-02-04 04:10:24 UTC 
A better name for the option would be --check since it needs to work with --downgrade <xyz> as well.

So the workflow can be:

beaker-init --background
(wait for /var/run/beaker-init.pid to no longer exist)
beaker-init --check

If the final command returns 0 then everything is good. Otherwise it means the backgrounded beaker-init failed for some reason.

Ansible comes with an implementation of waiting for a file to be absent, so I don't think we need to put anything into beaker-init itself to support that.
Comment 4 Dan Callaghan 2016-02-04 04:14:59 UTC 
http://gerrit.beaker-project.org/4658
http://gerrit.beaker-project.org/4659
Comment 7 Dan Callaghan 2016-07-07 23:10:32 UTC 
Beaker 23.0 has been released.