Red Hat Bugzilla – Bug 1302942
[RFE] beaker-init background mode with watcher
Last modified: 2016-07-07 19:10:32 EDT
The beaker-init command performs our database upgrades. Because MySQL lacks transactional DDL, the beaker-init process is quite fragile: once it's started, it really needs to run to completion without being interrupted (for example by SIGHUPs from an SSH connection dropping) otherwise the database can be left in an inconsistent state and needs manual recovery, inspecting the database state by hand and fixing up any incomplete migration.
The migration process can also potentially take a very long time (measured in hours) if the Beaker deployment is very old and the migrations are performing ALTER TABLE statements on large tables. This is another inherent limitation of InnoDB, which has to write out a new copy of the entire table to disk during ALTER TABLE.
To deal with this fragility our Ansible playbooks for internal deployment currently have a manual step where the admin is required to run beaker-init in a screen session and monitor it for completion.
A nicer solution would be if beaker-init could detach itself from the terminal and also provide some mechanism to watch a running beaker-init and determine when it has finished. Thus the playbook can be restructured to fire off beaker-init in the background and then poll for its completion. That is more robust and not susceptible to SSH connection dropouts or ^C'ing a running playbook.
New option --background, causes beaker-init to detach itself from the terminal before performing any database operations.
New option --check-update, which skips all database operations and instead just checks the existing version of the database. Exit with status 0 if no migrations are needed, or 1 if some migrations are needed (or if the database needs to be populated).
The --check-update option is probably useful but what we really need here is for the backgrounded beaker-init to write a pid file, and then a way to check if that pid file exists (and the process is still alive). Just running --check-update in a polling loop is not a good approach because you would never notice if the backgrounded beaker-init had crashed for any reason.
A better name for the option would be --check since it needs to work with --downgrade <xyz> as well.
So the workflow can be:
(wait for /var/run/beaker-init.pid to no longer exist)
If the final command returns 0 then everything is good. Otherwise it means the backgrounded beaker-init failed for some reason.
Ansible comes with an implementation of waiting for a file to be absent, so I don't think we need to put anything into beaker-init itself to support that.
Beaker 23.0 has been released.