We run into several cases from time to time where jobs are aborted. It would be useful if these could be automatically re-tried. I understand that power events are retried but, when that fails, a job has to be aborted. When a Beaker job fails (at least like that), it would be helpful if the scheduler could automatically try to use a different server. This should also take multihost jobs into account where only one server isn't responding. It seems like it would hopefully be unnecessary to retry the entire job but, just the one recipe that is failing to start.
It doesn't necessarily always make sense to retry a job which has aborted. A human needs to intervene to figure out what went wrong. Note that there is a beaker-jobwatch script floating around, which tries to automate that process. You could give that a go.
I don't know how beaker-jobwatch works but, I will look into that. Would you consider adding an option to allow a retry similar to how beaker-jobwatch does it? Maybe just something set in the job xml and with a maximum (or selectable number of retry attempts before it gives up completely?