Currently, full Beaker admins can cancel arbitrary jobs and recipes. However, system owners cannot cancel recipes running on their systems.
Now that job activity tracking is working, this restriction should be lifted. A "cancel-recipe" permission should also be added. (Open question: aside from the system owner, which other existing users, if any, should get this permission granted automatically as part of the database migration)
As part of this change, some guidance should be added to the documentation on choosing between whether or not to have the system in Automated or Manual mode, as well as on when its appropriate to cancelling running recipes.
= Choosing between Manual and Automated mode =
Systems in Manual mode differ from systems in Automated mode in a two key ways:
* Systems in Manual mode can only be accessed through the scheduler by using the "forced scheduling" feature. This means that the system will only ever be accessed by name, and not by satisfying a "hostRequires" filter. This makes Manual mode appropriate when there are additional constraints on the system that potential users should explicitly opt into.
* Systems in Manual mode can always be "taken" indefinitely. This allows Beaker to be used for provisioning, but otherwise leave the system alone. No additional software will be implicitly installed, and no log files are automatically stored. When used this way, Beaker is just being used as an operating system library (including its boot menu support). (Automated systems can also be used this way, but only when an explicit loan is in place, and thus system access is already limited to that specific user for the duration of the loan)
= Cancelling running recipes =
As a general guideline, a system owner or Beaker administrator cancelling the running recipe of another user should be an exceptional event. Situations where this would be appropriate in general would be if the recipe is harming the Beaker installation in some way, such as flooding the lab network with traffic, uploading unreasonably large numbers of log files or producing excessive amounts of output on the console log.
If a system owner owner reserves the right to preempt arbitrary recipes (for example, if the system is one that is sometimes needed urgently for specific use cases), then it is more appropriate to either leave it in Manual mode, or create a group specifically for users that are happy to accept the risk of having their recipes cancelled unexpectedly, rather than add it to the general access pool in Automated mode.
Note that even the machine condition cannot be changed (via "bkr system-modify") while a job is running on the system. This means a machine owner cannot even schedule a downtime (temporarily mark the machine as Broken) if any job from any user is running.
This needs me to get maliciously creative like blocking Beaker's labcontroller on the firewall, automatically marking the machine as Broken over time or faking my identity for the labctl and finishing tasks on behalf of the user (I can do that since I can effectively do MITM).
Just pointing out that this should not be a low priority RFE for the sake of transparency between the user and the system owner.
(In reply to Jiri Jaburek from comment #1)
> Note that even the machine condition cannot be changed (via "bkr
> system-modify") while a job is running on the system. This means a machine
> owner cannot even schedule a downtime (temporarily mark the machine as
> Broken) if any job from any user is running.
> This needs me to get maliciously creative like blocking Beaker's
> labcontroller on the firewall, automatically marking the machine as Broken
> over time or faking my identity for the labctl and finishing tasks on behalf
> of the user (I can do that since I can effectively do MITM).
> Just pointing out that this should not be a low priority RFE for the sake of
> transparency between the user and the system owner.
Actually, that was just me hitting bug 1268811. Regardless, if a job takes too long (days/weeks), the system owner has still no way of cancelling it.
Issues like this should be handled by contacting the Admin of beaker instance (labs,sysops). Owner of the machine agreed with specific terms when he added the machine to the pool. There is always an option of reaching out to the user. Feature like this should not be supported.