Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1700926

Summary:	[RFE] Allow task to adjust local watchdog
Product:	[Retired] Restraint	Reporter:	Jeff Bastian <jbastian>
Component:	general	Assignee:	Carol Bouchard <cbouchar>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	low	Docs Contact:
Priority:	low
Version:	0.1.39	CC:	asavkov, bpeck, breilly, cbeer, cbouchar, mastyk, tklohna
Target Milestone:	0.1.40	Keywords:	FutureFeature, Triaged
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1753336 (view as bug list)		Environment:
Last Closed:	2019-07-17 19:36:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1753336

Description Jeff Bastian 2019-04-17 16:43:11 UTC

Description of problem:
The rstrnt-adjust-watchdog command can adjust the External Watchdog on the fly, but nothing can change the Local Watchdog once a task has started running. Please extend the command (or provide another mechanism) for a task to extend its own time if necessary.

It is possible to currently achieve a similar effect with no_localwatchdog=true in the task metadata and rely solely on the External Watchdog, but this is not ideal. The Local Watchdog will provide valuable debugging information via Magic-SysRq if it expires, whereas the External Watchdog simply aborts.

Some use-cases:

* allow a single task to run a "lite" or "heavy" version based on a task param.
The task could tell restraint to set the LWD to 30 minutes if running the lite
version, or 60 minutes for the heavy version. (The RSTRNT_MAX_TIME param can
also change the timeout, but it would be nice to not have to set two parameters
which is error prone.)

* allow a task to check the CPU type and count, e.g., Intel Xeon with 32 CPUs
vs. a single-core single-thread Intel Atom, and adjust the LWD based on how
fast the CPU(s) should be able to run the task.

* allow a task to simply request more time if it's making progress but not as
quickly as expected. This can happen on a virtual machine running on a host
where another VM is using most/all of the hosts' I/O and the other VMs are
nearly starving for resources. Tasks usually abort when this happens and it
is frustrating to check the logs and find out it was 90% done and would have
finished if just given a few more minutes. Also, some very large systems
take a very long time to POST (power on self test) while they diagnostics on
their terabytes of RAM and dozens of attached SAN LUNs. A simple reboot can
make a task take much longer than expected.

Version-Release number of selected component (if applicable):
restraint-0.1.39

Comment 1 Carol Bouchard 2019-05-10 12:16:39 UTC

This is what we've discussed and I'm wondering if this works for you.
Instead of having rstrt_adjust_watchdog adjust only the external_watchdog,
it will adjust the local watchdog at the same time.  So the results is
both watchdog timers will be affected by this CLI instead of just
the external_watchdog.

Comment 2 Jeff Bastian 2019-05-10 18:06:27 UTC

That would be perfect, and a lot easier than having to remember to run two different commands.

The EWD is usually a bit longer than LWD, if I recall correctly.  It has an extra 15 minute buffer I think?  I assume rstrnt_adjust_watchdog would factor this in?

Comment 3 Bill Peck 2019-05-10 18:12:09 UTC

You're right Jeff,

current call in restraint/task.c:
          restraint_task_watchdog (task, app_data, task->remaining_time + EWD_TIME);

and EWD_TIME is defined as: 
#define EWD_TIME 30 * 60 // amount of time to add to local watchdog for externl watchdog

That will need to factored into this patch as well.

Comment 4 Carol Bouchard 2019-05-10 19:02:20 UTC

Thanks for the heads up.  I'll investigate the affect on the external watchdog next as I didn't expect there to be any changes since that functionality already exists.  I'm currently trying to shim in changes for local
timers as there are 2 of them and I'd merely add to them.  Believe it or not, one timer's a bit tricky and I want to work with the harmony of existing code.

Comment 5 Carol Bouchard 2019-05-15 14:21:01 UTC

I've been spending time testing code without my code changes. When rstrnt_adjust_watchdog is called, I noticed the EWD_TIME is currently not added to the external watchdog.  I will add it to external watchdog with this change.

Comment 6 Martin Styk 2019-09-10 05:57:59 UTC

Restraint 0.1.40 has been released.