Bug 1262990

Summary: [bug] watchdog can be pushed out after a panic is detected
Product: [Retired] Beaker Reporter: Bill Peck <bpeck>
Component: lab controllerAssignee: beaker-dev-list
Status: CLOSED WONTFIX QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: jburke, mjia, pcahyna
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-21 14:13:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bill Peck 2015-09-14 20:19:22 UTC
Description of problem:
When a panic is detencted by the lab controller it sets the current watchdog to ten minutes from the current time to give the system a chance to kdump if configured.

The problem is if kdump default configuration is to reboot the system.  This means its possible that the system under test could push the watchdog out again and attempt to run the test and possibly panic again.

Version-Release number of selected component (if applicable):
20.2

How reproducible:
Sun, moon and stars need to align

Actual results:
We had a bad kernel go through beaker which ended up filling the netdump server disk and kept the systems busy until someone manually canceled the job.

Several things had to go wrong.
1. bad kernel that caused a panic
2. bug in kdump that kept the system from halting
3. fsck? rolled changes back in filesystem which made the harness think it was running for the first time (rebootcount==0)

Expected results:
Once initial 10 minute watchdog timeout is set because of a panic it should not be changeable.

Additional info:
There is some other code in the abort code which looks like it can modify the watchdog time as well if WATCHDOG_SCRIPT is defined. So this could be tricky.

Comment 1 Pavel Cahyna 2019-07-18 16:15:56 UTC
Bill Peck: how will one be able to collect the vmcore generated by kdump? Ten minutes is not enough.