Bug 975486 - beaker-watchdog fails to detect a panic if it crosses block boundaries
beaker-watchdog fails to detect a panic if it crosses block boundaries
Status: CLOSED CURRENTRELEASE
Product: Beaker
Classification: Community
Component: lab controller (Show other bugs)
0.12
Unspecified Unspecified
unspecified Severity unspecified (vote)
: 0.15.3
: ---
Assigned To: Dan Callaghan
tools-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-18 10:55 EDT by Gurhan Ozen
Modified: 2014-02-02 23:51 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-02 23:51:48 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 1 Dan Callaghan 2013-06-19 23:44:29 EDT
In this case beaker-watchdog was definitely reading the console log correctly (because it was uploading it to the job). But there is no log line "Panic detected for..." in watchdog.log, so beaker-watchdog never noticed the line "Kernel panic" in the console log.

One thing I noticed is that beaker-watchdog reads the log in chunks of 64KB, and if the "Kernel panic" phrase happened to span two chunks it would never be picked up. That's the only explanation I can come up with so far.
Comment 4 Dan Callaghan 2014-01-15 22:34:40 EST
I'm going to assume that this was indeed caused by the problem described in comment 1, since that bug definitely exists and can cause missed panics (although it should be pretty rare to hit it) and no other theories have emerged.
Comment 5 Dan Callaghan 2014-01-16 01:33:16 EST
I fixed this while working on bug 952661.

http://gerrit.beaker-project.org/2692
Comment 7 Nick Coghlan 2014-01-21 02:33:55 EST
Given that this is a probabilistic bug based on when a panic occurs relative to the internal buffering in Beaker's console log processing, I don't believe it's practical to test it explicitly on a live system.

Instead, the new automated tests added as part of the patch deliberately provoke the misbehaviour in the log processing by injecting data directly.
Comment 9 Nick Coghlan 2014-02-02 23:51:48 EST
This change is included in the Beaker 0.15.3 maintenance release:

http://beaker-project.org/docs/whats-new/release-0.15.html#beaker-0-15-3

Note You need to log in before you can comment on or make changes to this bug.