Red Hat Bugzilla – Bug 975486
beaker-watchdog fails to detect a panic if it crosses block boundaries
Last modified: 2018-02-05 19:41:31 EST
In this case beaker-watchdog was definitely reading the console log correctly (because it was uploading it to the job). But there is no log line "Panic detected for..." in watchdog.log, so beaker-watchdog never noticed the line "Kernel panic" in the console log.
One thing I noticed is that beaker-watchdog reads the log in chunks of 64KB, and if the "Kernel panic" phrase happened to span two chunks it would never be picked up. That's the only explanation I can come up with so far.
I'm going to assume that this was indeed caused by the problem described in comment 1, since that bug definitely exists and can cause missed panics (although it should be pretty rare to hit it) and no other theories have emerged.
I fixed this while working on bug 952661.
Given that this is a probabilistic bug based on when a panic occurs relative to the internal buffering in Beaker's console log processing, I don't believe it's practical to test it explicitly on a live system.
Instead, the new automated tests added as part of the patch deliberately provoke the misbehaviour in the log processing by injecting data directly.
This change is included in the Beaker 0.15.3 maintenance release: