Description of problem:
One of our partners has seen several crash dumps where crash notes regs were missing, affecting ability to root cause issue from kdump.

Version-Release number of selected component (if applicable):
all rhel7 kernels affected, rhel8 is not affected

How reproducible:
unknown

Additional info:
It took a while to realize why is that happening but we believe to have the cause. This investigation was done out-of-band with the original customer calls, this is about the crash dump inconsistency, not about the cause of the panic.

There is a bug in “kernel/watchdog.c” where watchdog_overflow_callback() is over-writing passed NMI regs argument by doing:

/* Callback function for perf event subsystem */
static void watchdog_overflow_callback(struct perf_event *event,
                 struct perf_sample_data *data,
                 struct pt_regs *regs)
{
...
        if (is_hardlockup()) {
               int this_cpu = smp_processor_id();
               struct pt_regs *regs = get_irq_regs();      <<<<< HERE
               ...

This bug has been fixed in upstream by removing this single line, see:
https://github.com/torvalds/linux/commit/4d1f0fb096aedea7bb5489af93498a82e467c480#diff-bcebb2b2d89ecc04ae073c76a55631c86a560b5d44a1ab066bc2fa2c5bc8fd62

If we get an NMI in regular kernel context (eg. not on IRQ), then get_irq_regs() may pull a NULL pointer and pass it to nmi_panic() up to crash_save_cpu(), and we would get a page fault, and we would end up with a processor stalling without saving crash notes regs. Removing the above HERE line would prevent running into another case where we are missing crash notes regs.