Bug 2152619 - mount command fails with 0 in SUPERBLK column and last output line shows '0 mount: invalid kernel virtual address: 0 type: "super_block buffer"'
Summary: mount command fails with 0 in SUPERBLK column and last output line shows '0 m...
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: crash
Version: 8.6
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 8.9
Assignee: lijiang
QA Contact: xiaoying yan
URL:
Whiteboard:
Depends On:
Blocks: 2177109
TreeView+ depends on / blocked
 
Reported: 2022-12-12 14:06 UTC by Dave Wysochanski
Modified: 2023-06-05 07:57 UTC (History)
3 users (show)

Fixed In Version: crash-7.3.2-6.el8
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2177109 (view as bug list)
Environment:
Last Closed:
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-141906 0 None None None 2022-12-12 14:16:09 UTC

Description Dave Wysochanski 2022-12-12 14:06:10 UTC
Description of problem:
This problem seems to have popped up in more recent kernels, and I don't think it's due to damaged vmcore.

The mount command output looks like this on some vmcores now:
crash> mount
     MOUNT           SUPERBLK     TYPE   DEVNAME   DIRNAME
ffff9d0c067e0f00 ffff9d0c00014800 rootfs none      /         
...
ffff9d10258e5a20                0 mount: invalid kernel virtual address: 0  type: "super_block buffer"



Version-Release number of selected component (if applicable):
crash-7.3.1-5.el8.x86_64
upstream crash (latest 8.x) also seems affected

How reproducible:
Easy

Steps to Reproduce:
1. Run crash against an affected vmcore
2. Use 'mount' command

Actual results:
the last line of 'mount' command output looks like this:
ffff9d10258e5a20                0 mount: invalid kernel virtual address: 0  type: "super_block buffer"


Expected results:
the last line should contain valid data


Additional info:
There's multiple instances of this problem on our production system and I'll put into the private comments or you can just search on our system for it in the 'results' directory of all vmcores on production vmcore system.  This first showed up through pykdump command which uses 'mount' command and then parses output for some commands.  However just running the command directly from crash gives the same output so this is definitely an issue with crash, not pykdump.

I only briefly started investigating this.  I think we'll need an upstream patch and then a clone for RHEL9 of this bug, assuming I didn't miss anything like some damage to vmcore(s) that are affected.  This doesn't happen on a lot of vmcores but if it's starting to affect later vmcores then we may see more of it.  Given the likely higher use of 'mount' command I'm marking as 'medium' severity for now.

Comment 3 Dave Wysochanski 2022-12-12 14:18:41 UTC
Kernels affected so far

$ for f in $(grep -l "0 mount: invalid kernel virtual address: 0" /cores/retrace/tasks/*/results/nfs*); do D=$(dirname $f); grep RELEASE $D/sys; done | sort | uniq
grep: /cores/retrace/tasks/972810818/results/nfs_debug: Is a directory
     RELEASE: 4.18.0-305.3.1.el8_4.x86_64
     RELEASE: 4.18.0-372.26.1.el8_6.x86_64
     RELEASE: 4.18.0-372.32.1.el8_6.x86_64
     RELEASE: 4.18.0-372.9.1.el8.x86_64


Note You need to log in before you can comment on or make changes to this bug.