2149634 – Multipath device hangs forever on read, triggers kernel backtrace

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2149634 - Multipath device hangs forever on read, triggers kernel backtrace

Summary: Multipath device hangs forever on read, triggers kernel backtrace

Keywords:
Status:	CLOSED DUPLICATE of bug 2144258
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	device-mapper-multipath
Sub Component:
Version:	8.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Ben Marzinski
QA Contact:	Lin Li
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-11-30 13:25 UTC by Marius Vollmer
Modified:	2022-12-12 19:10 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-12-12 19:09:56 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-140889	0	None	None	None	2022-11-30 13:28:44 UTC

Description Marius Vollmer 2022-11-30 13:25:14 UTC

Description of problem:

The device mapper device set up by multipathd is non-responsive.  Reading from it hangs forever and produces these kernel messages:

Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: INFO: task cat:4401 blocked for more than 120 seconds.
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:       Not tainted 4.18.0-438.el8.x86_64 #1
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: task:cat             state:D stack:    0 pid: 4401 ppid:  4157 flags:0x00000000
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: Call Trace:
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  __schedule+0x2d1/0x870
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  schedule+0x55/0xf0
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  schedule_preempt_disabled+0xa/0x10
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  __mutex_lock.isra.7+0x314/0x430
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  ? _cond_resched+0x15/0x30
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  blkdev_get+0x193/0x2d0
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  ? igrab+0x19/0x50
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  ? bd_acquire+0xd0/0xd0
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  do_dentry_open+0x133/0x350
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  path_openat+0x558/0x1500
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  ? filemap_map_pages+0x271/0x410
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  do_filp_open+0x93/0x100
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  ? getname_flags+0x4a/0x1e0
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  ? __check_object_size+0xac/0x173
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  ? __alloc_fd+0x44/0x150
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  do_sys_openat2+0x211/0x2b0
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  do_sys_open+0x4b/0x80
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  do_syscall_64+0x5b/0x1b0
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xc6
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: RIP: 0033:0x7f1df05176d2
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: Code: Unable to access opcode bytes at RIP 0x7f1df05176a8.
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: RSP: 002b:00007ffe6149d670 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007f1df05176d2
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: RDX: 0000000000000000 RSI: 00007ffe6149e5f0 RDI: 00000000ffffff9c
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055e6dbc0a790
Nov 30 08:11:49 rhel-8-8-127-0-0-2-2201 kernel: R13: 00007ffe6149d920 R14: 0000000000000000 R15: 0000000000020000


Version-Release number of selected component (if applicable):
kernel-4.18.0-438.el8.x86_64
device-mapper-multipath-0.8.4-32.el8.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Set up a multipath scenario.  We have two disks added to a VM that have the same serial 'number':

# udevadm info /dev/sda
P: /devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:2/0:0:2:0/block/sda
N: sda
S: disk/by-id/scsi-0QEMU_QEMU_HARDDISK_MYSERIAL
S: disk/by-path/pci-0000:00:03.0-scsi-0:0:2:0
E: DEVLINKS=/dev/disk/by-path/pci-0000:00:03.0-scsi-0:0:2:0 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_MYSERIAL
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:2/0:0:2:0/block/sda
E: DEVTYPE=disk
E: DM_DEL_PART_NODES=1
E: DM_MULTIPATH_DEVICE_PATH=1
E: ID_BUS=scsi
E: ID_FS_TYPE=mpath_member
E: ID_MODEL=QEMU_HARDDISK
E: ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
E: ID_PATH=pci-0000:00:03.0-scsi-0:0:2:0
E: ID_PATH_TAG=pci-0000_00_03_0-scsi-0_0_2_0
E: ID_REVISION=2.5+
E: ID_SCSI=1
E: ID_SCSI_INQUIRY=1
E: ID_SCSI_SERIAL=MYSERIAL
E: ID_SERIAL=0QEMU_QEMU_HARDDISK_MYSERIAL
E: ID_SERIAL_SHORT=MYSERIAL
E: ID_TYPE=disk
E: ID_VENDOR=QEMU
E: ID_VENDOR_ENC=QEMU\x20\x20\x20\x20
E: ID_WWN=MYSERIAL
E: MAJOR=8
E: MINOR=0
E: MPATH_SBIN_PATH=/sbin
E: SCSI_IDENT_LUN_VENDOR=MYSERIAL
E: SCSI_IDENT_SERIAL=MYSERIAL
E: SCSI_MODEL=QEMU_HARDDISK
E: SCSI_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
E: SCSI_REVISION=2.5+
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=QEMU
E: SCSI_VENDOR_ENC=QEMU\x20\x20\x20\x20
E: SUBSYSTEM=block
E: SYSTEMD_READY=0
E: TAGS=:systemd:
E: USEC_INITIALIZED=19646925

# udevadm info /dev/sdb
P: /devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:2/0:0:2:1/block/sdb
N: sdb
S: disk/by-id/scsi-0QEMU_QEMU_HARDDISK_MYSERIAL
S: disk/by-path/pci-0000:00:03.0-scsi-0:0:2:1
E: DEVLINKS=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_MYSERIAL /dev/disk/by-path/pci-0000:00:03.0-scsi-0:0:2:1
E: DEVNAME=/dev/sdb
E: DEVPATH=/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:2/0:0:2:1/block/sdb
E: DEVTYPE=disk
E: DM_DEL_PART_NODES=1
E: DM_MULTIPATH_DEVICE_PATH=1
E: ID_BUS=scsi
E: ID_FS_TYPE=mpath_member
E: ID_MODEL=QEMU_HARDDISK
E: ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
E: ID_PATH=pci-0000:00:03.0-scsi-0:0:2:1
E: ID_PATH_TAG=pci-0000_00_03_0-scsi-0_0_2_1
E: ID_REVISION=2.5+
E: ID_SCSI=1
E: ID_SCSI_INQUIRY=1
E: ID_SCSI_SERIAL=MYSERIAL
E: ID_SERIAL=0QEMU_QEMU_HARDDISK_MYSERIAL
E: ID_SERIAL_SHORT=MYSERIAL
E: ID_TYPE=disk
E: ID_VENDOR=QEMU
E: ID_VENDOR_ENC=QEMU\x20\x20\x20\x20
E: ID_WWN=MYSERIAL
E: MAJOR=8
E: MINOR=16
E: MPATH_SBIN_PATH=/sbin
E: SCSI_IDENT_LUN_VENDOR=MYSERIAL
E: SCSI_IDENT_SERIAL=MYSERIAL
E: SCSI_MODEL=QEMU_HARDDISK
E: SCSI_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
E: SCSI_REVISION=2.5+
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=QEMU
E: SCSI_VENDOR_ENC=QEMU\x20\x20\x20\x20
E: SUBSYSTEM=block
E: SYSTEMD_READY=0
E: TAGS=:systemd:
E: USEC_INITIALIZED=22121611

2. Start multipathd:

# mpathconf --enable --with_multipathd y
# dmsetup table
mpatha: 0 20480 multipath 0 0 2 1 service-time 0 1 2 8:0 1 1 service-time 0 1 2 8:16 1 1 

3. Access the new multipath master.
# cat /dev/mapper/mpatha >/dev/null
cat: /dev/mapper/mpatha: No such file or directory
# cat /dev/dm-0 >/dev/null
<hangs unkillably, kernel complains like above>

4. Try to kill "cat"
# pkill -9 cat
<cat doesn't exit, kernel keeps complaining>

Actual results:
/dev/mapper/mpatha should exist and work.

Expected results:
/dev/mapper/mpatha is not created because udev can't process /dev/dm-0 because reading hangs.
The "cat" process also hangs and can't be killed.

Additional info:
This used to work with device-mapper-multipath 0.8.4-31 and kernel 4.18.0-437.

Comment 1 Zhang Yi 2022-11-30 13:30:39 UTC

Should be same with Bug 2144258 - [RHEL-8.8] system cannot display the mapping for /dev/mapper/mpatha

Comment 2 Marius Vollmer 2022-11-30 13:52:36 UTC

(In reply to Zhang Yi from comment #1)
> Should be same with Bug 2144258 - [RHEL-8.8] system cannot display the
> mapping for /dev/mapper/mpatha

Yes, looks like it.  Thanks!

Comment 3 Ben Marzinski 2022-12-12 19:09:56 UTC


*** This bug has been marked as a duplicate of bug 2144258 ***

Note You need to log in before you can comment on or make changes to this bug.