RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1897815 - [ppc64le] segfault happened. multipathd[77038]: unhandled signal 11 at 0000000000000000 nip 00007fff807f8148 lr 00007fff80820f70 code 1
Summary: [ppc64le] segfault happened. multipathd[77038]: unhandled signal 11 at 000000...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: device-mapper-multipath
Version: 8.4
Hardware: ppc64le
OS: Linux
high
high
Target Milestone: rc
: 8.4
Assignee: Ben Marzinski
QA Contact: Lin Li
URL:
Whiteboard:
Depends On:
Blocks: 1796871
TreeView+ depends on / blocked
 
Reported: 2020-11-14 19:10 UTC by Lin Li
Modified: 2021-09-06 15:19 UTC (History)
11 users (show)

Fixed In Version: device-mapper-multipath-0.8.4-6.el8
Doc Type: Bug Fix
Doc Text:
Cause: Multipath allowed multipaths section configurations that didn't include a WWID, and when printing the configuration, attempted to dereference the missing WWID Consequence: If multipaths sections with no WWID were included in /etc/multipath.conf, multipathd would crash when running the "show config local" command. Fix: Since a multipaths section without a WWID will never be used, multipath now ignores them. Result: Multipathd no longer crashes when running the show config local" command.
Clone Of:
Environment:
Last Closed: 2021-05-18 15:06:46 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 189294 0 None None None 2020-11-17 08:49:58 UTC

Description Lin Li 2020-11-14 19:10:06 UTC 
Description of problem:
[ppc64le] "error -5 receiving packet" is getting reported while running "multipathd show config local"

Version-Release number of selected component (if applicable):
device-mapper-multipath-0.8.4-5.el8.ppc64le


How reproducible:
always

Steps to Reproduce:
1.[root@ibm-p9z-20-lp15 ~]# rpm -qa | grep multipath
device-mapper-multipath-libs-0.8.4-5.el8.ppc64le
device-mapper-multipath-0.8.4-5.el8.ppc64le

2,[root@ibm-p9z-20-lp15 ~]# multipathd show config local
defaults {
	verbosity 2
	polling_interval 5
	max_polling_interval 20
	reassign_maps "no"
	multipath_dir "/lib64/multipath"
	path_selector "service-time 0"
	path_grouping_policy "failover"
	uid_attribute "ID_SERIAL"
	prio "const"
	prio_args ""
	features "0"
	path_checker "tur"
	alias_prefix "mpath"
	failback "manual"
	rr_min_io 1000
	rr_min_io_rq 1
	max_fds "max"
	rr_weight "uniform"
	queue_without_daemon "no"
	flush_on_last_del "no"
	user_friendly_names "yes"
	fast_io_fail_tmo 5
	bindings_file "/etc/multipath/bindings"
	wwids_file "/etc/multipath/wwids"
	prkeys_file "/etc/multipath/prkeys"
	log_checker_err always
	all_tg_pt "no"
	retain_attached_hw_handler "yes"
	detect_prio "yes"
	detect_checker "yes"
	force_sync "no"
	strict_timing "no"
	deferred_remove "no"
	config_dir "/etc/multipath/conf.d"
	delay_watch_checks "no"
	delay_wait_checks "no"
	san_path_err_threshold "no"
	san_path_err_forget_rate "no"
	san_path_err_recovery_time "no"
	marginal_path_err_sample_time "no"
	marginal_path_err_rate_threshold "no"
	marginal_path_err_recheck_gap_time "no"
	marginal_path_double_failed_time "no"
	find_multipaths "smart"
	uxsock_timeout 4000
	retrigger_tries 3
	retrigger_delay 10
	missing_uev_wait_timeout 30
	skip_kpartx "no"
	disable_changed_wwids ignored
	remove_retries 0
	ghost_delay "no"
	find_multipaths_timeout -10
	enable_foreign ""
	marginal_pathgroups "no"
}
blacklist {
	devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])"
	device {
		vendor "SGI"
		product "Universal Xport"
	}
	device {
		vendor "^DGC"
		product "LUNZ"
	}
	device {
		vendor "EMC"
		product "LUNZ"
	}
	device {
		vendor "DELL"
		product "Universal Xport"
	}
	device {
		vendor "IBM"
		product "Universal Xport"
	}
	device {
		vendor "IBM"
		product "S/390"
	}
	device {
		vendor "LENOVO"
		product "Universal Xport"
	}
	device {
		vendor "(NETAPP|LSI|ENGENIO)"
		product "Universal Xport"
	}
	device {
		vendor "STK"
		product "Universal Xport"
	}
	device {
		vendor "SUN"
		product "Universal Xport"
	}
	device {
		vendor "(Intel|INTEL)"
		product "VTrak V-LUN"
	}
	device {
		vendor "Promise"
		product "VTrak V-LUN"
	}
	device {
		vendor "Promise"
		product "Vess V-LUN"
	}
}
blacklist_exceptions {
}
devices {
	device {
		vendor "AIX"
		product "VDASD"
		path_grouping_policy "multibus"
		no_path_retry 60
	}
	device {
		vendor "^DGC"
		product "^(RAID|DISK|VRAID)"
		product_blacklist "LUNZ"
		path_grouping_policy "group_by_prio"
		path_checker "emc_clariion"
		hardware_handler "1 emc"
		prio "emc"
		failback "immediate"
		no_path_retry 60
	}
}
overrides {
}
multipaths {
	multipath {
		wwid "3600601605e203f008e22d0221f07eb11"
		# alias "mpatha"
	}
}



blacklist {
}

multipaths {
	multipath {
#		wwid 3600601605e203f008e22d0221f07eb11
		# alias "mpatha"
#		path_grouping_policy failover
	}
}


3.edit /etc/multipath.conf 
[root@ibm-p9z-20-lp15 ~]# cat /etc/multipath.conf 
defaults {
#	find_multipaths yes
	find_multipaths smart
	user_friendly_names yes
}


blacklist {
}

multipaths {
	multipath {
#		wwid 3600601605e203f008e22d0221f07eb11
		# alias "mpatha"
#		path_grouping_policy failover
	}
}

4.[root@ibm-p9z-20-lp15 ~]# service multipathd reload
Redirecting to /bin/systemctl reload multipathd.service

5,[root@ibm-p9z-20-lp15 ~]# multipathd show config local
error -5 receiving packet   <---------------------------------

6,[root@ibm-p9z-20-lp15 ~]# dmesg
[43777.894648] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[85754.079808] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[86124.403868] device-mapper: multipath: 253:0: Failing path 8:64.
[86349.466824] device-mapper: multipath: 253:0: Failing path 8:32.
[86429.649835] device-mapper: multipath: 253:0: Reinstating path 8:64.
[86429.649858] device-mapper: multipath: 253:0: Reinstating path 8:32.
[86434.490220] device-mapper: multipath: 253:0: Failing path 8:64.
[86434.490345] device-mapper: multipath: 253:0: Failing path 8:32.
[86448.331408] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[86448.332638] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[86448.332819] sd 2:0:0:0: Asymmetric access state changed
[86448.351293] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[86449.601476] device-mapper: multipath: 253:0: Failing path 8:64.
[86449.601510] device-mapper: multipath: 253:0: Failing path 8:32.
[86449.621486] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[87174.811626] sd 1:0:1:0: Asymmetric access state changed
[87174.827678] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[87175.812971] device-mapper: multipath: 253:0: Reinstating path 8:64.
[87175.827756] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[87175.829838] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87175.987654] device-mapper: multipath: 253:0: Failing path 8:32.
[87176.027711] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87181.078300] device-mapper: multipath: 253:0: Failing path 8:32.
[87181.108448] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87185.079431] sd 1:0:0:0: Asymmetric access state changed
[87185.099216] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[87192.083034] sd 2:0:0:0: Asymmetric access state changed
[87192.100047] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[87205.087235] sd 2:0:1:0: Asymmetric access state changed
[87205.101872] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87206.088908] device-mapper: multipath: 253:0: Reinstating path 8:32.
[87206.102090] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87347.772774] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87538.178847] sd 1:0:0:0: Asymmetric access state changed
[87538.190558] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[87543.181785] sd 2:0:0:0: Asymmetric access state changed
[87543.211398] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[88198.354326] device-mapper: multipath: 253:0: Failing path 8:64.
[88268.307365] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[88503.438820] device-mapper: multipath: 253:0: Failing path 8:32.
[88546.188097] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[88546.189172] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[88564.455412] sd 2:0:0:0: Asymmetric access state changed
[88564.470559] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[88739.505812] device-mapper: multipath: 253:0: Failing path 8:48.
[88753.118183] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[88925.562271] device-mapper: multipath: 253:0: Failing path 8:16.
[89098.928348] INFO: task dd:73103 blocked for more than 120 seconds.
[89098.928357]       Not tainted 4.18.0-247.el8.ppc64le #1
[89098.928359] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[89098.928362] dd              D    0 73103  44566 0x00040080
[89098.928366] Call Trace:
[89098.928369] [c000000146457660] [c00000014436bc80] 0xc00000014436bc80 (unreliable)
[89098.928375] [c000000146457830] [c000000000020650] __switch_to+0x2e0/0x4f0
[89098.928378] [c000000146457890] [c000000000e4f834] __schedule+0x2d4/0x9b0
[89098.928382] [c000000146457960] [c000000000e4ff78] schedule+0x68/0x130
[89098.928385] [c000000146457990] [c000000000e5049c] io_schedule+0x5c/0x90
[89098.928388] [c0000001464579c0] [c000000000e50ca8] bit_wait_io+0x28/0x80
[89098.928391] [c0000001464579f0] [c000000000e51004] out_of_line_wait_on_bit+0x104/0x1b0
[89098.928395] [c000000146457a80] [c0000000005a8bf0] __block_write_begin_int+0x2e0/0x750
[89098.928399] [c000000146457b50] [c0000000005a90ec] block_write_begin+0x6c/0x160
[89098.928402] [c000000146457ba0] [c0000000005ab880] blkdev_write_begin+0x40/0x60
[89098.928406] [c000000146457bc0] [c0000000003edc80] generic_perform_write+0x100/0x2a0
[89098.928410] [c000000146457c60] [c0000000003f2184] __generic_file_write_iter+0x104/0x2b0
[89098.928413] [c000000146457cc0] [c0000000005ac658] blkdev_write_iter+0xb8/0x180
[89098.928417] [c000000146457d30] [c0000000005284e8] new_sync_write+0x148/0x1d0
[89098.928420] [c000000146457dd0] [c00000000052e350] sys_write+0x2c0/0x3a0
[89098.928424] [c000000146457e30] [c00000000000b408] system_call+0x5c/0x70
[89221.826303] INFO: task dd:73103 blocked for more than 120 seconds.
[89221.826312]       Not tainted 4.18.0-247.el8.ppc64le #1
[89221.826315] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[89221.826318] dd              D    0 73103  44566 0x00040082
[89221.826321] Call Trace:
[89221.826324] [c000000146457660] [c00000014436bc80] 0xc00000014436bc80 (unreliable)
[89221.826330] [c000000146457830] [c000000000020650] __switch_to+0x2e0/0x4f0
[89221.826334] [c000000146457890] [c000000000e4f834] __schedule+0x2d4/0x9b0
[89221.826337] [c000000146457960] [c000000000e4ff78] schedule+0x68/0x130
[89221.826341] [c000000146457990] [c000000000e5049c] io_schedule+0x5c/0x90
[89221.826344] [c0000001464579c0] [c000000000e50ca8] bit_wait_io+0x28/0x80
[89221.826347] [c0000001464579f0] [c000000000e51004] out_of_line_wait_on_bit+0x104/0x1b0
[89221.826352] [c000000146457a80] [c0000000005a8bf0] __block_write_begin_int+0x2e0/0x750
[89221.826355] [c000000146457b50] [c0000000005a90ec] block_write_begin+0x6c/0x160
[89221.826359] [c000000146457ba0] [c0000000005ab880] blkdev_write_begin+0x40/0x60
[89221.826363] [c000000146457bc0] [c0000000003edc80] generic_perform_write+0x100/0x2a0
[89221.826367] [c000000146457c60] [c0000000003f2184] __generic_file_write_iter+0x104/0x2b0
[89221.826370] [c000000146457cc0] [c0000000005ac658] blkdev_write_iter+0xb8/0x180
[89221.826374] [c000000146457d30] [c0000000005284e8] new_sync_write+0x148/0x1d0
[89221.826378] [c000000146457dd0] [c00000000052e350] sys_write+0x2c0/0x3a0
[89221.826382] [c000000146457e30] [c00000000000b408] system_call+0x5c/0x70
[89225.647980] blk_update_request: I/O error, dev dm-0, sector 1152 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[89225.647994] blk_update_request: I/O error, dev dm-0, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 9 prio class 0
[89225.647999] Buffer I/O error on dev dm-0, logical block 0, lost async page write
[89225.648006] Buffer I/O error on dev dm-0, logical block 1, lost async page write
[89225.648009] Buffer I/O error on dev dm-0, logical block 2, lost async page write
[89225.648013] Buffer I/O error on dev dm-0, logical block 3, lost async page write
[89225.648016] Buffer I/O error on dev dm-0, logical block 4, lost async page write
[89225.648020] Buffer I/O error on dev dm-0, logical block 5, lost async page write
[89225.648023] Buffer I/O error on dev dm-0, logical block 6, lost async page write
[89225.648027] Buffer I/O error on dev dm-0, logical block 7, lost async page write
[89225.648030] Buffer I/O error on dev dm-0, logical block 8, lost async page write
[89318.677007] sd 1:0:1:0: Asymmetric access state changed
[89318.690765] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[89319.678547] device-mapper: multipath: 253:0: Reinstating path 8:64.
[89319.711023] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[89319.712106] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[89324.680086] sd 1:0:0:0: Asymmetric access state changed
[89324.691811] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[89325.681743] device-mapper: multipath: 253:0: Reinstating path 8:48.
[89335.685261] sd 2:0:0:0: Asymmetric access state changed
[89335.703340] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[89336.686532] device-mapper: multipath: 253:0: Reinstating path 8:16.
[89338.687357] sd 2:0:1:0: Asymmetric access state changed
[89338.703695] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[89339.688595] device-mapper: multipath: 253:0: Reinstating path 8:32.
[89425.346405] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[89425.346549] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[89425.347641] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[89425.347765] sd 1:0:1:0: Asymmetric access state changed
[89425.366557] sd 2:0:0:0: Asymmetric access state changed
[89425.376510] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[89425.386343] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[89425.387980] sd 2:0:1:0: Asymmetric access state changed
[89425.416501] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[89720.788614] device-mapper: multipath: 253:0: Failing path 8:32.
[89940.847053] device-mapper: multipath: 253:0: Failing path 8:16.
[91034.911516] device-mapper: multipath: 253:0: Failing path 8:16.
[91034.911554] device-mapper: multipath: 253:0: Failing path 8:32.
[91034.951613] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[91034.951696] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[91034.953025] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[91034.953522] sd 1:0:0:0: Asymmetric access state changed
[91034.971609] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91180.184737] sd 2:0:0:0: Asymmetric access state changed
[91180.204350] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91181.204581] device-mapper: multipath: 253:0: Reinstating path 8:16.
[91181.233121] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91181.234309] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[91185.205844] sd 1:0:0:0: Asymmetric access state changed
[91185.206916] sd 1:0:1:0: Asymmetric access state changed
[91185.233634] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[91185.233737] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[91190.211311] sd 2:0:1:0: Asymmetric access state changed
[91190.236149] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[91191.235781] device-mapper: multipath: 253:0: Reinstating path 8:32.
[91191.254644] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[91191.255827] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[91196.237244] sd 2:0:0:0: Asymmetric access state changed
[91196.255324] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91197.625052] sd 1:0:0:0: Asymmetric access state changed
[91197.645521] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91197.649414] sd 1:0:1:0: Asymmetric access state changed
[91197.665243] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[91602.504518] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[91652.988479] multipathd[949]: unhandled signal 11 at 0000000000000000 nip 00007fffb58f8148 lr 00007fffb5920f70 code 1
[91698.003836] multipathd[74929]: unhandled signal 11 at 0000000000000000 nip 00007fff865c8148 lr 00007fff865f0f70 code 1
[91769.205724] multipathd[75033]: unhandled signal 11 at 0000000000000000 nip 00007fffa6148148 lr 00007fffa6170f70 code 1
[91769.248861] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[root@ibm-p9z-20-lp15 ~]# vim /var/log/messages 
[root@ibm-p9z-20-lp15 ~]# dmesg
[43777.894648] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[85754.079808] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[86124.403868] device-mapper: multipath: 253:0: Failing path 8:64.
[86349.466824] device-mapper: multipath: 253:0: Failing path 8:32.
[86429.649835] device-mapper: multipath: 253:0: Reinstating path 8:64.
[86429.649858] device-mapper: multipath: 253:0: Reinstating path 8:32.
[86434.490220] device-mapper: multipath: 253:0: Failing path 8:64.
[86434.490345] device-mapper: multipath: 253:0: Failing path 8:32.
[86448.331408] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[86448.332638] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[86448.332819] sd 2:0:0:0: Asymmetric access state changed
[86448.351293] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[86449.601476] device-mapper: multipath: 253:0: Failing path 8:64.
[86449.601510] device-mapper: multipath: 253:0: Failing path 8:32.
[86449.621486] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[87174.811626] sd 1:0:1:0: Asymmetric access state changed
[87174.827678] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[87175.812971] device-mapper: multipath: 253:0: Reinstating path 8:64.
[87175.827756] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[87175.829838] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87175.987654] device-mapper: multipath: 253:0: Failing path 8:32.
[87176.027711] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87181.078300] device-mapper: multipath: 253:0: Failing path 8:32.
[87181.108448] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87185.079431] sd 1:0:0:0: Asymmetric access state changed
[87185.099216] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[87192.083034] sd 2:0:0:0: Asymmetric access state changed
[87192.100047] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[87205.087235] sd 2:0:1:0: Asymmetric access state changed
[87205.101872] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87206.088908] device-mapper: multipath: 253:0: Reinstating path 8:32.
[87206.102090] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87347.772774] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[87538.178847] sd 1:0:0:0: Asymmetric access state changed
[87538.190558] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[87543.181785] sd 2:0:0:0: Asymmetric access state changed
[87543.211398] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[88198.354326] device-mapper: multipath: 253:0: Failing path 8:64.
[88268.307365] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[88503.438820] device-mapper: multipath: 253:0: Failing path 8:32.
[88546.188097] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[88546.189172] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[88564.455412] sd 2:0:0:0: Asymmetric access state changed
[88564.470559] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[88739.505812] device-mapper: multipath: 253:0: Failing path 8:48.
[88753.118183] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[88925.562271] device-mapper: multipath: 253:0: Failing path 8:16.
[89098.928348] INFO: task dd:73103 blocked for more than 120 seconds.
[89098.928357]       Not tainted 4.18.0-247.el8.ppc64le #1
[89098.928359] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[89098.928362] dd              D    0 73103  44566 0x00040080
[89098.928366] Call Trace:
[89098.928369] [c000000146457660] [c00000014436bc80] 0xc00000014436bc80 (unreliable)
[89098.928375] [c000000146457830] [c000000000020650] __switch_to+0x2e0/0x4f0
[89098.928378] [c000000146457890] [c000000000e4f834] __schedule+0x2d4/0x9b0
[89098.928382] [c000000146457960] [c000000000e4ff78] schedule+0x68/0x130
[89098.928385] [c000000146457990] [c000000000e5049c] io_schedule+0x5c/0x90
[89098.928388] [c0000001464579c0] [c000000000e50ca8] bit_wait_io+0x28/0x80
[89098.928391] [c0000001464579f0] [c000000000e51004] out_of_line_wait_on_bit+0x104/0x1b0
[89098.928395] [c000000146457a80] [c0000000005a8bf0] __block_write_begin_int+0x2e0/0x750
[89098.928399] [c000000146457b50] [c0000000005a90ec] block_write_begin+0x6c/0x160
[89098.928402] [c000000146457ba0] [c0000000005ab880] blkdev_write_begin+0x40/0x60
[89098.928406] [c000000146457bc0] [c0000000003edc80] generic_perform_write+0x100/0x2a0
[89098.928410] [c000000146457c60] [c0000000003f2184] __generic_file_write_iter+0x104/0x2b0
[89098.928413] [c000000146457cc0] [c0000000005ac658] blkdev_write_iter+0xb8/0x180
[89098.928417] [c000000146457d30] [c0000000005284e8] new_sync_write+0x148/0x1d0
[89098.928420] [c000000146457dd0] [c00000000052e350] sys_write+0x2c0/0x3a0
[89098.928424] [c000000146457e30] [c00000000000b408] system_call+0x5c/0x70
[89221.826303] INFO: task dd:73103 blocked for more than 120 seconds.
[89221.826312]       Not tainted 4.18.0-247.el8.ppc64le #1
[89221.826315] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[89221.826318] dd              D    0 73103  44566 0x00040082
[89221.826321] Call Trace:
[89221.826324] [c000000146457660] [c00000014436bc80] 0xc00000014436bc80 (unreliable)
[89221.826330] [c000000146457830] [c000000000020650] __switch_to+0x2e0/0x4f0
[89221.826334] [c000000146457890] [c000000000e4f834] __schedule+0x2d4/0x9b0
[89221.826337] [c000000146457960] [c000000000e4ff78] schedule+0x68/0x130
[89221.826341] [c000000146457990] [c000000000e5049c] io_schedule+0x5c/0x90
[89221.826344] [c0000001464579c0] [c000000000e50ca8] bit_wait_io+0x28/0x80
[89221.826347] [c0000001464579f0] [c000000000e51004] out_of_line_wait_on_bit+0x104/0x1b0
[89221.826352] [c000000146457a80] [c0000000005a8bf0] __block_write_begin_int+0x2e0/0x750
[89221.826355] [c000000146457b50] [c0000000005a90ec] block_write_begin+0x6c/0x160
[89221.826359] [c000000146457ba0] [c0000000005ab880] blkdev_write_begin+0x40/0x60
[89221.826363] [c000000146457bc0] [c0000000003edc80] generic_perform_write+0x100/0x2a0
[89221.826367] [c000000146457c60] [c0000000003f2184] __generic_file_write_iter+0x104/0x2b0
[89221.826370] [c000000146457cc0] [c0000000005ac658] blkdev_write_iter+0xb8/0x180
[89221.826374] [c000000146457d30] [c0000000005284e8] new_sync_write+0x148/0x1d0
[89221.826378] [c000000146457dd0] [c00000000052e350] sys_write+0x2c0/0x3a0
[89221.826382] [c000000146457e30] [c00000000000b408] system_call+0x5c/0x70
[89225.647980] blk_update_request: I/O error, dev dm-0, sector 1152 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[89225.647994] blk_update_request: I/O error, dev dm-0, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 9 prio class 0
[89225.647999] Buffer I/O error on dev dm-0, logical block 0, lost async page write
[89225.648006] Buffer I/O error on dev dm-0, logical block 1, lost async page write
[89225.648009] Buffer I/O error on dev dm-0, logical block 2, lost async page write
[89225.648013] Buffer I/O error on dev dm-0, logical block 3, lost async page write
[89225.648016] Buffer I/O error on dev dm-0, logical block 4, lost async page write
[89225.648020] Buffer I/O error on dev dm-0, logical block 5, lost async page write
[89225.648023] Buffer I/O error on dev dm-0, logical block 6, lost async page write
[89225.648027] Buffer I/O error on dev dm-0, logical block 7, lost async page write
[89225.648030] Buffer I/O error on dev dm-0, logical block 8, lost async page write
[89318.677007] sd 1:0:1:0: Asymmetric access state changed
[89318.690765] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[89319.678547] device-mapper: multipath: 253:0: Reinstating path 8:64.
[89319.711023] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[89319.712106] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[89324.680086] sd 1:0:0:0: Asymmetric access state changed
[89324.691811] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[89325.681743] device-mapper: multipath: 253:0: Reinstating path 8:48.
[89335.685261] sd 2:0:0:0: Asymmetric access state changed
[89335.703340] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[89336.686532] device-mapper: multipath: 253:0: Reinstating path 8:16.
[89338.687357] sd 2:0:1:0: Asymmetric access state changed
[89338.703695] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[89339.688595] device-mapper: multipath: 253:0: Reinstating path 8:32.
[89425.346405] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[89425.346549] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[89425.347641] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[89425.347765] sd 1:0:1:0: Asymmetric access state changed
[89425.366557] sd 2:0:0:0: Asymmetric access state changed
[89425.376510] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[89425.386343] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[89425.387980] sd 2:0:1:0: Asymmetric access state changed
[89425.416501] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[89720.788614] device-mapper: multipath: 253:0: Failing path 8:32.
[89940.847053] device-mapper: multipath: 253:0: Failing path 8:16.
[91034.911516] device-mapper: multipath: 253:0: Failing path 8:16.
[91034.911554] device-mapper: multipath: 253:0: Failing path 8:32.
[91034.951613] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[91034.951696] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[91034.953025] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[91034.953522] sd 1:0:0:0: Asymmetric access state changed
[91034.971609] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91180.184737] sd 2:0:0:0: Asymmetric access state changed
[91180.204350] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91181.204581] device-mapper: multipath: 253:0: Reinstating path 8:16.
[91181.233121] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91181.234309] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[91185.205844] sd 1:0:0:0: Asymmetric access state changed
[91185.206916] sd 1:0:1:0: Asymmetric access state changed
[91185.233634] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[91185.233737] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[91190.211311] sd 2:0:1:0: Asymmetric access state changed
[91190.236149] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[91191.235781] device-mapper: multipath: 253:0: Reinstating path 8:32.
[91191.254644] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[91191.255827] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[91196.237244] sd 2:0:0:0: Asymmetric access state changed
[91196.255324] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91197.625052] sd 1:0:0:0: Asymmetric access state changed
[91197.645521] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[91197.649414] sd 1:0:1:0: Asymmetric access state changed
[91197.665243] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[91602.504518] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[91652.988479] multipathd[949]: unhandled signal 11 at 0000000000000000 nip 00007fffb58f8148 lr 00007fffb5920f70 code 1
[91698.003836] multipathd[74929]: unhandled signal 11 at 0000000000000000 nip 00007fff865c8148 lr 00007fff865f0f70 code 1
[91769.205724] multipathd[75033]: unhandled signal 11 at 0000000000000000 nip 00007fffa6148148 lr 00007fffa6170f70 code 1
[91769.248861] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[92369.076509] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[92594.409442] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[93050.296087] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[93158.431936] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[93183.245174] multipathd[75308]: unhandled signal 11 at 0000000000000000 nip 00007fffa83a8148 lr 00007fffa83d0f70 code 1
[93289.061270] multipathd[76077]: unhandled signal 11 at 0000000000000000 nip 00007fffacd08148 lr 00007fffacd30f70 code 1
[93299.852564] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[93509.790665] multipathd[76192]: unhandled signal 11 at 0000000000000000 nip 00007fffb45a8148 lr 00007fffb45d0f70 code 1
[93984.147103] multipathd[76553]: unhandled signal 11 at 0000000000000000 nip 00007fffb0df8148 lr 00007fffb0e20f70 code 1
[93984.202661] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[94084.707436] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[94084.709016] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[94084.709177] sd 2:0:0:0: Asymmetric access state changed
[94084.727331] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[94085.297809] sd 1:0:1:0: Asymmetric access state changed
[94085.299389] sd 2:0:1:0: Asymmetric access state changed
[94085.327613] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[94085.367813] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[94085.368784] sd 2:0:1:0: Asymmetric access state changed
[94085.368971] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[94085.370441] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[94086.427664] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[94090.378883] sd 1:0:0:0: Asymmetric access state changed
[94090.398165] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[94095.381254] sd 2:0:0:0: Asymmetric access state changed
[94095.399009] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[94100.781568] multipathd[76622]: unhandled signal 11 at 0000000000000000 nip 00007fff85e48148 lr 00007fff85e70f70 code 1
[94321.512047] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[94354.316742] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[95031.505824] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[95031.507429] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[95031.507583] sd 2:0:0:0: Asymmetric access state changed
[95031.535728] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[95031.816092] sd 1:0:1:0: Asymmetric access state changed
[95031.817717] sd 2:0:1:0: Asymmetric access state changed
[95031.846238] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[95031.865929] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[95031.867619] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[95031.867798] sd 2:0:1:0: Asymmetric access state changed
[95031.886045] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[95032.956013] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[95036.907431] sd 1:0:0:0: Asymmetric access state changed
[95036.926760] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[95038.166327] sd 2:0:0:0: Asymmetric access state changed
[95038.176948] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[95321.248205] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[95327.569198] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[95327.570392] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[95327.570546] sd 2:0:0:0: Asymmetric access state changed
[95327.589097] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
[95328.279336] sd 1:0:1:0: Asymmetric access state changed
[95328.281062] sd 2:0:1:0: Asymmetric access state changed
[95328.309208] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[95328.359329] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
[95328.360830] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[95328.361012] sd 2:0:1:0: Asymmetric access state changed
[95328.389238] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[95329.409412] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[95333.360576] sd 1:0:0:0: Asymmetric access state changed
[95333.390161] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[95338.362986] sd 2:0:0:0: Asymmetric access state changed
[95338.391093] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
[95354.519712] multipathd[77038]: unhandled signal 11 at 0000000000000000 nip 00007fff807f8148 lr 00007fff80820f70 code 1
[95374.466019] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA



7,[root@ibm-p9z-20-lp15 ~]# tail -n 100 /var/log/messages 
Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: 8:48: reinstated
Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: mpatha: queue_if_no_path enabled
Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Recovered to normal mode
Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 1
Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 2 service-time 0 2 1 8:48 1 8:16 1 service-time 0 2 1 8:64 1 8:32 1]
Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: Asymmetric access state changed
Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: Asymmetric access state changed
Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: Asymmetric access state changed
Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
Nov 14 13:53:26 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 2 1 8:64 1 8:32 1 service-time 0 2 1 8:48 1 8:16 1]
Nov 14 13:53:26 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
Nov 14 13:53:30 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: Asymmetric access state changed
Nov 14 13:53:30 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
Nov 14 13:53:31 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: Asymmetric access state changed
Nov 14 13:53:31 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
Nov 14 13:56:42 ibm-p9z-20-lp15 multipathd[77037]: Multipath configuration updated.
Nov 14 13:56:42 ibm-p9z-20-lp15 multipathd[77037]: Reload multipathd for changes to take effect
Nov 14 13:58:14 ibm-p9z-20-lp15 systemd[1]: Reloading Device-Mapper Multipath Device Controller.
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: reconfigure (operator)
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77850]: ok
Nov 14 13:58:14 ibm-p9z-20-lp15 systemd[1]: Reloaded Device-Mapper Multipath Device Controller.
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: setting up map with 4/4 path checkers pending
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 2 1 8:48 1 8:16 1 service-time 0 2 1 8:64 1 8:32 1]
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Entering recovery mode: max_retries=60
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sdd - tur checker reports path is up
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: 8:48: reinstated
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: queue_if_no_path enabled
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Recovered to normal mode
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 1
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sde - tur checker reports path is up
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: 8:64: reinstated
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 2
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: switch to path group #2
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sdb - tur checker reports path is up
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: 8:16: reinstated
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 3
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sdc - tur checker reports path is up
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: 8:32: reinstated
Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 4
Nov 14 13:58:15 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
Nov 14 13:58:20 ibm-p9z-20-lp15 systemd[1]: Reloading Device-Mapper Multipath Device Controller.
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: reconfigure (operator)
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77885]: ok
Nov 14 13:58:21 ibm-p9z-20-lp15 systemd[1]: Reloaded Device-Mapper Multipath Device Controller.
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: setting up map with 4/4 path checkers pending
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 2 1 8:48 1 8:16 1 service-time 0 2 1 8:64 1 8:32 1]
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Entering recovery mode: max_retries=60
Nov 14 13:58:21 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
Nov 14 13:58:21 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
Nov 14 13:58:21 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: Asymmetric access state changed
Nov 14 13:58:21 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sdd - tur checker reports path is up
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: 8:48: reinstated
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: queue_if_no_path enabled
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Recovered to normal mode
Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 1
Nov 14 13:58:22 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 2 service-time 0 2 1 8:48 1 8:16 1 service-time 0 2 1 8:64 1 8:32 1]
Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: Asymmetric access state changed
Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: Asymmetric access state changed
Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA
Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: Asymmetric access state changed
Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA
Nov 14 13:58:23 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 2 1 8:64 1 8:32 1 service-time 0 2 1 8:48 1 8:16 1]
Nov 14 13:58:23 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
Nov 14 13:58:27 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: Asymmetric access state changed
Nov 14 13:58:27 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
Nov 14 13:58:32 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: Asymmetric access state changed
Nov 14 13:58:32 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA
Nov 14 13:58:48 ibm-p9z-20-lp15 kernel: multipathd[77038]: unhandled signal 11 at 0000000000000000 nip 00007fff807f8148 lr 00007fff80820f70 code 1
Nov 14 13:58:48 ibm-p9z-20-lp15 systemd[1]: Started Process Core Dump (PID 77991/UID 0).
Nov 14 13:58:48 ibm-p9z-20-lp15 systemd[1]: multipathd.service: Main process exited, code=killed, status=11/SEGV
Nov 14 13:58:48 ibm-p9z-20-lp15 systemd[1]: multipathd.service: Failed with result 'signal'.
Nov 14 13:58:48 ibm-p9z-20-lp15 systemd-coredump[77992]: Process 77037 (multipathd) of user 0 dumped core.#012#012Stack trace of thread 77038:#012#0  0x00007fff807f8148 find_mp_by_wwid (libmultipath.so.0)#012#1  0x00007fff80820f70 snprint_config (libmultipath.so.0)#012#2  0x000000010f1043e8 show_config (multipathd)#012#3  0x000000010f105344 cli_list_config_local (multipathd)#012#4  0x000000010f10392c parse_cmd (multipathd)#012#5  0x000000010f0f8fd8 uxsock_trigger (multipathd)#012#6  0x000000010f101794 uxsock_listen (multipathd)#012#7  0x000000010f0fa67c uxlsnrloop (multipathd)#012#8  0x00007fff805d87c8 start_thread (libpthread.so.0)#012#9  0x00007fff801c0158 __clone (libc.so.6)
Nov 14 13:58:49 ibm-p9z-20-lp15 systemd[1]: systemd-coredump: Succeeded.
Nov 14 13:59:07 ibm-p9z-20-lp15 systemd[1]: Starting dnf makecache...
Nov 14 13:59:07 ibm-p9z-20-lp15 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: --------start up--------
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: read /etc/multipath.conf
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: path checkers start up
Nov 14 13:59:08 ibm-p9z-20-lp15 systemd[1]: Started Device-Mapper Multipath Device Controller.
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: switch to path group #2
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: sde - tur checker reports path is up
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: 8:64: reinstated
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: remaining active paths: 3
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: switch to path group #1
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: sdc - tur checker reports path is up
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: 8:32: reinstated
Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: remaining active paths: 4
Nov 14 13:59:08 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
Nov 14 13:59:08 ibm-p9z-20-lp15 dnf[78010]: Updating Subscription Management repositories.
Nov 14 13:59:08 ibm-p9z-20-lp15 dnf[78010]: Unable to read consumer identity
Nov 14 13:59:08 ibm-p9z-20-lp15 dnf[78010]: This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Nov 14 13:59:08 ibm-p9z-20-lp15 dnf[78010]: Metadata cache refreshed recently.
Nov 14 13:59:08 ibm-p9z-20-lp15 systemd[1]: dnf-makecache.service: Succeeded.
Nov 14 13:59:08 ibm-p9z-20-lp15 systemd[1]: Started dnf makecache.



Actual results:
segfault happened. multipathd[77038]: unhandled signal 11 at 0000000000000000 nip 00007fff807f8148 lr 00007fff80820f70 code 1

Expected results:
No segfault

Additional info:
This is a IBM ppc64le server.
Vendor:DGC
Product:VRAID

[root@ibm-p9z-20-lp15 ~]# multipath -ll
mpatha (3600601605e203f008e22d0221f07eb11) dm-0 DGC,VRAID
size=8.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:1:0 sde 8:64 active ready running
| `- 2:0:1:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 1:0:0:0 sdd 8:48 active ready running
  `- 2:0:0:0 sdb 8:16 active ready running
Comment 3 Ben Marzinski 2020-12-08 03:55:46 UTC 
Fixed
Comment 6 Lin Li 2020-12-10 07:10:00 UTC 
Verified on device-mapper-multipath-0.8.4-6.el8.ppc64le
1,[root@ibm-p9z-20-lp15 ~]# rpm -qa | grep multipath
device-mapper-multipath-libs-0.8.4-6.el8.ppc64le
device-mapper-multipath-0.8.4-6.el8.ppc64le

2,[root@ibm-p9z-20-lp15 ~]# multipathd show config local
defaults {
	verbosity 2
	polling_interval 5
	max_polling_interval 20
	reassign_maps "no"
	multipath_dir "/lib64/multipath"
	path_selector "service-time 0"
	path_grouping_policy "failover"
	uid_attribute "ID_SERIAL"
	prio "const"
	prio_args ""
	features "0"
	path_checker "tur"
	alias_prefix "mpath"
	failback "manual"
	rr_min_io 1000
	rr_min_io_rq 1
	max_fds "max"
	rr_weight "uniform"
	queue_without_daemon "no"
	flush_on_last_del "no"
	user_friendly_names "yes"
	fast_io_fail_tmo 5
	bindings_file "/etc/multipath/bindings"
	wwids_file "/etc/multipath/wwids"
	prkeys_file "/etc/multipath/prkeys"
	log_checker_err always
	all_tg_pt "no"
	retain_attached_hw_handler "yes"
	detect_prio "yes"
	detect_checker "yes"
	force_sync "no"
	strict_timing "no"
	deferred_remove "no"
	config_dir "/etc/multipath/conf.d"
	delay_watch_checks "no"
	delay_wait_checks "no"
	san_path_err_threshold "no"
	san_path_err_forget_rate "no"
	san_path_err_recovery_time "no"
	marginal_path_err_sample_time "no"
	marginal_path_err_rate_threshold "no"
	marginal_path_err_recheck_gap_time "no"
	marginal_path_double_failed_time "no"
	find_multipaths "smart"
	uxsock_timeout 4000
	retrigger_tries 3
	retrigger_delay 10
	missing_uev_wait_timeout 30
	skip_kpartx "no"
	disable_changed_wwids ignored
	remove_retries 0
	ghost_delay "no"
	find_multipaths_timeout -10
	enable_foreign ""
	marginal_pathgroups "no"
}
blacklist {
	devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])"
	device {
		vendor "SGI"
		product "Universal Xport"
	}
	device {
		vendor "^DGC"
		product "LUNZ"
	}
	device {
		vendor "EMC"
		product "LUNZ"
	}
	device {
		vendor "DELL"
		product "Universal Xport"
	}
	device {
		vendor "IBM"
		product "Universal Xport"
	}
	device {
		vendor "IBM"
		product "S/390"
	}
	device {
		vendor "LENOVO"
		product "Universal Xport"
	}
	device {
		vendor "(NETAPP|LSI|ENGENIO)"
		product "Universal Xport"
	}
	device {
		vendor "STK"
		product "Universal Xport"
	}
	device {
		vendor "SUN"
		product "Universal Xport"
	}
	device {
		vendor "(Intel|INTEL)"
		product "VTrak V-LUN"
	}
	device {
		vendor "Promise"
		product "VTrak V-LUN"
	}
	device {
		vendor "Promise"
		product "Vess V-LUN"
	}
}
blacklist_exceptions {
}
devices {
	device {
		vendor "AIX"
		product "VDASD"
		path_grouping_policy "multibus"
		no_path_retry 60
	}
	device {
		vendor "^DGC"
		product "^(RAID|DISK|VRAID)"
		product_blacklist "LUNZ"
		path_grouping_policy "group_by_prio"
		path_checker "emc_clariion"
		hardware_handler "1 emc"
		prio "emc"
		failback "immediate"
		no_path_retry 60
	}
}
overrides {
}
multipaths {
	multipath {
		wwid "3600601605e203f008e22d0221f07eb11"
		# alias "mpatha"
	}
}



blacklist {
}

multipaths {
	multipath {
#		wwid 3600601605e203f008e22d0221f07eb11
		# alias "mpatha"
#		path_grouping_policy failover
	}
}


3.edit /etc/multipath.conf 
[root@ibm-p9z-20-lp15 ~]# cat /etc/multipath.conf 
defaults {
#	find_multipaths yes
	find_multipaths smart
	user_friendly_names yes
}


blacklist {
}

multipaths {
	multipath {
#		wwid 3600601605e203f008e22d0221f07eb11
		# alias "mpatha"
#		path_grouping_policy failover
	}
}

4.[root@ibm-p9z-20-lp15 ~]# service multipathd reload
Redirecting to /bin/systemctl reload multipathd.service

5,[root@ibm-p9z-20-lp15 ~]# multipathd show config local
Dec 10 02:03:40 | multipaths config section missing wwid     <----------------------------------------------
defaults {
	verbosity 2
	polling_interval 5
	max_polling_interval 20
	reassign_maps "no"
	multipath_dir "/lib64/multipath"
	path_selector "service-time 0"
	path_grouping_policy "failover"
	uid_attribute "ID_SERIAL"
	prio "const"
	prio_args ""
	features "0"
	path_checker "tur"
	alias_prefix "mpath"
	failback "manual"
	rr_min_io 1000
	rr_min_io_rq 1
	max_fds "max"
	rr_weight "uniform"
	queue_without_daemon "no"
	flush_on_last_del "no"
	user_friendly_names "yes"
	fast_io_fail_tmo 5
	bindings_file "/etc/multipath/bindings"
	wwids_file "/etc/multipath/wwids"
	prkeys_file "/etc/multipath/prkeys"
	log_checker_err always
	all_tg_pt "no"
	retain_attached_hw_handler "yes"
	detect_prio "yes"
	detect_checker "yes"
	force_sync "no"
	strict_timing "no"
	deferred_remove "no"
	config_dir "/etc/multipath/conf.d"
	delay_watch_checks "no"
	delay_wait_checks "no"
	san_path_err_threshold "no"
	san_path_err_forget_rate "no"
	san_path_err_recovery_time "no"
	marginal_path_err_sample_time "no"
	marginal_path_err_rate_threshold "no"
	marginal_path_err_recheck_gap_time "no"
	marginal_path_double_failed_time "no"
	find_multipaths "smart"
	uxsock_timeout 4000
	retrigger_tries 3
	retrigger_delay 10
	missing_uev_wait_timeout 30
	skip_kpartx "no"
	disable_changed_wwids ignored
	remove_retries 0
	ghost_delay "no"
	find_multipaths_timeout -10
	enable_foreign ""
	marginal_pathgroups "no"
}
blacklist {
	devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])"
	device {
		vendor "SGI"
		product "Universal Xport"
	}
	device {
		vendor "^DGC"
		product "LUNZ"
	}
	device {
		vendor "EMC"
		product "LUNZ"
	}
	device {
		vendor "DELL"
		product "Universal Xport"
	}
	device {
		vendor "FUJITSU"
		product "Universal Xport"
	}
	device {
		vendor "IBM"
		product "Universal Xport"
	}
	device {
		vendor "IBM"
		product "S/390"
	}
	device {
		vendor "LENOVO"
		product "Universal Xport"
	}
	device {
		vendor "(NETAPP|LSI|ENGENIO)"
		product "Universal Xport"
	}
	device {
		vendor "STK"
		product "Universal Xport"
	}
	device {
		vendor "SUN"
		product "Universal Xport"
	}
	device {
		vendor "(Intel|INTEL)"
		product "VTrak V-LUN"
	}
	device {
		vendor "Promise"
		product "VTrak V-LUN"
	}
	device {
		vendor "Promise"
		product "Vess V-LUN"
	}
}
blacklist_exceptions {
}
devices {
	device {
		vendor "AIX"
		product "VDASD"
		path_grouping_policy "multibus"
		no_path_retry 60
	}
	device {
		vendor "^DGC"
		product "^(RAID|DISK|VRAID)"
		product_blacklist "LUNZ"
		path_grouping_policy "group_by_prio"
		path_checker "emc_clariion"
		hardware_handler "1 emc"
		prio "emc"
		failback "immediate"
		no_path_retry 60
	}
}
overrides {
}
multipaths {
	multipath {
		wwid "3600601605e203f008e22d0221f07eb11"
		# alias "mpatha"
	}
}



Test result: Multipathd no longer crashes when running the show config local" command.
Comment 8 errata-xmlrpc 2021-05-18 15:06:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (device-mapper-multipath bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:1685
Note You need to log in before you can comment on or make changes to this bug.