Bug 1897815
| Summary: | [ppc64le] segfault happened. multipathd[77038]: unhandled signal 11 at 0000000000000000 nip 00007fff807f8148 lr 00007fff80820f70 code 1 | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Lin Li <lilin> |
| Component: | device-mapper-multipath | Assignee: | Ben Marzinski <bmarzins> |
| Status: | CLOSED ERRATA | QA Contact: | Lin Li <lilin> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 8.4 | CC: | agk, bmarzins, bugproxy, chetjain, hannsj_uhl, heinzm, iranna.ankad, jbrassow, msnitzer, prajnoha, zkabelac |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 8.4 | Flags: | pm-rhel:
mirror+
|
| Hardware: | ppc64le | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | device-mapper-multipath-0.8.4-6.el8 | Doc Type: | Bug Fix |
| Doc Text: |
Cause: Multipath allowed multipaths section configurations that didn't include a WWID, and when printing the configuration, attempted to dereference the missing WWID
Consequence: If multipaths sections with no WWID were included in /etc/multipath.conf, multipathd would crash when running the "show config local" command.
Fix: Since a multipaths section without a WWID will never be used, multipath now ignores them.
Result: Multipathd no longer crashes when running the show config local" command.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-05-18 15:06:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1796871 | ||
Fixed Verified on device-mapper-multipath-0.8.4-6.el8.ppc64le
1,[root@ibm-p9z-20-lp15 ~]# rpm -qa | grep multipath
device-mapper-multipath-libs-0.8.4-6.el8.ppc64le
device-mapper-multipath-0.8.4-6.el8.ppc64le
2,[root@ibm-p9z-20-lp15 ~]# multipathd show config local
defaults {
verbosity 2
polling_interval 5
max_polling_interval 20
reassign_maps "no"
multipath_dir "/lib64/multipath"
path_selector "service-time 0"
path_grouping_policy "failover"
uid_attribute "ID_SERIAL"
prio "const"
prio_args ""
features "0"
path_checker "tur"
alias_prefix "mpath"
failback "manual"
rr_min_io 1000
rr_min_io_rq 1
max_fds "max"
rr_weight "uniform"
queue_without_daemon "no"
flush_on_last_del "no"
user_friendly_names "yes"
fast_io_fail_tmo 5
bindings_file "/etc/multipath/bindings"
wwids_file "/etc/multipath/wwids"
prkeys_file "/etc/multipath/prkeys"
log_checker_err always
all_tg_pt "no"
retain_attached_hw_handler "yes"
detect_prio "yes"
detect_checker "yes"
force_sync "no"
strict_timing "no"
deferred_remove "no"
config_dir "/etc/multipath/conf.d"
delay_watch_checks "no"
delay_wait_checks "no"
san_path_err_threshold "no"
san_path_err_forget_rate "no"
san_path_err_recovery_time "no"
marginal_path_err_sample_time "no"
marginal_path_err_rate_threshold "no"
marginal_path_err_recheck_gap_time "no"
marginal_path_double_failed_time "no"
find_multipaths "smart"
uxsock_timeout 4000
retrigger_tries 3
retrigger_delay 10
missing_uev_wait_timeout 30
skip_kpartx "no"
disable_changed_wwids ignored
remove_retries 0
ghost_delay "no"
find_multipaths_timeout -10
enable_foreign ""
marginal_pathgroups "no"
}
blacklist {
devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])"
device {
vendor "SGI"
product "Universal Xport"
}
device {
vendor "^DGC"
product "LUNZ"
}
device {
vendor "EMC"
product "LUNZ"
}
device {
vendor "DELL"
product "Universal Xport"
}
device {
vendor "IBM"
product "Universal Xport"
}
device {
vendor "IBM"
product "S/390"
}
device {
vendor "LENOVO"
product "Universal Xport"
}
device {
vendor "(NETAPP|LSI|ENGENIO)"
product "Universal Xport"
}
device {
vendor "STK"
product "Universal Xport"
}
device {
vendor "SUN"
product "Universal Xport"
}
device {
vendor "(Intel|INTEL)"
product "VTrak V-LUN"
}
device {
vendor "Promise"
product "VTrak V-LUN"
}
device {
vendor "Promise"
product "Vess V-LUN"
}
}
blacklist_exceptions {
}
devices {
device {
vendor "AIX"
product "VDASD"
path_grouping_policy "multibus"
no_path_retry 60
}
device {
vendor "^DGC"
product "^(RAID|DISK|VRAID)"
product_blacklist "LUNZ"
path_grouping_policy "group_by_prio"
path_checker "emc_clariion"
hardware_handler "1 emc"
prio "emc"
failback "immediate"
no_path_retry 60
}
}
overrides {
}
multipaths {
multipath {
wwid "3600601605e203f008e22d0221f07eb11"
# alias "mpatha"
}
}
blacklist {
}
multipaths {
multipath {
# wwid 3600601605e203f008e22d0221f07eb11
# alias "mpatha"
# path_grouping_policy failover
}
}
3.edit /etc/multipath.conf
[root@ibm-p9z-20-lp15 ~]# cat /etc/multipath.conf
defaults {
# find_multipaths yes
find_multipaths smart
user_friendly_names yes
}
blacklist {
}
multipaths {
multipath {
# wwid 3600601605e203f008e22d0221f07eb11
# alias "mpatha"
# path_grouping_policy failover
}
}
4.[root@ibm-p9z-20-lp15 ~]# service multipathd reload
Redirecting to /bin/systemctl reload multipathd.service
5,[root@ibm-p9z-20-lp15 ~]# multipathd show config local
Dec 10 02:03:40 | multipaths config section missing wwid <----------------------------------------------
defaults {
verbosity 2
polling_interval 5
max_polling_interval 20
reassign_maps "no"
multipath_dir "/lib64/multipath"
path_selector "service-time 0"
path_grouping_policy "failover"
uid_attribute "ID_SERIAL"
prio "const"
prio_args ""
features "0"
path_checker "tur"
alias_prefix "mpath"
failback "manual"
rr_min_io 1000
rr_min_io_rq 1
max_fds "max"
rr_weight "uniform"
queue_without_daemon "no"
flush_on_last_del "no"
user_friendly_names "yes"
fast_io_fail_tmo 5
bindings_file "/etc/multipath/bindings"
wwids_file "/etc/multipath/wwids"
prkeys_file "/etc/multipath/prkeys"
log_checker_err always
all_tg_pt "no"
retain_attached_hw_handler "yes"
detect_prio "yes"
detect_checker "yes"
force_sync "no"
strict_timing "no"
deferred_remove "no"
config_dir "/etc/multipath/conf.d"
delay_watch_checks "no"
delay_wait_checks "no"
san_path_err_threshold "no"
san_path_err_forget_rate "no"
san_path_err_recovery_time "no"
marginal_path_err_sample_time "no"
marginal_path_err_rate_threshold "no"
marginal_path_err_recheck_gap_time "no"
marginal_path_double_failed_time "no"
find_multipaths "smart"
uxsock_timeout 4000
retrigger_tries 3
retrigger_delay 10
missing_uev_wait_timeout 30
skip_kpartx "no"
disable_changed_wwids ignored
remove_retries 0
ghost_delay "no"
find_multipaths_timeout -10
enable_foreign ""
marginal_pathgroups "no"
}
blacklist {
devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])"
device {
vendor "SGI"
product "Universal Xport"
}
device {
vendor "^DGC"
product "LUNZ"
}
device {
vendor "EMC"
product "LUNZ"
}
device {
vendor "DELL"
product "Universal Xport"
}
device {
vendor "FUJITSU"
product "Universal Xport"
}
device {
vendor "IBM"
product "Universal Xport"
}
device {
vendor "IBM"
product "S/390"
}
device {
vendor "LENOVO"
product "Universal Xport"
}
device {
vendor "(NETAPP|LSI|ENGENIO)"
product "Universal Xport"
}
device {
vendor "STK"
product "Universal Xport"
}
device {
vendor "SUN"
product "Universal Xport"
}
device {
vendor "(Intel|INTEL)"
product "VTrak V-LUN"
}
device {
vendor "Promise"
product "VTrak V-LUN"
}
device {
vendor "Promise"
product "Vess V-LUN"
}
}
blacklist_exceptions {
}
devices {
device {
vendor "AIX"
product "VDASD"
path_grouping_policy "multibus"
no_path_retry 60
}
device {
vendor "^DGC"
product "^(RAID|DISK|VRAID)"
product_blacklist "LUNZ"
path_grouping_policy "group_by_prio"
path_checker "emc_clariion"
hardware_handler "1 emc"
prio "emc"
failback "immediate"
no_path_retry 60
}
}
overrides {
}
multipaths {
multipath {
wwid "3600601605e203f008e22d0221f07eb11"
# alias "mpatha"
}
}
Test result: Multipathd no longer crashes when running the show config local" command.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (device-mapper-multipath bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:1685 |
Description of problem: [ppc64le] "error -5 receiving packet" is getting reported while running "multipathd show config local" Version-Release number of selected component (if applicable): device-mapper-multipath-0.8.4-5.el8.ppc64le How reproducible: always Steps to Reproduce: 1.[root@ibm-p9z-20-lp15 ~]# rpm -qa | grep multipath device-mapper-multipath-libs-0.8.4-5.el8.ppc64le device-mapper-multipath-0.8.4-5.el8.ppc64le 2,[root@ibm-p9z-20-lp15 ~]# multipathd show config local defaults { verbosity 2 polling_interval 5 max_polling_interval 20 reassign_maps "no" multipath_dir "/lib64/multipath" path_selector "service-time 0" path_grouping_policy "failover" uid_attribute "ID_SERIAL" prio "const" prio_args "" features "0" path_checker "tur" alias_prefix "mpath" failback "manual" rr_min_io 1000 rr_min_io_rq 1 max_fds "max" rr_weight "uniform" queue_without_daemon "no" flush_on_last_del "no" user_friendly_names "yes" fast_io_fail_tmo 5 bindings_file "/etc/multipath/bindings" wwids_file "/etc/multipath/wwids" prkeys_file "/etc/multipath/prkeys" log_checker_err always all_tg_pt "no" retain_attached_hw_handler "yes" detect_prio "yes" detect_checker "yes" force_sync "no" strict_timing "no" deferred_remove "no" config_dir "/etc/multipath/conf.d" delay_watch_checks "no" delay_wait_checks "no" san_path_err_threshold "no" san_path_err_forget_rate "no" san_path_err_recovery_time "no" marginal_path_err_sample_time "no" marginal_path_err_rate_threshold "no" marginal_path_err_recheck_gap_time "no" marginal_path_double_failed_time "no" find_multipaths "smart" uxsock_timeout 4000 retrigger_tries 3 retrigger_delay 10 missing_uev_wait_timeout 30 skip_kpartx "no" disable_changed_wwids ignored remove_retries 0 ghost_delay "no" find_multipaths_timeout -10 enable_foreign "" marginal_pathgroups "no" } blacklist { devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])" device { vendor "SGI" product "Universal Xport" } device { vendor "^DGC" product "LUNZ" } device { vendor "EMC" product "LUNZ" } device { vendor "DELL" product "Universal Xport" } device { vendor "IBM" product "Universal Xport" } device { vendor "IBM" product "S/390" } device { vendor "LENOVO" product "Universal Xport" } device { vendor "(NETAPP|LSI|ENGENIO)" product "Universal Xport" } device { vendor "STK" product "Universal Xport" } device { vendor "SUN" product "Universal Xport" } device { vendor "(Intel|INTEL)" product "VTrak V-LUN" } device { vendor "Promise" product "VTrak V-LUN" } device { vendor "Promise" product "Vess V-LUN" } } blacklist_exceptions { } devices { device { vendor "AIX" product "VDASD" path_grouping_policy "multibus" no_path_retry 60 } device { vendor "^DGC" product "^(RAID|DISK|VRAID)" product_blacklist "LUNZ" path_grouping_policy "group_by_prio" path_checker "emc_clariion" hardware_handler "1 emc" prio "emc" failback "immediate" no_path_retry 60 } } overrides { } multipaths { multipath { wwid "3600601605e203f008e22d0221f07eb11" # alias "mpatha" } } blacklist { } multipaths { multipath { # wwid 3600601605e203f008e22d0221f07eb11 # alias "mpatha" # path_grouping_policy failover } } 3.edit /etc/multipath.conf [root@ibm-p9z-20-lp15 ~]# cat /etc/multipath.conf defaults { # find_multipaths yes find_multipaths smart user_friendly_names yes } blacklist { } multipaths { multipath { # wwid 3600601605e203f008e22d0221f07eb11 # alias "mpatha" # path_grouping_policy failover } } 4.[root@ibm-p9z-20-lp15 ~]# service multipathd reload Redirecting to /bin/systemctl reload multipathd.service 5,[root@ibm-p9z-20-lp15 ~]# multipathd show config local error -5 receiving packet <--------------------------------- 6,[root@ibm-p9z-20-lp15 ~]# dmesg [43777.894648] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [85754.079808] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [86124.403868] device-mapper: multipath: 253:0: Failing path 8:64. [86349.466824] device-mapper: multipath: 253:0: Failing path 8:32. [86429.649835] device-mapper: multipath: 253:0: Reinstating path 8:64. [86429.649858] device-mapper: multipath: 253:0: Reinstating path 8:32. [86434.490220] device-mapper: multipath: 253:0: Failing path 8:64. [86434.490345] device-mapper: multipath: 253:0: Failing path 8:32. [86448.331408] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [86448.332638] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [86448.332819] sd 2:0:0:0: Asymmetric access state changed [86448.351293] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [86449.601476] device-mapper: multipath: 253:0: Failing path 8:64. [86449.601510] device-mapper: multipath: 253:0: Failing path 8:32. [86449.621486] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [87174.811626] sd 1:0:1:0: Asymmetric access state changed [87174.827678] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [87175.812971] device-mapper: multipath: 253:0: Reinstating path 8:64. [87175.827756] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [87175.829838] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87175.987654] device-mapper: multipath: 253:0: Failing path 8:32. [87176.027711] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87181.078300] device-mapper: multipath: 253:0: Failing path 8:32. [87181.108448] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87185.079431] sd 1:0:0:0: Asymmetric access state changed [87185.099216] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [87192.083034] sd 2:0:0:0: Asymmetric access state changed [87192.100047] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [87205.087235] sd 2:0:1:0: Asymmetric access state changed [87205.101872] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87206.088908] device-mapper: multipath: 253:0: Reinstating path 8:32. [87206.102090] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87347.772774] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87538.178847] sd 1:0:0:0: Asymmetric access state changed [87538.190558] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [87543.181785] sd 2:0:0:0: Asymmetric access state changed [87543.211398] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [88198.354326] device-mapper: multipath: 253:0: Failing path 8:64. [88268.307365] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [88503.438820] device-mapper: multipath: 253:0: Failing path 8:32. [88546.188097] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [88546.189172] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [88564.455412] sd 2:0:0:0: Asymmetric access state changed [88564.470559] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [88739.505812] device-mapper: multipath: 253:0: Failing path 8:48. [88753.118183] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [88925.562271] device-mapper: multipath: 253:0: Failing path 8:16. [89098.928348] INFO: task dd:73103 blocked for more than 120 seconds. [89098.928357] Not tainted 4.18.0-247.el8.ppc64le #1 [89098.928359] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [89098.928362] dd D 0 73103 44566 0x00040080 [89098.928366] Call Trace: [89098.928369] [c000000146457660] [c00000014436bc80] 0xc00000014436bc80 (unreliable) [89098.928375] [c000000146457830] [c000000000020650] __switch_to+0x2e0/0x4f0 [89098.928378] [c000000146457890] [c000000000e4f834] __schedule+0x2d4/0x9b0 [89098.928382] [c000000146457960] [c000000000e4ff78] schedule+0x68/0x130 [89098.928385] [c000000146457990] [c000000000e5049c] io_schedule+0x5c/0x90 [89098.928388] [c0000001464579c0] [c000000000e50ca8] bit_wait_io+0x28/0x80 [89098.928391] [c0000001464579f0] [c000000000e51004] out_of_line_wait_on_bit+0x104/0x1b0 [89098.928395] [c000000146457a80] [c0000000005a8bf0] __block_write_begin_int+0x2e0/0x750 [89098.928399] [c000000146457b50] [c0000000005a90ec] block_write_begin+0x6c/0x160 [89098.928402] [c000000146457ba0] [c0000000005ab880] blkdev_write_begin+0x40/0x60 [89098.928406] [c000000146457bc0] [c0000000003edc80] generic_perform_write+0x100/0x2a0 [89098.928410] [c000000146457c60] [c0000000003f2184] __generic_file_write_iter+0x104/0x2b0 [89098.928413] [c000000146457cc0] [c0000000005ac658] blkdev_write_iter+0xb8/0x180 [89098.928417] [c000000146457d30] [c0000000005284e8] new_sync_write+0x148/0x1d0 [89098.928420] [c000000146457dd0] [c00000000052e350] sys_write+0x2c0/0x3a0 [89098.928424] [c000000146457e30] [c00000000000b408] system_call+0x5c/0x70 [89221.826303] INFO: task dd:73103 blocked for more than 120 seconds. [89221.826312] Not tainted 4.18.0-247.el8.ppc64le #1 [89221.826315] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [89221.826318] dd D 0 73103 44566 0x00040082 [89221.826321] Call Trace: [89221.826324] [c000000146457660] [c00000014436bc80] 0xc00000014436bc80 (unreliable) [89221.826330] [c000000146457830] [c000000000020650] __switch_to+0x2e0/0x4f0 [89221.826334] [c000000146457890] [c000000000e4f834] __schedule+0x2d4/0x9b0 [89221.826337] [c000000146457960] [c000000000e4ff78] schedule+0x68/0x130 [89221.826341] [c000000146457990] [c000000000e5049c] io_schedule+0x5c/0x90 [89221.826344] [c0000001464579c0] [c000000000e50ca8] bit_wait_io+0x28/0x80 [89221.826347] [c0000001464579f0] [c000000000e51004] out_of_line_wait_on_bit+0x104/0x1b0 [89221.826352] [c000000146457a80] [c0000000005a8bf0] __block_write_begin_int+0x2e0/0x750 [89221.826355] [c000000146457b50] [c0000000005a90ec] block_write_begin+0x6c/0x160 [89221.826359] [c000000146457ba0] [c0000000005ab880] blkdev_write_begin+0x40/0x60 [89221.826363] [c000000146457bc0] [c0000000003edc80] generic_perform_write+0x100/0x2a0 [89221.826367] [c000000146457c60] [c0000000003f2184] __generic_file_write_iter+0x104/0x2b0 [89221.826370] [c000000146457cc0] [c0000000005ac658] blkdev_write_iter+0xb8/0x180 [89221.826374] [c000000146457d30] [c0000000005284e8] new_sync_write+0x148/0x1d0 [89221.826378] [c000000146457dd0] [c00000000052e350] sys_write+0x2c0/0x3a0 [89221.826382] [c000000146457e30] [c00000000000b408] system_call+0x5c/0x70 [89225.647980] blk_update_request: I/O error, dev dm-0, sector 1152 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [89225.647994] blk_update_request: I/O error, dev dm-0, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 9 prio class 0 [89225.647999] Buffer I/O error on dev dm-0, logical block 0, lost async page write [89225.648006] Buffer I/O error on dev dm-0, logical block 1, lost async page write [89225.648009] Buffer I/O error on dev dm-0, logical block 2, lost async page write [89225.648013] Buffer I/O error on dev dm-0, logical block 3, lost async page write [89225.648016] Buffer I/O error on dev dm-0, logical block 4, lost async page write [89225.648020] Buffer I/O error on dev dm-0, logical block 5, lost async page write [89225.648023] Buffer I/O error on dev dm-0, logical block 6, lost async page write [89225.648027] Buffer I/O error on dev dm-0, logical block 7, lost async page write [89225.648030] Buffer I/O error on dev dm-0, logical block 8, lost async page write [89318.677007] sd 1:0:1:0: Asymmetric access state changed [89318.690765] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [89319.678547] device-mapper: multipath: 253:0: Reinstating path 8:64. [89319.711023] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [89319.712106] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [89324.680086] sd 1:0:0:0: Asymmetric access state changed [89324.691811] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [89325.681743] device-mapper: multipath: 253:0: Reinstating path 8:48. [89335.685261] sd 2:0:0:0: Asymmetric access state changed [89335.703340] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [89336.686532] device-mapper: multipath: 253:0: Reinstating path 8:16. [89338.687357] sd 2:0:1:0: Asymmetric access state changed [89338.703695] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [89339.688595] device-mapper: multipath: 253:0: Reinstating path 8:32. [89425.346405] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [89425.346549] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [89425.347641] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [89425.347765] sd 1:0:1:0: Asymmetric access state changed [89425.366557] sd 2:0:0:0: Asymmetric access state changed [89425.376510] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [89425.386343] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [89425.387980] sd 2:0:1:0: Asymmetric access state changed [89425.416501] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA [89720.788614] device-mapper: multipath: 253:0: Failing path 8:32. [89940.847053] device-mapper: multipath: 253:0: Failing path 8:16. [91034.911516] device-mapper: multipath: 253:0: Failing path 8:16. [91034.911554] device-mapper: multipath: 253:0: Failing path 8:32. [91034.951613] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [91034.951696] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [91034.953025] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [91034.953522] sd 1:0:0:0: Asymmetric access state changed [91034.971609] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91180.184737] sd 2:0:0:0: Asymmetric access state changed [91180.204350] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91181.204581] device-mapper: multipath: 253:0: Reinstating path 8:16. [91181.233121] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91181.234309] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [91185.205844] sd 1:0:0:0: Asymmetric access state changed [91185.206916] sd 1:0:1:0: Asymmetric access state changed [91185.233634] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [91185.233737] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [91190.211311] sd 2:0:1:0: Asymmetric access state changed [91190.236149] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA [91191.235781] device-mapper: multipath: 253:0: Reinstating path 8:32. [91191.254644] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA [91191.255827] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [91196.237244] sd 2:0:0:0: Asymmetric access state changed [91196.255324] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91197.625052] sd 1:0:0:0: Asymmetric access state changed [91197.645521] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91197.649414] sd 1:0:1:0: Asymmetric access state changed [91197.665243] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [91602.504518] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [91652.988479] multipathd[949]: unhandled signal 11 at 0000000000000000 nip 00007fffb58f8148 lr 00007fffb5920f70 code 1 [91698.003836] multipathd[74929]: unhandled signal 11 at 0000000000000000 nip 00007fff865c8148 lr 00007fff865f0f70 code 1 [91769.205724] multipathd[75033]: unhandled signal 11 at 0000000000000000 nip 00007fffa6148148 lr 00007fffa6170f70 code 1 [91769.248861] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [root@ibm-p9z-20-lp15 ~]# vim /var/log/messages [root@ibm-p9z-20-lp15 ~]# dmesg [43777.894648] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [85754.079808] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [86124.403868] device-mapper: multipath: 253:0: Failing path 8:64. [86349.466824] device-mapper: multipath: 253:0: Failing path 8:32. [86429.649835] device-mapper: multipath: 253:0: Reinstating path 8:64. [86429.649858] device-mapper: multipath: 253:0: Reinstating path 8:32. [86434.490220] device-mapper: multipath: 253:0: Failing path 8:64. [86434.490345] device-mapper: multipath: 253:0: Failing path 8:32. [86448.331408] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [86448.332638] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [86448.332819] sd 2:0:0:0: Asymmetric access state changed [86448.351293] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [86449.601476] device-mapper: multipath: 253:0: Failing path 8:64. [86449.601510] device-mapper: multipath: 253:0: Failing path 8:32. [86449.621486] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [87174.811626] sd 1:0:1:0: Asymmetric access state changed [87174.827678] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [87175.812971] device-mapper: multipath: 253:0: Reinstating path 8:64. [87175.827756] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [87175.829838] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87175.987654] device-mapper: multipath: 253:0: Failing path 8:32. [87176.027711] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87181.078300] device-mapper: multipath: 253:0: Failing path 8:32. [87181.108448] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87185.079431] sd 1:0:0:0: Asymmetric access state changed [87185.099216] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [87192.083034] sd 2:0:0:0: Asymmetric access state changed [87192.100047] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [87205.087235] sd 2:0:1:0: Asymmetric access state changed [87205.101872] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87206.088908] device-mapper: multipath: 253:0: Reinstating path 8:32. [87206.102090] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87347.772774] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [87538.178847] sd 1:0:0:0: Asymmetric access state changed [87538.190558] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [87543.181785] sd 2:0:0:0: Asymmetric access state changed [87543.211398] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [88198.354326] device-mapper: multipath: 253:0: Failing path 8:64. [88268.307365] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [88503.438820] device-mapper: multipath: 253:0: Failing path 8:32. [88546.188097] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [88546.189172] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [88564.455412] sd 2:0:0:0: Asymmetric access state changed [88564.470559] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [88739.505812] device-mapper: multipath: 253:0: Failing path 8:48. [88753.118183] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [88925.562271] device-mapper: multipath: 253:0: Failing path 8:16. [89098.928348] INFO: task dd:73103 blocked for more than 120 seconds. [89098.928357] Not tainted 4.18.0-247.el8.ppc64le #1 [89098.928359] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [89098.928362] dd D 0 73103 44566 0x00040080 [89098.928366] Call Trace: [89098.928369] [c000000146457660] [c00000014436bc80] 0xc00000014436bc80 (unreliable) [89098.928375] [c000000146457830] [c000000000020650] __switch_to+0x2e0/0x4f0 [89098.928378] [c000000146457890] [c000000000e4f834] __schedule+0x2d4/0x9b0 [89098.928382] [c000000146457960] [c000000000e4ff78] schedule+0x68/0x130 [89098.928385] [c000000146457990] [c000000000e5049c] io_schedule+0x5c/0x90 [89098.928388] [c0000001464579c0] [c000000000e50ca8] bit_wait_io+0x28/0x80 [89098.928391] [c0000001464579f0] [c000000000e51004] out_of_line_wait_on_bit+0x104/0x1b0 [89098.928395] [c000000146457a80] [c0000000005a8bf0] __block_write_begin_int+0x2e0/0x750 [89098.928399] [c000000146457b50] [c0000000005a90ec] block_write_begin+0x6c/0x160 [89098.928402] [c000000146457ba0] [c0000000005ab880] blkdev_write_begin+0x40/0x60 [89098.928406] [c000000146457bc0] [c0000000003edc80] generic_perform_write+0x100/0x2a0 [89098.928410] [c000000146457c60] [c0000000003f2184] __generic_file_write_iter+0x104/0x2b0 [89098.928413] [c000000146457cc0] [c0000000005ac658] blkdev_write_iter+0xb8/0x180 [89098.928417] [c000000146457d30] [c0000000005284e8] new_sync_write+0x148/0x1d0 [89098.928420] [c000000146457dd0] [c00000000052e350] sys_write+0x2c0/0x3a0 [89098.928424] [c000000146457e30] [c00000000000b408] system_call+0x5c/0x70 [89221.826303] INFO: task dd:73103 blocked for more than 120 seconds. [89221.826312] Not tainted 4.18.0-247.el8.ppc64le #1 [89221.826315] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [89221.826318] dd D 0 73103 44566 0x00040082 [89221.826321] Call Trace: [89221.826324] [c000000146457660] [c00000014436bc80] 0xc00000014436bc80 (unreliable) [89221.826330] [c000000146457830] [c000000000020650] __switch_to+0x2e0/0x4f0 [89221.826334] [c000000146457890] [c000000000e4f834] __schedule+0x2d4/0x9b0 [89221.826337] [c000000146457960] [c000000000e4ff78] schedule+0x68/0x130 [89221.826341] [c000000146457990] [c000000000e5049c] io_schedule+0x5c/0x90 [89221.826344] [c0000001464579c0] [c000000000e50ca8] bit_wait_io+0x28/0x80 [89221.826347] [c0000001464579f0] [c000000000e51004] out_of_line_wait_on_bit+0x104/0x1b0 [89221.826352] [c000000146457a80] [c0000000005a8bf0] __block_write_begin_int+0x2e0/0x750 [89221.826355] [c000000146457b50] [c0000000005a90ec] block_write_begin+0x6c/0x160 [89221.826359] [c000000146457ba0] [c0000000005ab880] blkdev_write_begin+0x40/0x60 [89221.826363] [c000000146457bc0] [c0000000003edc80] generic_perform_write+0x100/0x2a0 [89221.826367] [c000000146457c60] [c0000000003f2184] __generic_file_write_iter+0x104/0x2b0 [89221.826370] [c000000146457cc0] [c0000000005ac658] blkdev_write_iter+0xb8/0x180 [89221.826374] [c000000146457d30] [c0000000005284e8] new_sync_write+0x148/0x1d0 [89221.826378] [c000000146457dd0] [c00000000052e350] sys_write+0x2c0/0x3a0 [89221.826382] [c000000146457e30] [c00000000000b408] system_call+0x5c/0x70 [89225.647980] blk_update_request: I/O error, dev dm-0, sector 1152 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [89225.647994] blk_update_request: I/O error, dev dm-0, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 9 prio class 0 [89225.647999] Buffer I/O error on dev dm-0, logical block 0, lost async page write [89225.648006] Buffer I/O error on dev dm-0, logical block 1, lost async page write [89225.648009] Buffer I/O error on dev dm-0, logical block 2, lost async page write [89225.648013] Buffer I/O error on dev dm-0, logical block 3, lost async page write [89225.648016] Buffer I/O error on dev dm-0, logical block 4, lost async page write [89225.648020] Buffer I/O error on dev dm-0, logical block 5, lost async page write [89225.648023] Buffer I/O error on dev dm-0, logical block 6, lost async page write [89225.648027] Buffer I/O error on dev dm-0, logical block 7, lost async page write [89225.648030] Buffer I/O error on dev dm-0, logical block 8, lost async page write [89318.677007] sd 1:0:1:0: Asymmetric access state changed [89318.690765] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [89319.678547] device-mapper: multipath: 253:0: Reinstating path 8:64. [89319.711023] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [89319.712106] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [89324.680086] sd 1:0:0:0: Asymmetric access state changed [89324.691811] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [89325.681743] device-mapper: multipath: 253:0: Reinstating path 8:48. [89335.685261] sd 2:0:0:0: Asymmetric access state changed [89335.703340] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [89336.686532] device-mapper: multipath: 253:0: Reinstating path 8:16. [89338.687357] sd 2:0:1:0: Asymmetric access state changed [89338.703695] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [89339.688595] device-mapper: multipath: 253:0: Reinstating path 8:32. [89425.346405] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [89425.346549] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [89425.347641] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [89425.347765] sd 1:0:1:0: Asymmetric access state changed [89425.366557] sd 2:0:0:0: Asymmetric access state changed [89425.376510] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [89425.386343] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [89425.387980] sd 2:0:1:0: Asymmetric access state changed [89425.416501] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA [89720.788614] device-mapper: multipath: 253:0: Failing path 8:32. [89940.847053] device-mapper: multipath: 253:0: Failing path 8:16. [91034.911516] device-mapper: multipath: 253:0: Failing path 8:16. [91034.911554] device-mapper: multipath: 253:0: Failing path 8:32. [91034.951613] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [91034.951696] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [91034.953025] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [91034.953522] sd 1:0:0:0: Asymmetric access state changed [91034.971609] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91180.184737] sd 2:0:0:0: Asymmetric access state changed [91180.204350] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91181.204581] device-mapper: multipath: 253:0: Reinstating path 8:16. [91181.233121] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91181.234309] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [91185.205844] sd 1:0:0:0: Asymmetric access state changed [91185.206916] sd 1:0:1:0: Asymmetric access state changed [91185.233634] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [91185.233737] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [91190.211311] sd 2:0:1:0: Asymmetric access state changed [91190.236149] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA [91191.235781] device-mapper: multipath: 253:0: Reinstating path 8:32. [91191.254644] sd 2:0:1:0: alua: port group 02 state N preferred supports tolUsNA [91191.255827] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [91196.237244] sd 2:0:0:0: Asymmetric access state changed [91196.255324] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91197.625052] sd 1:0:0:0: Asymmetric access state changed [91197.645521] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [91197.649414] sd 1:0:1:0: Asymmetric access state changed [91197.665243] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [91602.504518] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [91652.988479] multipathd[949]: unhandled signal 11 at 0000000000000000 nip 00007fffb58f8148 lr 00007fffb5920f70 code 1 [91698.003836] multipathd[74929]: unhandled signal 11 at 0000000000000000 nip 00007fff865c8148 lr 00007fff865f0f70 code 1 [91769.205724] multipathd[75033]: unhandled signal 11 at 0000000000000000 nip 00007fffa6148148 lr 00007fffa6170f70 code 1 [91769.248861] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [92369.076509] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [92594.409442] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [93050.296087] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [93158.431936] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [93183.245174] multipathd[75308]: unhandled signal 11 at 0000000000000000 nip 00007fffa83a8148 lr 00007fffa83d0f70 code 1 [93289.061270] multipathd[76077]: unhandled signal 11 at 0000000000000000 nip 00007fffacd08148 lr 00007fffacd30f70 code 1 [93299.852564] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [93509.790665] multipathd[76192]: unhandled signal 11 at 0000000000000000 nip 00007fffb45a8148 lr 00007fffb45d0f70 code 1 [93984.147103] multipathd[76553]: unhandled signal 11 at 0000000000000000 nip 00007fffb0df8148 lr 00007fffb0e20f70 code 1 [93984.202661] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [94084.707436] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [94084.709016] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [94084.709177] sd 2:0:0:0: Asymmetric access state changed [94084.727331] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [94085.297809] sd 1:0:1:0: Asymmetric access state changed [94085.299389] sd 2:0:1:0: Asymmetric access state changed [94085.327613] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [94085.367813] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [94085.368784] sd 2:0:1:0: Asymmetric access state changed [94085.368971] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [94085.370441] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [94086.427664] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [94090.378883] sd 1:0:0:0: Asymmetric access state changed [94090.398165] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [94095.381254] sd 2:0:0:0: Asymmetric access state changed [94095.399009] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [94100.781568] multipathd[76622]: unhandled signal 11 at 0000000000000000 nip 00007fff85e48148 lr 00007fff85e70f70 code 1 [94321.512047] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [94354.316742] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [95031.505824] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [95031.507429] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [95031.507583] sd 2:0:0:0: Asymmetric access state changed [95031.535728] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [95031.816092] sd 1:0:1:0: Asymmetric access state changed [95031.817717] sd 2:0:1:0: Asymmetric access state changed [95031.846238] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [95031.865929] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [95031.867619] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [95031.867798] sd 2:0:1:0: Asymmetric access state changed [95031.886045] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [95032.956013] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [95036.907431] sd 1:0:0:0: Asymmetric access state changed [95036.926760] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [95038.166327] sd 2:0:0:0: Asymmetric access state changed [95038.176948] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [95321.248205] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [95327.569198] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [95327.570392] sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [95327.570546] sd 2:0:0:0: Asymmetric access state changed [95327.589097] sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA [95328.279336] sd 1:0:1:0: Asymmetric access state changed [95328.281062] sd 2:0:1:0: Asymmetric access state changed [95328.309208] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [95328.359329] sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA [95328.360830] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [95328.361012] sd 2:0:1:0: Asymmetric access state changed [95328.389238] sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA [95329.409412] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA [95333.360576] sd 1:0:0:0: Asymmetric access state changed [95333.390161] sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [95338.362986] sd 2:0:0:0: Asymmetric access state changed [95338.391093] sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA [95354.519712] multipathd[77038]: unhandled signal 11 at 0000000000000000 nip 00007fff807f8148 lr 00007fff80820f70 code 1 [95374.466019] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA 7,[root@ibm-p9z-20-lp15 ~]# tail -n 100 /var/log/messages Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: 8:48: reinstated Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: mpatha: queue_if_no_path enabled Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Recovered to normal mode Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 1 Nov 14 13:53:25 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 2 service-time 0 2 1 8:48 1 8:16 1 service-time 0 2 1 8:64 1 8:32 1] Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: Asymmetric access state changed Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: Asymmetric access state changed Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: Asymmetric access state changed Nov 14 13:53:25 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA Nov 14 13:53:26 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 2 1 8:64 1 8:32 1 service-time 0 2 1 8:48 1 8:16 1] Nov 14 13:53:26 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA Nov 14 13:53:30 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: Asymmetric access state changed Nov 14 13:53:30 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA Nov 14 13:53:31 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: Asymmetric access state changed Nov 14 13:53:31 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA Nov 14 13:56:42 ibm-p9z-20-lp15 multipathd[77037]: Multipath configuration updated. Nov 14 13:56:42 ibm-p9z-20-lp15 multipathd[77037]: Reload multipathd for changes to take effect Nov 14 13:58:14 ibm-p9z-20-lp15 systemd[1]: Reloading Device-Mapper Multipath Device Controller. Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: reconfigure (operator) Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77850]: ok Nov 14 13:58:14 ibm-p9z-20-lp15 systemd[1]: Reloaded Device-Mapper Multipath Device Controller. Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: setting up map with 4/4 path checkers pending Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 2 1 8:48 1 8:16 1 service-time 0 2 1 8:64 1 8:32 1] Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Entering recovery mode: max_retries=60 Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sdd - tur checker reports path is up Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: 8:48: reinstated Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: queue_if_no_path enabled Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Recovered to normal mode Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 1 Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sde - tur checker reports path is up Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: 8:64: reinstated Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 2 Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: switch to path group #2 Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sdb - tur checker reports path is up Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: 8:16: reinstated Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 3 Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sdc - tur checker reports path is up Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: 8:32: reinstated Nov 14 13:58:14 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 4 Nov 14 13:58:15 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA Nov 14 13:58:20 ibm-p9z-20-lp15 systemd[1]: Reloading Device-Mapper Multipath Device Controller. Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: reconfigure (operator) Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77885]: ok Nov 14 13:58:21 ibm-p9z-20-lp15 systemd[1]: Reloaded Device-Mapper Multipath Device Controller. Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: setting up map with 4/4 path checkers pending Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 2 1 8:48 1 8:16 1 service-time 0 2 1 8:64 1 8:32 1] Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Entering recovery mode: max_retries=60 Nov 14 13:58:21 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA Nov 14 13:58:21 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA Nov 14 13:58:21 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: Asymmetric access state changed Nov 14 13:58:21 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: alua: port group 01 state A non-preferred supports tolUsNA Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: sdd - tur checker reports path is up Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: 8:48: reinstated Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: queue_if_no_path enabled Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: Recovered to normal mode Nov 14 13:58:21 ibm-p9z-20-lp15 multipathd[77037]: mpatha: remaining active paths: 1 Nov 14 13:58:22 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 2 service-time 0 2 1 8:48 1 8:16 1 service-time 0 2 1 8:64 1 8:32 1] Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: Asymmetric access state changed Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: Asymmetric access state changed Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state N preferred supports tolUsNA Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: Asymmetric access state changed Nov 14 13:58:22 ibm-p9z-20-lp15 kernel: sd 2:0:1:0: alua: port group 02 state A preferred supports tolUsNA Nov 14 13:58:23 ibm-p9z-20-lp15 multipathd[77037]: mpatha: load table [0 16777216 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 2 1 8:64 1 8:32 1 service-time 0 2 1 8:48 1 8:16 1] Nov 14 13:58:23 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA Nov 14 13:58:27 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: Asymmetric access state changed Nov 14 13:58:27 ibm-p9z-20-lp15 kernel: sd 1:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA Nov 14 13:58:32 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: Asymmetric access state changed Nov 14 13:58:32 ibm-p9z-20-lp15 kernel: sd 2:0:0:0: alua: port group 01 state N non-preferred supports tolUsNA Nov 14 13:58:48 ibm-p9z-20-lp15 kernel: multipathd[77038]: unhandled signal 11 at 0000000000000000 nip 00007fff807f8148 lr 00007fff80820f70 code 1 Nov 14 13:58:48 ibm-p9z-20-lp15 systemd[1]: Started Process Core Dump (PID 77991/UID 0). Nov 14 13:58:48 ibm-p9z-20-lp15 systemd[1]: multipathd.service: Main process exited, code=killed, status=11/SEGV Nov 14 13:58:48 ibm-p9z-20-lp15 systemd[1]: multipathd.service: Failed with result 'signal'. Nov 14 13:58:48 ibm-p9z-20-lp15 systemd-coredump[77992]: Process 77037 (multipathd) of user 0 dumped core.#012#012Stack trace of thread 77038:#012#0 0x00007fff807f8148 find_mp_by_wwid (libmultipath.so.0)#012#1 0x00007fff80820f70 snprint_config (libmultipath.so.0)#012#2 0x000000010f1043e8 show_config (multipathd)#012#3 0x000000010f105344 cli_list_config_local (multipathd)#012#4 0x000000010f10392c parse_cmd (multipathd)#012#5 0x000000010f0f8fd8 uxsock_trigger (multipathd)#012#6 0x000000010f101794 uxsock_listen (multipathd)#012#7 0x000000010f0fa67c uxlsnrloop (multipathd)#012#8 0x00007fff805d87c8 start_thread (libpthread.so.0)#012#9 0x00007fff801c0158 __clone (libc.so.6) Nov 14 13:58:49 ibm-p9z-20-lp15 systemd[1]: systemd-coredump: Succeeded. Nov 14 13:59:07 ibm-p9z-20-lp15 systemd[1]: Starting dnf makecache... Nov 14 13:59:07 ibm-p9z-20-lp15 systemd[1]: Starting Device-Mapper Multipath Device Controller... Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: --------start up-------- Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: read /etc/multipath.conf Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: path checkers start up Nov 14 13:59:08 ibm-p9z-20-lp15 systemd[1]: Started Device-Mapper Multipath Device Controller. Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: switch to path group #2 Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: sde - tur checker reports path is up Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: 8:64: reinstated Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: remaining active paths: 3 Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: switch to path group #1 Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: sdc - tur checker reports path is up Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: 8:32: reinstated Nov 14 13:59:08 ibm-p9z-20-lp15 multipathd[78016]: mpatha: remaining active paths: 4 Nov 14 13:59:08 ibm-p9z-20-lp15 kernel: sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA Nov 14 13:59:08 ibm-p9z-20-lp15 dnf[78010]: Updating Subscription Management repositories. Nov 14 13:59:08 ibm-p9z-20-lp15 dnf[78010]: Unable to read consumer identity Nov 14 13:59:08 ibm-p9z-20-lp15 dnf[78010]: This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Nov 14 13:59:08 ibm-p9z-20-lp15 dnf[78010]: Metadata cache refreshed recently. Nov 14 13:59:08 ibm-p9z-20-lp15 systemd[1]: dnf-makecache.service: Succeeded. Nov 14 13:59:08 ibm-p9z-20-lp15 systemd[1]: Started dnf makecache. Actual results: segfault happened. multipathd[77038]: unhandled signal 11 at 0000000000000000 nip 00007fff807f8148 lr 00007fff80820f70 code 1 Expected results: No segfault Additional info: This is a IBM ppc64le server. Vendor:DGC Product:VRAID [root@ibm-p9z-20-lp15 ~]# multipath -ll mpatha (3600601605e203f008e22d0221f07eb11) dm-0 DGC,VRAID size=8.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | |- 1:0:1:0 sde 8:64 active ready running | `- 2:0:1:0 sdc 8:32 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:0:0 sdd 8:48 active ready running `- 2:0:0:0 sdb 8:16 active ready running