Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1484952 Details for
Bug 1628529
upconvert attempt after secondary mirror leg failure causes clvmd to hang and gfs to withdraw
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
log of test run with "remove" policy
mckinley-01.log (text/plain), 31.11 KB, created by
Corey Marthaler
on 2018-09-19 21:56:25 UTC
(
hide
)
Description:
log of test run with "remove" policy
Filename:
MIME Type:
Creator:
Corey Marthaler
Created:
2018-09-19 21:56:25 UTC
Size:
31.11 KB
patch
obsolete
>Sep 19 16:32:20 mckinley-01 qarshd[9620]: Running cmdline: lvcreate -ay --type mirror -m 2 -n syncd_secondary_3legs_1 -L 500M helter_skelter /dev/mapper/mpathb1:0-2400 /dev/mapper/mpathg1:0-2400 /dev/mapper/mpathf1:0-2400 /dev/mapper/mpathc1:0-150 >Sep 19 16:32:20 mckinley-01 multipathd: dm-19: remove map (uevent) >Sep 19 16:32:20 mckinley-01 multipathd: dm-19: devmap not registered, can't remove >Sep 19 16:32:20 mckinley-01 multipathd: dm-19: remove map (uevent) >Sep 19 16:32:21 mckinley-01 lvm[6635]: Monitoring mirror device helter_skelter-syncd_secondary_3legs_1 for events. >Sep 19 16:32:21 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50214). >Sep 19 16:32:21 mckinley-01 qarshd[9691]: Talking to peer ::ffff:10.15.80.218:50214 (IPv6) >Sep 19 16:32:21 mckinley-01 qarshd[9691]: Running cmdline: lvs -a -o lv_name,lv_attr,lv_size,copy_percent,devices >Sep 19 16:32:26 mckinley-01 lvm[6635]: helter_skelter-syncd_secondary_3legs_1 is now in-sync. >Sep 19 16:32:31 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50220). >Sep 19 16:32:31 mckinley-01 qarshd[9792]: Talking to peer ::ffff:10.15.80.218:50220 (IPv6) >Sep 19 16:32:31 mckinley-01 qarshd[9792]: Running cmdline: dmsetup status helter_skelter-syncd_secondary_3legs_1-real >Sep 19 16:32:31 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50222). >Sep 19 16:32:31 mckinley-01 qarshd[9797]: Talking to peer ::ffff:10.15.80.218:50222 (IPv6) >Sep 19 16:32:32 mckinley-01 qarshd[9797]: Running cmdline: dmsetup status helter_skelter-syncd_secondary_3legs_1 >Sep 19 16:32:47 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50244). >Sep 19 16:32:47 mckinley-01 qarshd[9834]: Talking to peer ::ffff:10.15.80.218:50244 (IPv6) >Sep 19 16:32:47 mckinley-01 qarshd[9834]: Running cmdline: dmsetup status helter_skelter-syncd_secondary_3legs_1-real >Sep 19 16:32:47 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50246). >Sep 19 16:32:47 mckinley-01 qarshd[9839]: Talking to peer ::ffff:10.15.80.218:50246 (IPv6) >Sep 19 16:32:47 mckinley-01 qarshd[9839]: Running cmdline: dmsetup status helter_skelter-syncd_secondary_3legs_1 >Sep 19 16:33:02 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50258). >Sep 19 16:33:02 mckinley-01 qarshd[9898]: Talking to peer ::ffff:10.15.80.218:50258 (IPv6) >Sep 19 16:33:02 mckinley-01 qarshd[9898]: Running cmdline: dmsetup info >Sep 19 16:33:34 mckinley-01 systemd-logind: New session 3 of user root. >Sep 19 16:33:34 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50300). >Sep 19 16:33:34 mckinley-01 systemd: Started Session 3 of user root. >Sep 19 16:33:34 mckinley-01 qarshd[10038]: Talking to peer ::ffff:10.15.80.218:50300 (IPv6) >Sep 19 16:33:34 mckinley-01 qarshd[10038]: Running cmdline: pcs status >Sep 19 16:33:35 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50302). >Sep 19 16:33:35 mckinley-01 qarshd[10076]: Talking to peer ::ffff:10.15.80.218:50302 (IPv6) >Sep 19 16:33:35 mckinley-01 qarshd[10076]: Running cmdline: mkfs.gfs2 -o align=0 -J 32M -j 3 -p lock_dlm -t MCKINLEY:gfs1 /dev/helter_skelter/syncd_secondary_3legs_1 -O >Sep 19 16:33:38 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50312). >Sep 19 16:33:38 mckinley-01 qarshd[10085]: Talking to peer ::ffff:10.15.80.218:50312 (IPv6) >Sep 19 16:33:38 mckinley-01 qarshd[10085]: Running cmdline: mkdir -p /mnt/syncd_secondary_3legs_1 >Sep 19 16:33:38 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50314). >Sep 19 16:33:38 mckinley-01 qarshd[10090]: Talking to peer ::ffff:10.15.80.218:50314 (IPv6) >Sep 19 16:33:39 mckinley-01 qarshd[10090]: Running cmdline: mount /dev/helter_skelter/syncd_secondary_3legs_1 /mnt/syncd_secondary_3legs_1 >Sep 19 16:33:39 mckinley-01 kernel: GFS2 installed >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1: Trying to join cluster "lock_dlm", "MCKINLEY:gfs1" >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1: first mounter control generation 0 >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1: Joined cluster. Now mounting FS... >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=0, already locked for use >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=0: Looking at journal... >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=0: Done >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=1: Trying to acquire journal lock... >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=1: Looking at journal... >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=1: Done >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=2: Trying to acquire journal lock... >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=2: Looking at journal... >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=2: Done >Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: first mount done, others may mount >Sep 19 16:33:40 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: recover generation 2 done >Sep 19 16:33:41 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: recover generation 3 done >Sep 19 16:33:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50328). >Sep 19 16:33:42 mckinley-01 qarshd[10110]: Talking to peer ::ffff:10.15.80.218:50328 (IPv6) >Sep 19 16:33:42 mckinley-01 qarshd[10110]: Running cmdline: lvs -a -o +devices >Sep 19 16:33:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50330). >Sep 19 16:33:43 mckinley-01 qarshd[10116]: Talking to peer ::ffff:10.15.80.218:50330 (IPv6) >Sep 19 16:33:43 mckinley-01 qarshd[10116]: Running cmdline: lvs --noheadings -a -o name,devices >Sep 19 16:33:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50332). >Sep 19 16:33:43 mckinley-01 qarshd[10122]: Talking to peer ::ffff:10.15.80.218:50332 (IPv6) >Sep 19 16:33:43 mckinley-01 qarshd[10122]: Running cmdline: lvs -a -o +devices >Sep 19 16:33:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50334). >Sep 19 16:33:43 mckinley-01 qarshd[10140]: Talking to peer ::ffff:10.15.80.218:50334 (IPv6) >Sep 19 16:33:43 mckinley-01 qarshd[10140]: Running cmdline: lvs -a -o +devices >Sep 19 16:33:44 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50336). >Sep 19 16:33:44 mckinley-01 qarshd[10146]: Talking to peer ::ffff:10.15.80.218:50336 (IPv6) >Sep 19 16:33:44 mckinley-01 qarshd[10146]: Running cmdline: lvs --noheadings -a -o name,devices >Sep 19 16:33:44 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50338). >Sep 19 16:33:44 mckinley-01 qarshd[10152]: Talking to peer ::ffff:10.15.80.218:50338 (IPv6) >Sep 19 16:33:44 mckinley-01 qarshd[10152]: Running cmdline: lvs -a -o +devices >Sep 19 16:33:44 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50340). >Sep 19 16:33:44 mckinley-01 qarshd[10168]: Talking to peer ::ffff:10.15.80.218:50340 (IPv6) >Sep 19 16:33:45 mckinley-01 qarshd[10168]: Running cmdline: mkdir -p /mnt/syncd_secondary_3legs_1/checkit >Sep 19 16:33:45 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50342). >Sep 19 16:33:45 mckinley-01 qarshd[10174]: Talking to peer ::ffff:10.15.80.218:50342 (IPv6) >Sep 19 16:33:45 mckinley-01 qarshd[10174]: Running cmdline: /usr/tests/sts-rhel7.6/bin/checkit -w /mnt/syncd_secondary_3legs_1/checkit -f /tmp/checkit_syncd_secondary_3legs_1 -n 100 >Sep 19 16:33:49 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50360). >Sep 19 16:33:49 mckinley-01 qarshd[10186]: Talking to peer ::ffff:10.15.80.218:50360 (IPv6) >Sep 19 16:33:49 mckinley-01 qarshd[10186]: Running cmdline: /usr/tests/sts-rhel7.6/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/syncd_secondary_3legs_1/mckinley-01_load | /usr/tests/sts-rhel7.6/bin/xdoio -v >Sep 19 16:34:06 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50408). >Sep 19 16:34:06 mckinley-01 qarshd[10292]: Talking to peer ::ffff:10.15.80.218:50408 (IPv6) >Sep 19 16:34:06 mckinley-01 qarshd[10292]: Running cmdline: dmstats create --filemap /mnt/syncd_secondary_3legs_1/mckinley-01_load >Sep 19 16:34:07 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50410). >Sep 19 16:34:07 mckinley-01 qarshd[10303]: Talking to peer ::ffff:10.15.80.218:50410 (IPv6) >Sep 19 16:34:07 mckinley-01 qarshd[10303]: Running cmdline: /usr/tests/sts-rhel7.6/bin/checkit -w /mnt/syncd_secondary_3legs_1/checkit -f /tmp/checkit_syncd_secondary_3legs_1 -v >Sep 19 16:34:13 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50432). >Sep 19 16:34:13 mckinley-01 qarshd[10314]: Talking to peer ::ffff:10.15.80.218:50432 (IPv6) >Sep 19 16:34:14 mckinley-01 qarshd[10314]: Running cmdline: dmstats report --group >Sep 19 16:34:14 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50434). >Sep 19 16:34:14 mckinley-01 qarshd[10318]: Talking to peer ::ffff:10.15.80.218:50434 (IPv6) >Sep 19 16:34:14 mckinley-01 qarshd[10318]: Running cmdline: dmstats list --group >Sep 19 16:34:14 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50436). >Sep 19 16:34:14 mckinley-01 qarshd[10322]: Talking to peer ::ffff:10.15.80.218:50436 (IPv6) >Sep 19 16:34:14 mckinley-01 qarshd[10322]: Running cmdline: dmstats delete /dev/helter_skelter/syncd_secondary_3legs_1 --allregions >Sep 19 16:34:14 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50438). >Sep 19 16:34:14 mckinley-01 qarshd[10326]: Talking to peer ::ffff:10.15.80.218:50438 (IPv6) >Sep 19 16:34:14 mckinley-01 qarshd[10326]: Running cmdline: tail -n1 /var/log/messages | cut -c 1-12 >Sep 19 16:34:15 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50440). >Sep 19 16:34:15 mckinley-01 qarshd[10332]: Talking to peer ::ffff:10.15.80.218:50440 (IPv6) >Sep 19 16:34:15 mckinley-01 qarshd[10332]: Running cmdline: multipathd fail path sdh >Sep 19 16:34:15 mckinley-01 multipathd: mpathg: fail path 8:112 (operator) >Sep 19 16:34:15 mckinley-01 kernel: device-mapper: multipath: Failing path 8:112. >Sep 19 16:34:15 mckinley-01 multipathd: sdh: mark as failed >Sep 19 16:34:15 mckinley-01 multipathd: mpathg: remaining active paths: 3 >Sep 19 16:34:15 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50442). >Sep 19 16:34:15 mckinley-01 qarshd[10339]: Talking to peer ::ffff:10.15.80.218:50442 (IPv6) >Sep 19 16:34:15 mckinley-01 qarshd[10339]: Running cmdline: multipathd fail path sdp >Sep 19 16:34:15 mckinley-01 multipathd: mpathg: fail path 8:240 (operator) >Sep 19 16:34:15 mckinley-01 kernel: device-mapper: multipath: Failing path 8:240. >Sep 19 16:34:15 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50444). >Sep 19 16:34:15 mckinley-01 qarshd[10344]: Talking to peer ::ffff:10.15.80.218:50444 (IPv6) >Sep 19 16:34:15 mckinley-01 qarshd[10344]: Running cmdline: multipathd fail path sdx >Sep 19 16:34:15 mckinley-01 multipathd: mpathg: fail path 65:112 (operator) >Sep 19 16:34:15 mckinley-01 kernel: device-mapper: multipath: Failing path 65:112. >Sep 19 16:34:15 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50446). >Sep 19 16:34:15 mckinley-01 qarshd[10353]: Talking to peer ::ffff:10.15.80.218:50446 (IPv6) >Sep 19 16:34:15 mckinley-01 qarshd[10353]: Running cmdline: multipathd fail path sdaf >Sep 19 16:34:15 mckinley-01 multipathd: mpathg: fail path 65:240 (operator) >Sep 19 16:34:15 mckinley-01 kernel: device-mapper: multipath: Failing path 65:240. >Sep 19 16:34:16 mckinley-01 multipathd: sdaf: mark as failed >Sep 19 16:34:16 mckinley-01 multipathd: mpathg: remaining active paths: 2 >Sep 19 16:34:16 mckinley-01 multipathd: sdp: mark as failed >Sep 19 16:34:16 mckinley-01 multipathd: mpathg: remaining active paths: 1 >Sep 19 16:34:16 mckinley-01 multipathd: sdx: mark as failed >Sep 19 16:34:16 mckinley-01 multipathd: mpathg: remaining active paths: 0 >Sep 19 16:34:19 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50470). >Sep 19 16:34:19 mckinley-01 qarshd[10358]: Talking to peer ::ffff:10.15.80.218:50470 (IPv6) >Sep 19 16:34:19 mckinley-01 qarshd[10358]: Running cmdline: pvs -a >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:19 mckinley-01 multipathd: sdh: unusable path >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:19 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50472). >Sep 19 16:34:19 mckinley-01 qarshd[10377]: Talking to peer ::ffff:10.15.80.218:50472 (IPv6) >Sep 19 16:34:19 mckinley-01 qarshd[10377]: Running cmdline: dd if=/dev/zero of=/mnt/syncd_secondary_3legs_1/ddfile count=10 bs=4M >Sep 19 16:34:20 mckinley-01 lvm[6635]: Secondary mirror device 253:21 has failed (D). >Sep 19 16:34:20 mckinley-01 lvm[6635]: Device failure in helter_skelter-syncd_secondary_3legs_1. >Sep 19 16:34:20 mckinley-01 lvm[6635]: WARNING: Disabling lvmetad cache for repair command. >Sep 19 16:34:20 mckinley-01 lvm[6635]: WARNING: Not using lvmetad because of repair. >Sep 19 16:34:20 mckinley-01 lvm[6635]: Couldn't find device with uuid Fj3xGt-mM7b-l6BF-q8KQ-ji7m-1jFZ-YVihZL. >Sep 19 16:34:20 mckinley-01 lvm[6635]: WARNING: Couldn't find all devices for LV helter_skelter/syncd_secondary_3legs_1_mimage_1 while checking used and assumed devices. >Sep 19 16:34:20 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50474). >Sep 19 16:34:20 mckinley-01 qarshd[10399]: Talking to peer ::ffff:10.15.80.218:50474 (IPv6) >Sep 19 16:34:20 mckinley-01 qarshd[10399]: Running cmdline: sync >Sep 19 16:34:20 mckinley-01 multipathd: dm-21: remove map (uevent) >Sep 19 16:34:20 mckinley-01 multipathd: dm-21: devmap not registered, can't remove >Sep 19 16:34:20 mckinley-01 multipathd: dm-21: remove map (uevent) >Sep 19 16:34:20 mckinley-01 multipathd: sdp: unusable path >Sep 19 16:34:20 mckinley-01 multipathd: sdx: unusable path >Sep 19 16:34:20 mckinley-01 multipathd: sdaf: unusable path >Sep 19 16:34:20 mckinley-01 multipathd: dm-24: remove map (uevent) >Sep 19 16:34:20 mckinley-01 multipathd: dm-24: devmap not registered, can't remove >Sep 19 16:34:20 mckinley-01 multipathd: dm-24: remove map (uevent) >Sep 19 16:34:23 mckinley-01 lvm[6635]: Mirror status: 1 of 3 images failed. >Sep 19 16:34:23 mckinley-01 lvm[6635]: Repair of mirrored device helter_skelter-syncd_secondary_3legs_1 finished successfully. >Sep 19 16:34:23 mckinley-01 lvm[6635]: helter_skelter-syncd_secondary_3legs_1 is now in-sync. >Sep 19 16:34:24 mckinley-01 multipathd: sdh: unusable path >Sep 19 16:34:25 mckinley-01 multipathd: sdp: unusable path >Sep 19 16:34:25 mckinley-01 multipathd: sdx: unusable path >Sep 19 16:34:25 mckinley-01 multipathd: sdaf: unusable path >Sep 19 16:34:26 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50486). >Sep 19 16:34:26 mckinley-01 qarshd[10502]: Talking to peer ::ffff:10.15.80.218:50486 (IPv6) >Sep 19 16:34:26 mckinley-01 qarshd[10502]: Running cmdline: lvs > /dev/null 2>&1 >Sep 19 16:34:26 mckinley-01 kernel: blk_update_request: 22 callbacks suppressed >Sep 19 16:34:26 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:27 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50488). >Sep 19 16:34:27 mckinley-01 qarshd[10510]: Talking to peer ::ffff:10.15.80.218:50488 (IPv6) >Sep 19 16:34:27 mckinley-01 qarshd[10510]: Running cmdline: lvs -a -o lv_name,lv_attr,lv_size,copy_percent,devices >Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:27 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50490). >Sep 19 16:34:27 mckinley-01 qarshd[10518]: Talking to peer ::ffff:10.15.80.218:50490 (IPv6) >Sep 19 16:34:27 mckinley-01 qarshd[10518]: Running cmdline: lvs -a -o +devices >Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:27 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50492). >Sep 19 16:34:27 mckinley-01 qarshd[10524]: Talking to peer ::ffff:10.15.80.218:50492 (IPv6) >Sep 19 16:34:27 mckinley-01 qarshd[10524]: Running cmdline: lvs -a -o +devices >Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:28 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50494). >Sep 19 16:34:28 mckinley-01 qarshd[10530]: Talking to peer ::ffff:10.15.80.218:50494 (IPv6) >Sep 19 16:34:28 mckinley-01 qarshd[10530]: Running cmdline: lvs -a -o +devices >Sep 19 16:34:28 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:28 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40 >Sep 19 16:34:28 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50496). >Sep 19 16:34:28 mckinley-01 qarshd[10535]: Talking to peer ::ffff:10.15.80.218:50496 (IPv6) >Sep 19 16:34:28 mckinley-01 qarshd[10535]: Running cmdline: lvs -a -o +devices >Sep 19 16:34:28 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:28 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50498). >Sep 19 16:34:28 mckinley-01 qarshd[10540]: Talking to peer ::ffff:10.15.80.218:50498 (IPv6) >Sep 19 16:34:28 mckinley-01 qarshd[10540]: Running cmdline: dmsetup ls | grep syncd_secondary_3legs_1_mimage_1 >Sep 19 16:34:29 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50504). >Sep 19 16:34:29 mckinley-01 qarshd[10546]: Talking to peer ::ffff:10.15.80.218:50504 (IPv6) >Sep 19 16:34:29 mckinley-01 multipathd: sdh: unusable path >Sep 19 16:34:29 mckinley-01 qarshd[10546]: Running cmdline: lvs --noheadings -a -o +devices >Sep 19 16:34:29 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50506). >Sep 19 16:34:29 mckinley-01 qarshd[10578]: Talking to peer ::ffff:10.15.80.218:50506 (IPv6) >Sep 19 16:34:30 mckinley-01 qarshd[10578]: Running cmdline: lvs --noheadings -a -o +devices >Sep 19 16:34:30 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50508). >Sep 19 16:34:30 mckinley-01 qarshd[10583]: Talking to peer ::ffff:10.15.80.218:50508 (IPv6) >Sep 19 16:34:30 mckinley-01 qarshd[10583]: Running cmdline: lvs --noheadings -a -o +devices >Sep 19 16:34:30 mckinley-01 multipathd: sdp: unusable path >Sep 19 16:34:30 mckinley-01 multipathd: sdx: unusable path >Sep 19 16:34:30 mckinley-01 multipathd: sdaf: unusable path >Sep 19 16:34:34 mckinley-01 multipathd: sdh: unusable path >Sep 19 16:34:35 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50510). >Sep 19 16:34:35 mckinley-01 kernel: blk_update_request: 7 callbacks suppressed >Sep 19 16:34:35 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0 >Sep 19 16:34:35 mckinley-01 qarshd[10607]: Talking to peer ::ffff:10.15.80.218:50510 (IPv6) >Sep 19 16:34:35 mckinley-01 multipathd: sdp: unusable path >Sep 19 16:34:35 mckinley-01 multipathd: sdx: unusable path >Sep 19 16:34:35 mckinley-01 multipathd: sdaf: unusable path >Sep 19 16:34:35 mckinley-01 qarshd[10607]: Running cmdline: /usr/tests/sts-rhel7.6/bin/checkit -w /mnt/syncd_secondary_3legs_1/checkit -f /tmp/checkit_syncd_secondary_3legs_1 -v >Sep 19 16:34:39 mckinley-01 multipathd: sdh: unusable path >Sep 19 16:34:40 mckinley-01 multipathd: sdp: unusable path >Sep 19 16:34:40 mckinley-01 multipathd: sdx: unusable path >Sep 19 16:34:40 mckinley-01 multipathd: sdaf: unusable path >Sep 19 16:34:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50520). >Sep 19 16:34:42 mckinley-01 qarshd[10613]: Talking to peer ::ffff:10.15.80.218:50520 (IPv6) >Sep 19 16:34:42 mckinley-01 qarshd[10613]: Running cmdline: dmstats report --group >Sep 19 16:34:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50522). >Sep 19 16:34:42 mckinley-01 qarshd[10617]: Talking to peer ::ffff:10.15.80.218:50522 (IPv6) >Sep 19 16:34:42 mckinley-01 qarshd[10617]: Running cmdline: dmstats list --group >Sep 19 16:34:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50524). >Sep 19 16:34:42 mckinley-01 qarshd[10624]: Talking to peer ::ffff:10.15.80.218:50524 (IPv6) >Sep 19 16:34:42 mckinley-01 qarshd[10624]: Running cmdline: dmstats delete /dev/helter_skelter/syncd_secondary_3legs_1 --allregions >Sep 19 16:34:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50526). >Sep 19 16:34:42 mckinley-01 qarshd[10628]: Talking to peer ::ffff:10.15.80.218:50526 (IPv6) >Sep 19 16:34:43 mckinley-01 qarshd[10628]: Running cmdline: multipathd reinstate path sdh >Sep 19 16:34:43 mckinley-01 multipathd: mpathg: reinstate path 8:112 (operator) >Sep 19 16:34:43 mckinley-01 kernel: device-mapper: multipath: Reinstating path 8:112. >Sep 19 16:34:43 mckinley-01 kernel: sd 7:0:0:6: alua: port group 01 state A preferred supports tolUsNA >Sep 19 16:34:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50528). >Sep 19 16:34:43 mckinley-01 qarshd[10640]: Talking to peer ::ffff:10.15.80.218:50528 (IPv6) >Sep 19 16:34:43 mckinley-01 qarshd[10640]: Running cmdline: multipathd reinstate path sdp >Sep 19 16:34:43 mckinley-01 multipathd: mpathg: reinstate path 8:240 (operator) >Sep 19 16:34:43 mckinley-01 kernel: device-mapper: multipath: Reinstating path 8:240. >Sep 19 16:34:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50530). >Sep 19 16:34:43 mckinley-01 qarshd[10645]: Talking to peer ::ffff:10.15.80.218:50530 (IPv6) >Sep 19 16:34:43 mckinley-01 qarshd[10645]: Running cmdline: multipathd reinstate path sdx >Sep 19 16:34:43 mckinley-01 multipathd: mpathg: reinstate path 65:112 (operator) >Sep 19 16:34:43 mckinley-01 kernel: device-mapper: multipath: Reinstating path 65:112. >Sep 19 16:34:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50532). >Sep 19 16:34:43 mckinley-01 qarshd[10662]: Talking to peer ::ffff:10.15.80.218:50532 (IPv6) >Sep 19 16:34:43 mckinley-01 qarshd[10662]: Running cmdline: multipathd reinstate path sdaf >Sep 19 16:34:43 mckinley-01 multipathd: mpathg: reinstate path 65:240 (operator) >Sep 19 16:34:43 mckinley-01 kernel: device-mapper: multipath: Reinstating path 65:240. >Sep 19 16:34:43 mckinley-01 kernel: sd 7:0:3:6: alua: port group 01 state A preferred supports tolUsNA >Sep 19 16:34:44 mckinley-01 multipathd: mpathg: sdh - tur checker reports path is up >Sep 19 16:34:44 mckinley-01 multipathd: 8:112: reinstated >Sep 19 16:34:44 mckinley-01 multipathd: mpathg: remaining active paths: 1 >Sep 19 16:34:45 mckinley-01 multipathd: mpathg: sdp - tur checker reports path is up >Sep 19 16:34:45 mckinley-01 multipathd: 8:240: reinstated >Sep 19 16:34:45 mckinley-01 multipathd: mpathg: remaining active paths: 2 >Sep 19 16:34:45 mckinley-01 multipathd: mpathg: sdx - tur checker reports path is up >Sep 19 16:34:45 mckinley-01 multipathd: 65:112: reinstated >Sep 19 16:34:45 mckinley-01 multipathd: mpathg: remaining active paths: 3 >Sep 19 16:34:45 mckinley-01 multipathd: mpathg: sdaf - tur checker reports path is up >Sep 19 16:34:45 mckinley-01 multipathd: 65:240: reinstated >Sep 19 16:34:45 mckinley-01 multipathd: mpathg: remaining active paths: 4 >Sep 19 16:34:56 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50566). >Sep 19 16:34:56 mckinley-01 qarshd[10706]: Talking to peer ::ffff:10.15.80.218:50566 (IPv6) >Sep 19 16:34:56 mckinley-01 qarshd[10706]: Running cmdline: vgreduce --removemissing --force helter_skelter >Sep 19 16:34:56 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50572). >Sep 19 16:34:56 mckinley-01 qarshd[10734]: Talking to peer ::ffff:10.15.80.218:50572 (IPv6) >Sep 19 16:34:56 mckinley-01 qarshd[10734]: Running cmdline: pvcreate /dev/mapper/mpathg1 >Sep 19 16:34:57 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50574). >Sep 19 16:34:57 mckinley-01 qarshd[10747]: Talking to peer ::ffff:10.15.80.218:50574 (IPv6) >Sep 19 16:34:57 mckinley-01 qarshd[10747]: Running cmdline: vgextend helter_skelter /dev/mapper/mpathg1 >Sep 19 16:34:57 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50576). >Sep 19 16:34:57 mckinley-01 qarshd[10761]: Talking to peer ::ffff:10.15.80.218:50576 (IPv6) > > >#### Everything is fine until the test runs its own upconvert due to the (remove policy), which is exactly what lvm does in alloc failure policy > > >Sep 19 16:34:57 mckinley-01 qarshd[10761]: Running cmdline: lvconvert --yes --type mirror --mirrorlog disk -m 2 helter_skelter/syncd_secondary_3legs_1 /dev/mapper/mpathb1:0-2400 /dev/mapper/mpathg1:0-2400 /dev/mapper/mpathf1:0-2400 /dev/mapper/mpathc1:0-150 >Sep 19 16:34:58 mckinley-01 dmeventd[6635]: No longer monitoring mirror device helter_skelter-syncd_secondary_3legs_1 for events. >Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: Error -5 writing to journal, jid=0 >Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: error -5: withdrawing the file system to prevent further damage. >Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: about to withdraw this file system >Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: telling LM to unmount >Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: withdrawn >Sep 19 16:34:58 mckinley-01 kernel: CPU: 1 PID: 10101 Comm: gfs2_logd Kdump: loaded Not tainted 3.10.0-951.el7.x86_64 #1 >Sep 19 16:34:58 mckinley-01 kernel: Hardware name: Dell Inc. PowerEdge R820/0RN9TC, BIOS 2.0.20 01/16/2014 >Sep 19 16:34:58 mckinley-01 kernel: Call Trace: >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff97b61ca8>] dump_stack+0x19/0x1b >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffffc09c7206>] gfs2_lm_withdraw+0x146/0x180 [gfs2] >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffffc09abe15>] gfs2_logd+0x315/0x340 [gfs2] >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff974c2d60>] ? wake_up_atomic_t+0x30/0x30 >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffffc09abb00>] ? gfs2_meta_syncfs+0x50/0x50 [gfs2] >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff974c1c91>] kthread+0xd1/0xe0 >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff974c1bc0>] ? insert_kthread_work+0x40/0x40 >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff97b74c37>] ret_from_fork_nospec_begin+0x21/0x21 >Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff974c1bc0>] ? insert_kthread_work+0x40/0x40 >Sep 19 16:35:01 mckinley-01 lvm[6635]: Monitoring mirror device helter_skelter-syncd_secondary_3legs_1_mimagetmp_3 for events. >Sep 19 16:35:01 mckinley-01 lvm[6635]: Monitoring mirror device helter_skelter-syncd_secondary_3legs_1 for events. >Sep 19 16:35:01 mckinley-01 systemd: Started LVM2 poll daemon. >Sep 19 16:35:02 mckinley-01 qarshd[10186]: Sending child 10187 signal 2 >Sep 19 16:35:08 mckinley-01 qarshd[10186]: Sending child 10187 signal 15 >Sep 19 16:35:17 mckinley-01 lvm[6635]: No longer monitoring mirror device helter_skelter-syncd_secondary_3legs_1_mimagetmp_3 for events. >Sep 19 16:35:17 mckinley-01 dmeventd[6635]: No longer monitoring mirror device helter_skelter-syncd_secondary_3legs_1 for events. >Sep 19 16:36:17 mckinley-01 lvmpolld: W: LVMPOLLD: polling for output of the lvm cmd (PID 10867) has timed out >Sep 19 16:36:32 mckinley-01 lvmpolld: W: #011LVPOLL: PID 10867: STDERR: ' Error locking on node 1: Command timed out' >Sep 19 16:36:44 mckinley-01 systemd: qarshd@268-10.15.104.61:5016-10.15.80.218:50360.service stop-sigterm timed out. Killing. >Sep 19 16:36:59 mckinley-01 kernel: INFO: task xdoio:10190 blocked for more than 120 seconds. >Sep 19 16:36:59 mckinley-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >Sep 19 16:36:59 mckinley-01 kernel: xdoio D ffff8fc97b3f9040 0 10190 1 0x00000084 >Sep 19 16:36:59 mckinley-01 kernel: Call Trace: >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97678461>] ? __block_commit_write.isra.20+0x81/0xb0 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b65b70>] ? out_of_line_wait_on_atomic_t+0x100/0x100 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b67aa9>] schedule+0x29/0x70 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b65b81>] bit_wait+0x11/0x50 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b656f7>] __wait_on_bit+0x67/0x90 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97644091>] ? __sb_end_write+0x31/0x60 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b65b70>] ? out_of_line_wait_on_atomic_t+0x100/0x100 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b65861>] out_of_line_wait_on_bit+0x81/0xb0 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff974c2e20>] ? wake_bit_function+0x40/0x40 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09a6c3e>] gfs2_glock_wait+0x3e/0x80 [gfs2] >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09a8720>] gfs2_glock_nq+0x250/0x420 [gfs2] >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09c6796>] gfs2_trans_begin+0xf6/0x1b0 [gfs2] >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09c523f>] gfs2_dirty_inode+0x1bf/0x280 [gfs2] >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09c526b>] ? gfs2_dirty_inode+0x1eb/0x280 [gfs2] >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff9766fedd>] __mark_inode_dirty+0x16d/0x270 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff9765dfd9>] update_time+0x89/0xd0 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff9765e0c0>] file_update_time+0xa0/0xf0 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff975b8668>] __generic_file_aio_write+0x198/0x400 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff975b8929>] generic_file_aio_write+0x59/0xa0 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09b1130>] gfs2_file_aio_write+0xd0/0x160 [gfs2] >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97640613>] do_sync_write+0x93/0xe0 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97641100>] vfs_write+0xc0/0x1f0 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b74d21>] ? system_call_after_swapgs+0xae/0x146 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97641f1f>] SyS_write+0x7f/0xf0 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b74d21>] ? system_call_after_swapgs+0xae/0x146 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b74ddb>] system_call_fastpath+0x22/0x27 >Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b74d21>] ? system_call_after_swapgs+0xae/0x146 >Sep 19 16:37:32 mckinley-01 lvmpolld: W: LVMPOLLD: polling for output of the lvm cmd (PID 10867) has timed out >Sep 19 16:38:14 mckinley-01 systemd: qarshd@268-10.15.104.61:5016-10.15.80.218:50360.service still around after SIGKILL. Ignoring. >Sep 19 16:38:32 mckinley-01 lvmpolld: W: LVMPOLLD: polling for output of the lvm cmd (PID 10867) has timed out >Sep 19 16:38:32 mckinley-01 lvmpolld: W: #011LVPOLL: PID 10867: STDERR: ' Failed to lock logical volume helter_skelter/syncd_secondary_3legs_1.' >Sep 19 16:38:32 mckinley-01 lvmpolld: W: #011LVPOLL: PID 10867: STDERR: ' Error locking on node 1: Command timed out'
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1628529
:
1483035
| 1484952 |
1484953
|
1485256
|
1486443
|
1487280
|
1487285
|
1487286
|
1487287
|
1487288
|
1492603