Attachment 1484952 Details for Bug 1628529 – log of test run with "remove" policy

log of test run with "remove" policy

mckinley-01.log (text/plain), 31.11 KB, created by Corey Marthaler on 2018-09-19 21:56:25 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2018-09-19 21:56:25 UTC

Size: 31.11 KB

patch

obsolete

>Sep 19 16:32:20 mckinley-01 qarshd[9620]: Running cmdline: lvcreate -ay --type mirror -m 2 -n syncd_secondary_3legs_1 -L 500M helter_skelter /dev/mapper/mpathb1:0-2400 /dev/mapper/mpathg1:0-2400 /dev/mapper/mpathf1:0-2400 /dev/mapper/mpathc1:0-150
>Sep 19 16:32:20 mckinley-01 multipathd: dm-19: remove map (uevent)
>Sep 19 16:32:20 mckinley-01 multipathd: dm-19: devmap not registered, can't remove
>Sep 19 16:32:20 mckinley-01 multipathd: dm-19: remove map (uevent)
>Sep 19 16:32:21 mckinley-01 lvm[6635]: Monitoring mirror device helter_skelter-syncd_secondary_3legs_1 for events.
>Sep 19 16:32:21 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50214).
>Sep 19 16:32:21 mckinley-01 qarshd[9691]: Talking to peer ::ffff:10.15.80.218:50214 (IPv6)
>Sep 19 16:32:21 mckinley-01 qarshd[9691]: Running cmdline: lvs -a -o lv_name,lv_attr,lv_size,copy_percent,devices
>Sep 19 16:32:26 mckinley-01 lvm[6635]: helter_skelter-syncd_secondary_3legs_1 is now in-sync.
>Sep 19 16:32:31 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50220).
>Sep 19 16:32:31 mckinley-01 qarshd[9792]: Talking to peer ::ffff:10.15.80.218:50220 (IPv6)
>Sep 19 16:32:31 mckinley-01 qarshd[9792]: Running cmdline: dmsetup status helter_skelter-syncd_secondary_3legs_1-real
>Sep 19 16:32:31 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50222).
>Sep 19 16:32:31 mckinley-01 qarshd[9797]: Talking to peer ::ffff:10.15.80.218:50222 (IPv6)
>Sep 19 16:32:32 mckinley-01 qarshd[9797]: Running cmdline: dmsetup status helter_skelter-syncd_secondary_3legs_1
>Sep 19 16:32:47 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50244).
>Sep 19 16:32:47 mckinley-01 qarshd[9834]: Talking to peer ::ffff:10.15.80.218:50244 (IPv6)
>Sep 19 16:32:47 mckinley-01 qarshd[9834]: Running cmdline: dmsetup status helter_skelter-syncd_secondary_3legs_1-real
>Sep 19 16:32:47 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50246).
>Sep 19 16:32:47 mckinley-01 qarshd[9839]: Talking to peer ::ffff:10.15.80.218:50246 (IPv6)
>Sep 19 16:32:47 mckinley-01 qarshd[9839]: Running cmdline: dmsetup status helter_skelter-syncd_secondary_3legs_1
>Sep 19 16:33:02 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50258).
>Sep 19 16:33:02 mckinley-01 qarshd[9898]: Talking to peer ::ffff:10.15.80.218:50258 (IPv6)
>Sep 19 16:33:02 mckinley-01 qarshd[9898]: Running cmdline: dmsetup info
>Sep 19 16:33:34 mckinley-01 systemd-logind: New session 3 of user root.
>Sep 19 16:33:34 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50300).
>Sep 19 16:33:34 mckinley-01 systemd: Started Session 3 of user root.
>Sep 19 16:33:34 mckinley-01 qarshd[10038]: Talking to peer ::ffff:10.15.80.218:50300 (IPv6)
>Sep 19 16:33:34 mckinley-01 qarshd[10038]: Running cmdline: pcs status
>Sep 19 16:33:35 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50302).
>Sep 19 16:33:35 mckinley-01 qarshd[10076]: Talking to peer ::ffff:10.15.80.218:50302 (IPv6)
>Sep 19 16:33:35 mckinley-01 qarshd[10076]: Running cmdline: mkfs.gfs2 -o align=0 -J 32M -j 3 -p lock_dlm -t MCKINLEY:gfs1 /dev/helter_skelter/syncd_secondary_3legs_1 -O
>Sep 19 16:33:38 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50312).
>Sep 19 16:33:38 mckinley-01 qarshd[10085]: Talking to peer ::ffff:10.15.80.218:50312 (IPv6)
>Sep 19 16:33:38 mckinley-01 qarshd[10085]: Running cmdline: mkdir -p /mnt/syncd_secondary_3legs_1
>Sep 19 16:33:38 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50314).
>Sep 19 16:33:38 mckinley-01 qarshd[10090]: Talking to peer ::ffff:10.15.80.218:50314 (IPv6)
>Sep 19 16:33:39 mckinley-01 qarshd[10090]: Running cmdline: mount /dev/helter_skelter/syncd_secondary_3legs_1 /mnt/syncd_secondary_3legs_1
>Sep 19 16:33:39 mckinley-01 kernel: GFS2 installed
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1: Trying to join cluster "lock_dlm", "MCKINLEY:gfs1"
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1: first mounter control generation 0
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1: Joined cluster. Now mounting FS...
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=0, already locked for use
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=0: Looking at journal...
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=0: Done
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=1: Trying to acquire journal lock...
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=1: Looking at journal...
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=1: Done
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=2: Trying to acquire journal lock...
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=2: Looking at journal...
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: jid=2: Done
>Sep 19 16:33:39 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: first mount done, others may mount
>Sep 19 16:33:40 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: recover generation 2 done
>Sep 19 16:33:41 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: recover generation 3 done
>Sep 19 16:33:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50328).
>Sep 19 16:33:42 mckinley-01 qarshd[10110]: Talking to peer ::ffff:10.15.80.218:50328 (IPv6)
>Sep 19 16:33:42 mckinley-01 qarshd[10110]: Running cmdline: lvs -a -o +devices
>Sep 19 16:33:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50330).
>Sep 19 16:33:43 mckinley-01 qarshd[10116]: Talking to peer ::ffff:10.15.80.218:50330 (IPv6)
>Sep 19 16:33:43 mckinley-01 qarshd[10116]: Running cmdline: lvs --noheadings -a -o name,devices
>Sep 19 16:33:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50332).
>Sep 19 16:33:43 mckinley-01 qarshd[10122]: Talking to peer ::ffff:10.15.80.218:50332 (IPv6)
>Sep 19 16:33:43 mckinley-01 qarshd[10122]: Running cmdline: lvs -a -o +devices
>Sep 19 16:33:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50334).
>Sep 19 16:33:43 mckinley-01 qarshd[10140]: Talking to peer ::ffff:10.15.80.218:50334 (IPv6)
>Sep 19 16:33:43 mckinley-01 qarshd[10140]: Running cmdline: lvs -a -o +devices
>Sep 19 16:33:44 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50336).
>Sep 19 16:33:44 mckinley-01 qarshd[10146]: Talking to peer ::ffff:10.15.80.218:50336 (IPv6)
>Sep 19 16:33:44 mckinley-01 qarshd[10146]: Running cmdline: lvs --noheadings -a -o name,devices
>Sep 19 16:33:44 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50338).
>Sep 19 16:33:44 mckinley-01 qarshd[10152]: Talking to peer ::ffff:10.15.80.218:50338 (IPv6)
>Sep 19 16:33:44 mckinley-01 qarshd[10152]: Running cmdline: lvs -a -o +devices
>Sep 19 16:33:44 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50340).
>Sep 19 16:33:44 mckinley-01 qarshd[10168]: Talking to peer ::ffff:10.15.80.218:50340 (IPv6)
>Sep 19 16:33:45 mckinley-01 qarshd[10168]: Running cmdline: mkdir -p /mnt/syncd_secondary_3legs_1/checkit
>Sep 19 16:33:45 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50342).
>Sep 19 16:33:45 mckinley-01 qarshd[10174]: Talking to peer ::ffff:10.15.80.218:50342 (IPv6)
>Sep 19 16:33:45 mckinley-01 qarshd[10174]: Running cmdline: /usr/tests/sts-rhel7.6/bin/checkit -w /mnt/syncd_secondary_3legs_1/checkit -f /tmp/checkit_syncd_secondary_3legs_1 -n 100
>Sep 19 16:33:49 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50360).
>Sep 19 16:33:49 mckinley-01 qarshd[10186]: Talking to peer ::ffff:10.15.80.218:50360 (IPv6)
>Sep 19 16:33:49 mckinley-01 qarshd[10186]: Running cmdline: /usr/tests/sts-rhel7.6/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/syncd_secondary_3legs_1/mckinley-01_load | /usr/tests/sts-rhel7.6/bin/xdoio -v
>Sep 19 16:34:06 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50408).
>Sep 19 16:34:06 mckinley-01 qarshd[10292]: Talking to peer ::ffff:10.15.80.218:50408 (IPv6)
>Sep 19 16:34:06 mckinley-01 qarshd[10292]: Running cmdline: dmstats create --filemap /mnt/syncd_secondary_3legs_1/mckinley-01_load
>Sep 19 16:34:07 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50410).
>Sep 19 16:34:07 mckinley-01 qarshd[10303]: Talking to peer ::ffff:10.15.80.218:50410 (IPv6)
>Sep 19 16:34:07 mckinley-01 qarshd[10303]: Running cmdline: /usr/tests/sts-rhel7.6/bin/checkit -w /mnt/syncd_secondary_3legs_1/checkit -f /tmp/checkit_syncd_secondary_3legs_1 -v
>Sep 19 16:34:13 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50432).
>Sep 19 16:34:13 mckinley-01 qarshd[10314]: Talking to peer ::ffff:10.15.80.218:50432 (IPv6)
>Sep 19 16:34:14 mckinley-01 qarshd[10314]: Running cmdline: dmstats report --group
>Sep 19 16:34:14 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50434).
>Sep 19 16:34:14 mckinley-01 qarshd[10318]: Talking to peer ::ffff:10.15.80.218:50434 (IPv6)
>Sep 19 16:34:14 mckinley-01 qarshd[10318]: Running cmdline: dmstats list --group
>Sep 19 16:34:14 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50436).
>Sep 19 16:34:14 mckinley-01 qarshd[10322]: Talking to peer ::ffff:10.15.80.218:50436 (IPv6)
>Sep 19 16:34:14 mckinley-01 qarshd[10322]: Running cmdline: dmstats delete  /dev/helter_skelter/syncd_secondary_3legs_1 --allregions
>Sep 19 16:34:14 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50438).
>Sep 19 16:34:14 mckinley-01 qarshd[10326]: Talking to peer ::ffff:10.15.80.218:50438 (IPv6)
>Sep 19 16:34:14 mckinley-01 qarshd[10326]: Running cmdline: tail -n1 /var/log/messages | cut -c 1-12
>Sep 19 16:34:15 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50440).
>Sep 19 16:34:15 mckinley-01 qarshd[10332]: Talking to peer ::ffff:10.15.80.218:50440 (IPv6)
>Sep 19 16:34:15 mckinley-01 qarshd[10332]: Running cmdline: multipathd fail path sdh
>Sep 19 16:34:15 mckinley-01 multipathd: mpathg: fail path 8:112 (operator)
>Sep 19 16:34:15 mckinley-01 kernel: device-mapper: multipath: Failing path 8:112.
>Sep 19 16:34:15 mckinley-01 multipathd: sdh: mark as failed
>Sep 19 16:34:15 mckinley-01 multipathd: mpathg: remaining active paths: 3
>Sep 19 16:34:15 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50442).
>Sep 19 16:34:15 mckinley-01 qarshd[10339]: Talking to peer ::ffff:10.15.80.218:50442 (IPv6)
>Sep 19 16:34:15 mckinley-01 qarshd[10339]: Running cmdline: multipathd fail path sdp
>Sep 19 16:34:15 mckinley-01 multipathd: mpathg: fail path 8:240 (operator)
>Sep 19 16:34:15 mckinley-01 kernel: device-mapper: multipath: Failing path 8:240.
>Sep 19 16:34:15 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50444).
>Sep 19 16:34:15 mckinley-01 qarshd[10344]: Talking to peer ::ffff:10.15.80.218:50444 (IPv6)
>Sep 19 16:34:15 mckinley-01 qarshd[10344]: Running cmdline: multipathd fail path sdx
>Sep 19 16:34:15 mckinley-01 multipathd: mpathg: fail path 65:112 (operator)
>Sep 19 16:34:15 mckinley-01 kernel: device-mapper: multipath: Failing path 65:112.
>Sep 19 16:34:15 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50446).
>Sep 19 16:34:15 mckinley-01 qarshd[10353]: Talking to peer ::ffff:10.15.80.218:50446 (IPv6)
>Sep 19 16:34:15 mckinley-01 qarshd[10353]: Running cmdline: multipathd fail path sdaf
>Sep 19 16:34:15 mckinley-01 multipathd: mpathg: fail path 65:240 (operator)
>Sep 19 16:34:15 mckinley-01 kernel: device-mapper: multipath: Failing path 65:240.
>Sep 19 16:34:16 mckinley-01 multipathd: sdaf: mark as failed
>Sep 19 16:34:16 mckinley-01 multipathd: mpathg: remaining active paths: 2
>Sep 19 16:34:16 mckinley-01 multipathd: sdp: mark as failed
>Sep 19 16:34:16 mckinley-01 multipathd: mpathg: remaining active paths: 1
>Sep 19 16:34:16 mckinley-01 multipathd: sdx: mark as failed
>Sep 19 16:34:16 mckinley-01 multipathd: mpathg: remaining active paths: 0
>Sep 19 16:34:19 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50470).
>Sep 19 16:34:19 mckinley-01 qarshd[10358]: Talking to peer ::ffff:10.15.80.218:50470 (IPv6)
>Sep 19 16:34:19 mckinley-01 qarshd[10358]: Running cmdline: pvs -a
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:19 mckinley-01 multipathd: sdh: unusable path
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:19 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:19 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50472).
>Sep 19 16:34:19 mckinley-01 qarshd[10377]: Talking to peer ::ffff:10.15.80.218:50472 (IPv6)
>Sep 19 16:34:19 mckinley-01 qarshd[10377]: Running cmdline: dd if=/dev/zero of=/mnt/syncd_secondary_3legs_1/ddfile count=10 bs=4M
>Sep 19 16:34:20 mckinley-01 lvm[6635]: Secondary mirror device 253:21 has failed (D).
>Sep 19 16:34:20 mckinley-01 lvm[6635]: Device failure in helter_skelter-syncd_secondary_3legs_1.
>Sep 19 16:34:20 mckinley-01 lvm[6635]: WARNING: Disabling lvmetad cache for repair command.
>Sep 19 16:34:20 mckinley-01 lvm[6635]: WARNING: Not using lvmetad because of repair.
>Sep 19 16:34:20 mckinley-01 lvm[6635]: Couldn't find device with uuid Fj3xGt-mM7b-l6BF-q8KQ-ji7m-1jFZ-YVihZL.
>Sep 19 16:34:20 mckinley-01 lvm[6635]: WARNING: Couldn't find all devices for LV helter_skelter/syncd_secondary_3legs_1_mimage_1 while checking used and assumed devices.
>Sep 19 16:34:20 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50474).
>Sep 19 16:34:20 mckinley-01 qarshd[10399]: Talking to peer ::ffff:10.15.80.218:50474 (IPv6)
>Sep 19 16:34:20 mckinley-01 qarshd[10399]: Running cmdline: sync
>Sep 19 16:34:20 mckinley-01 multipathd: dm-21: remove map (uevent)
>Sep 19 16:34:20 mckinley-01 multipathd: dm-21: devmap not registered, can't remove
>Sep 19 16:34:20 mckinley-01 multipathd: dm-21: remove map (uevent)
>Sep 19 16:34:20 mckinley-01 multipathd: sdp: unusable path
>Sep 19 16:34:20 mckinley-01 multipathd: sdx: unusable path
>Sep 19 16:34:20 mckinley-01 multipathd: sdaf: unusable path
>Sep 19 16:34:20 mckinley-01 multipathd: dm-24: remove map (uevent)
>Sep 19 16:34:20 mckinley-01 multipathd: dm-24: devmap not registered, can't remove
>Sep 19 16:34:20 mckinley-01 multipathd: dm-24: remove map (uevent)
>Sep 19 16:34:23 mckinley-01 lvm[6635]: Mirror status: 1 of 3 images failed.
>Sep 19 16:34:23 mckinley-01 lvm[6635]: Repair of mirrored device helter_skelter-syncd_secondary_3legs_1 finished successfully.
>Sep 19 16:34:23 mckinley-01 lvm[6635]: helter_skelter-syncd_secondary_3legs_1 is now in-sync.
>Sep 19 16:34:24 mckinley-01 multipathd: sdh: unusable path
>Sep 19 16:34:25 mckinley-01 multipathd: sdp: unusable path
>Sep 19 16:34:25 mckinley-01 multipathd: sdx: unusable path
>Sep 19 16:34:25 mckinley-01 multipathd: sdaf: unusable path
>Sep 19 16:34:26 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50486).
>Sep 19 16:34:26 mckinley-01 qarshd[10502]: Talking to peer ::ffff:10.15.80.218:50486 (IPv6)
>Sep 19 16:34:26 mckinley-01 qarshd[10502]: Running cmdline: lvs > /dev/null 2>&1
>Sep 19 16:34:26 mckinley-01 kernel: blk_update_request: 22 callbacks suppressed
>Sep 19 16:34:26 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:27 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50488).
>Sep 19 16:34:27 mckinley-01 qarshd[10510]: Talking to peer ::ffff:10.15.80.218:50488 (IPv6)
>Sep 19 16:34:27 mckinley-01 qarshd[10510]: Running cmdline: lvs -a -o lv_name,lv_attr,lv_size,copy_percent,devices
>Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:27 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50490).
>Sep 19 16:34:27 mckinley-01 qarshd[10518]: Talking to peer ::ffff:10.15.80.218:50490 (IPv6)
>Sep 19 16:34:27 mckinley-01 qarshd[10518]: Running cmdline: lvs -a -o +devices
>Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:27 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50492).
>Sep 19 16:34:27 mckinley-01 qarshd[10524]: Talking to peer ::ffff:10.15.80.218:50492 (IPv6)
>Sep 19 16:34:27 mckinley-01 qarshd[10524]: Running cmdline: lvs -a -o +devices
>Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:27 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:28 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50494).
>Sep 19 16:34:28 mckinley-01 qarshd[10530]: Talking to peer ::ffff:10.15.80.218:50494 (IPv6)
>Sep 19 16:34:28 mckinley-01 qarshd[10530]: Running cmdline: lvs -a -o +devices
>Sep 19 16:34:28 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:28 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 40
>Sep 19 16:34:28 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50496).
>Sep 19 16:34:28 mckinley-01 qarshd[10535]: Talking to peer ::ffff:10.15.80.218:50496 (IPv6)
>Sep 19 16:34:28 mckinley-01 qarshd[10535]: Running cmdline: lvs -a -o +devices
>Sep 19 16:34:28 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:28 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50498).
>Sep 19 16:34:28 mckinley-01 qarshd[10540]: Talking to peer ::ffff:10.15.80.218:50498 (IPv6)
>Sep 19 16:34:28 mckinley-01 qarshd[10540]: Running cmdline: dmsetup ls | grep syncd_secondary_3legs_1_mimage_1
>Sep 19 16:34:29 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50504).
>Sep 19 16:34:29 mckinley-01 qarshd[10546]: Talking to peer ::ffff:10.15.80.218:50504 (IPv6)
>Sep 19 16:34:29 mckinley-01 multipathd: sdh: unusable path
>Sep 19 16:34:29 mckinley-01 qarshd[10546]: Running cmdline: lvs --noheadings -a -o +devices
>Sep 19 16:34:29 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50506).
>Sep 19 16:34:29 mckinley-01 qarshd[10578]: Talking to peer ::ffff:10.15.80.218:50506 (IPv6)
>Sep 19 16:34:30 mckinley-01 qarshd[10578]: Running cmdline: lvs --noheadings -a -o +devices
>Sep 19 16:34:30 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50508).
>Sep 19 16:34:30 mckinley-01 qarshd[10583]: Talking to peer ::ffff:10.15.80.218:50508 (IPv6)
>Sep 19 16:34:30 mckinley-01 qarshd[10583]: Running cmdline: lvs --noheadings -a -o +devices
>Sep 19 16:34:30 mckinley-01 multipathd: sdp: unusable path
>Sep 19 16:34:30 mckinley-01 multipathd: sdx: unusable path
>Sep 19 16:34:30 mckinley-01 multipathd: sdaf: unusable path
>Sep 19 16:34:34 mckinley-01 multipathd: sdh: unusable path
>Sep 19 16:34:35 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50510).
>Sep 19 16:34:35 mckinley-01 kernel: blk_update_request: 7 callbacks suppressed
>Sep 19 16:34:35 mckinley-01 kernel: blk_update_request: I/O error, dev dm-14, sector 0
>Sep 19 16:34:35 mckinley-01 qarshd[10607]: Talking to peer ::ffff:10.15.80.218:50510 (IPv6)
>Sep 19 16:34:35 mckinley-01 multipathd: sdp: unusable path
>Sep 19 16:34:35 mckinley-01 multipathd: sdx: unusable path
>Sep 19 16:34:35 mckinley-01 multipathd: sdaf: unusable path
>Sep 19 16:34:35 mckinley-01 qarshd[10607]: Running cmdline: /usr/tests/sts-rhel7.6/bin/checkit -w /mnt/syncd_secondary_3legs_1/checkit -f /tmp/checkit_syncd_secondary_3legs_1 -v
>Sep 19 16:34:39 mckinley-01 multipathd: sdh: unusable path
>Sep 19 16:34:40 mckinley-01 multipathd: sdp: unusable path
>Sep 19 16:34:40 mckinley-01 multipathd: sdx: unusable path
>Sep 19 16:34:40 mckinley-01 multipathd: sdaf: unusable path
>Sep 19 16:34:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50520).
>Sep 19 16:34:42 mckinley-01 qarshd[10613]: Talking to peer ::ffff:10.15.80.218:50520 (IPv6)
>Sep 19 16:34:42 mckinley-01 qarshd[10613]: Running cmdline: dmstats report --group
>Sep 19 16:34:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50522).
>Sep 19 16:34:42 mckinley-01 qarshd[10617]: Talking to peer ::ffff:10.15.80.218:50522 (IPv6)
>Sep 19 16:34:42 mckinley-01 qarshd[10617]: Running cmdline: dmstats list --group
>Sep 19 16:34:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50524).
>Sep 19 16:34:42 mckinley-01 qarshd[10624]: Talking to peer ::ffff:10.15.80.218:50524 (IPv6)
>Sep 19 16:34:42 mckinley-01 qarshd[10624]: Running cmdline: dmstats delete  /dev/helter_skelter/syncd_secondary_3legs_1 --allregions
>Sep 19 16:34:42 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50526).
>Sep 19 16:34:42 mckinley-01 qarshd[10628]: Talking to peer ::ffff:10.15.80.218:50526 (IPv6)
>Sep 19 16:34:43 mckinley-01 qarshd[10628]: Running cmdline: multipathd reinstate path sdh
>Sep 19 16:34:43 mckinley-01 multipathd: mpathg: reinstate path 8:112 (operator)
>Sep 19 16:34:43 mckinley-01 kernel: device-mapper: multipath: Reinstating path 8:112.
>Sep 19 16:34:43 mckinley-01 kernel: sd 7:0:0:6: alua: port group 01 state A preferred supports tolUsNA
>Sep 19 16:34:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50528).
>Sep 19 16:34:43 mckinley-01 qarshd[10640]: Talking to peer ::ffff:10.15.80.218:50528 (IPv6)
>Sep 19 16:34:43 mckinley-01 qarshd[10640]: Running cmdline: multipathd reinstate path sdp
>Sep 19 16:34:43 mckinley-01 multipathd: mpathg: reinstate path 8:240 (operator)
>Sep 19 16:34:43 mckinley-01 kernel: device-mapper: multipath: Reinstating path 8:240.
>Sep 19 16:34:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50530).
>Sep 19 16:34:43 mckinley-01 qarshd[10645]: Talking to peer ::ffff:10.15.80.218:50530 (IPv6)
>Sep 19 16:34:43 mckinley-01 qarshd[10645]: Running cmdline: multipathd reinstate path sdx
>Sep 19 16:34:43 mckinley-01 multipathd: mpathg: reinstate path 65:112 (operator)
>Sep 19 16:34:43 mckinley-01 kernel: device-mapper: multipath: Reinstating path 65:112.
>Sep 19 16:34:43 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50532).
>Sep 19 16:34:43 mckinley-01 qarshd[10662]: Talking to peer ::ffff:10.15.80.218:50532 (IPv6)
>Sep 19 16:34:43 mckinley-01 qarshd[10662]: Running cmdline: multipathd reinstate path sdaf
>Sep 19 16:34:43 mckinley-01 multipathd: mpathg: reinstate path 65:240 (operator)
>Sep 19 16:34:43 mckinley-01 kernel: device-mapper: multipath: Reinstating path 65:240.
>Sep 19 16:34:43 mckinley-01 kernel: sd 7:0:3:6: alua: port group 01 state A preferred supports tolUsNA
>Sep 19 16:34:44 mckinley-01 multipathd: mpathg: sdh - tur checker reports path is up
>Sep 19 16:34:44 mckinley-01 multipathd: 8:112: reinstated
>Sep 19 16:34:44 mckinley-01 multipathd: mpathg: remaining active paths: 1
>Sep 19 16:34:45 mckinley-01 multipathd: mpathg: sdp - tur checker reports path is up
>Sep 19 16:34:45 mckinley-01 multipathd: 8:240: reinstated
>Sep 19 16:34:45 mckinley-01 multipathd: mpathg: remaining active paths: 2
>Sep 19 16:34:45 mckinley-01 multipathd: mpathg: sdx - tur checker reports path is up
>Sep 19 16:34:45 mckinley-01 multipathd: 65:112: reinstated
>Sep 19 16:34:45 mckinley-01 multipathd: mpathg: remaining active paths: 3
>Sep 19 16:34:45 mckinley-01 multipathd: mpathg: sdaf - tur checker reports path is up
>Sep 19 16:34:45 mckinley-01 multipathd: 65:240: reinstated
>Sep 19 16:34:45 mckinley-01 multipathd: mpathg: remaining active paths: 4
>Sep 19 16:34:56 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50566).
>Sep 19 16:34:56 mckinley-01 qarshd[10706]: Talking to peer ::ffff:10.15.80.218:50566 (IPv6)
>Sep 19 16:34:56 mckinley-01 qarshd[10706]: Running cmdline: vgreduce --removemissing --force helter_skelter
>Sep 19 16:34:56 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50572).
>Sep 19 16:34:56 mckinley-01 qarshd[10734]: Talking to peer ::ffff:10.15.80.218:50572 (IPv6)
>Sep 19 16:34:56 mckinley-01 qarshd[10734]: Running cmdline: pvcreate /dev/mapper/mpathg1
>Sep 19 16:34:57 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50574).
>Sep 19 16:34:57 mckinley-01 qarshd[10747]: Talking to peer ::ffff:10.15.80.218:50574 (IPv6)
>Sep 19 16:34:57 mckinley-01 qarshd[10747]: Running cmdline: vgextend helter_skelter /dev/mapper/mpathg1
>Sep 19 16:34:57 mckinley-01 systemd: Started qarsh Per-Connection Server (10.15.80.218:50576).
>Sep 19 16:34:57 mckinley-01 qarshd[10761]: Talking to peer ::ffff:10.15.80.218:50576 (IPv6)
>
>
>#### Everything is fine until the test runs its own upconvert due to the (remove policy), which is exactly what lvm does in alloc failure policy
>
>
>Sep 19 16:34:57 mckinley-01 qarshd[10761]: Running cmdline: lvconvert --yes --type mirror --mirrorlog disk -m 2 helter_skelter/syncd_secondary_3legs_1 /dev/mapper/mpathb1:0-2400 /dev/mapper/mpathg1:0-2400 /dev/mapper/mpathf1:0-2400 /dev/mapper/mpathc1:0-150
>Sep 19 16:34:58 mckinley-01 dmeventd[6635]: No longer monitoring mirror device helter_skelter-syncd_secondary_3legs_1 for events.
>Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: Error -5 writing to journal, jid=0
>Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: error -5: withdrawing the file system to prevent further damage.
>Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: about to withdraw this file system
>Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: telling LM to unmount
>Sep 19 16:34:58 mckinley-01 kernel: GFS2: fsid=MCKINLEY:gfs1.0: withdrawn
>Sep 19 16:34:58 mckinley-01 kernel: CPU: 1 PID: 10101 Comm: gfs2_logd Kdump: loaded Not tainted 3.10.0-951.el7.x86_64 #1
>Sep 19 16:34:58 mckinley-01 kernel: Hardware name: Dell Inc. PowerEdge R820/0RN9TC, BIOS 2.0.20 01/16/2014
>Sep 19 16:34:58 mckinley-01 kernel: Call Trace:
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff97b61ca8>] dump_stack+0x19/0x1b
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffffc09c7206>] gfs2_lm_withdraw+0x146/0x180 [gfs2]
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffffc09abe15>] gfs2_logd+0x315/0x340 [gfs2]
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff974c2d60>] ? wake_up_atomic_t+0x30/0x30
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffffc09abb00>] ? gfs2_meta_syncfs+0x50/0x50 [gfs2]
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff974c1c91>] kthread+0xd1/0xe0
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff974c1bc0>] ? insert_kthread_work+0x40/0x40
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff97b74c37>] ret_from_fork_nospec_begin+0x21/0x21
>Sep 19 16:34:58 mckinley-01 kernel: [<ffffffff974c1bc0>] ? insert_kthread_work+0x40/0x40
>Sep 19 16:35:01 mckinley-01 lvm[6635]: Monitoring mirror device helter_skelter-syncd_secondary_3legs_1_mimagetmp_3 for events.
>Sep 19 16:35:01 mckinley-01 lvm[6635]: Monitoring mirror device helter_skelter-syncd_secondary_3legs_1 for events.
>Sep 19 16:35:01 mckinley-01 systemd: Started LVM2 poll daemon.
>Sep 19 16:35:02 mckinley-01 qarshd[10186]: Sending child 10187 signal 2
>Sep 19 16:35:08 mckinley-01 qarshd[10186]: Sending child 10187 signal 15
>Sep 19 16:35:17 mckinley-01 lvm[6635]: No longer monitoring mirror device helter_skelter-syncd_secondary_3legs_1_mimagetmp_3 for events.
>Sep 19 16:35:17 mckinley-01 dmeventd[6635]: No longer monitoring mirror device helter_skelter-syncd_secondary_3legs_1 for events.
>Sep 19 16:36:17 mckinley-01 lvmpolld: W: LVMPOLLD: polling for output of the lvm cmd (PID 10867) has timed out
>Sep 19 16:36:32 mckinley-01 lvmpolld: W: #011LVPOLL: PID 10867: STDERR: '  Error locking on node 1: Command timed out'
>Sep 19 16:36:44 mckinley-01 systemd: qarshd@268-10.15.104.61:5016-10.15.80.218:50360.service stop-sigterm timed out. Killing.
>Sep 19 16:36:59 mckinley-01 kernel: INFO: task xdoio:10190 blocked for more than 120 seconds.
>Sep 19 16:36:59 mckinley-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep 19 16:36:59 mckinley-01 kernel: xdoio           D ffff8fc97b3f9040     0 10190      1 0x00000084
>Sep 19 16:36:59 mckinley-01 kernel: Call Trace:
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97678461>] ? __block_commit_write.isra.20+0x81/0xb0
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b65b70>] ? out_of_line_wait_on_atomic_t+0x100/0x100
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b67aa9>] schedule+0x29/0x70
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b65b81>] bit_wait+0x11/0x50
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b656f7>] __wait_on_bit+0x67/0x90
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97644091>] ? __sb_end_write+0x31/0x60
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b65b70>] ? out_of_line_wait_on_atomic_t+0x100/0x100
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b65861>] out_of_line_wait_on_bit+0x81/0xb0
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff974c2e20>] ? wake_bit_function+0x40/0x40
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09a6c3e>] gfs2_glock_wait+0x3e/0x80 [gfs2]
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09a8720>] gfs2_glock_nq+0x250/0x420 [gfs2]
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09c6796>] gfs2_trans_begin+0xf6/0x1b0 [gfs2]
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09c523f>] gfs2_dirty_inode+0x1bf/0x280 [gfs2]
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09c526b>] ? gfs2_dirty_inode+0x1eb/0x280 [gfs2]
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff9766fedd>] __mark_inode_dirty+0x16d/0x270
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff9765dfd9>] update_time+0x89/0xd0
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff9765e0c0>] file_update_time+0xa0/0xf0
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff975b8668>] __generic_file_aio_write+0x198/0x400
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff975b8929>] generic_file_aio_write+0x59/0xa0
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffffc09b1130>] gfs2_file_aio_write+0xd0/0x160 [gfs2]
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97640613>] do_sync_write+0x93/0xe0
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97641100>] vfs_write+0xc0/0x1f0
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b74d21>] ? system_call_after_swapgs+0xae/0x146
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97641f1f>] SyS_write+0x7f/0xf0
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b74d21>] ? system_call_after_swapgs+0xae/0x146
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b74ddb>] system_call_fastpath+0x22/0x27
>Sep 19 16:36:59 mckinley-01 kernel: [<ffffffff97b74d21>] ? system_call_after_swapgs+0xae/0x146
>Sep 19 16:37:32 mckinley-01 lvmpolld: W: LVMPOLLD: polling for output of the lvm cmd (PID 10867) has timed out
>Sep 19 16:38:14 mckinley-01 systemd: qarshd@268-10.15.104.61:5016-10.15.80.218:50360.service still around after SIGKILL. Ignoring.
>Sep 19 16:38:32 mckinley-01 lvmpolld: W: LVMPOLLD: polling for output of the lvm cmd (PID 10867) has timed out
>Sep 19 16:38:32 mckinley-01 lvmpolld: W: #011LVPOLL: PID 10867: STDERR: '  Failed to lock logical volume helter_skelter/syncd_secondary_3legs_1.'
>Sep 19 16:38:32 mckinley-01 lvmpolld: W: #011LVPOLL: PID 10867: STDERR: '  Error locking on node 1: Command timed out'

Actions: View

Attachments on bug 1628529: 1483035 | 1484952 | 1484953 | 1485256 | 1486443 | 1487280 | 1487285 | 1487286 | 1487287 | 1487288 | 1492603