Bug 1448123
| Summary: | RAID TAKEOVER: deadlock during cluster VG lvconvert attempt (Unable to change name on mapped device) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Corey Marthaler <cmarthal> | ||||||
| Component: | lvm2 | Assignee: | Heinz Mauelshagen <heinzm> | ||||||
| lvm2 sub component: | Mirroring and RAID | QA Contact: | cluster-qe <cluster-qe> | ||||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||||
| Severity: | high | ||||||||
| Priority: | unspecified | CC: | agk, heinzm, jbrassow, lmiksik, msnitzer, mthacker, prajnoha, prockai, yizhan, zkabelac | ||||||
| Version: | 7.4 | ||||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | lvm2-2.02.171-5.el7 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2017-08-01 21:54:18 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Sounds like userspace issuing the wrong sequence of operations. Need -vvvv and clvmd debug logs to check. This looks like https://bugzilla.redhat.com/show_bug.cgi?id=1443999 for the mddev_suspend and md_write_start race part. This still exists while running the kernel fix for bug 1443999. I'll attempt to provide -vvvv lvconvert log. 3.10.0-679.el7.bz1443999.x86_64 lvm2-2.02.171-4.el7 BUILT: Wed Jun 7 09:16:17 CDT 2017 lvm2-libs-2.02.171-4.el7 BUILT: Wed Jun 7 09:16:17 CDT 2017 lvm2-cluster-2.02.171-4.el7 BUILT: Wed Jun 7 09:16:17 CDT 2017 device-mapper-1.02.140-4.el7 BUILT: Wed Jun 7 09:16:17 CDT 2017 device-mapper-libs-1.02.140-4.el7 BUILT: Wed Jun 7 09:16:17 CDT 2017 device-mapper-event-1.02.140-4.el7 BUILT: Wed Jun 7 09:16:17 CDT 2017 device-mapper-event-libs-1.02.140-4.el7 BUILT: Wed Jun 7 09:16:17 CDT 2017 device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 10:15:46 CDT 2017 TAKEOVER: lvconvert --yes --type raid0 centipede2/takeover Error locking on node 2: Failed to rename centipede2-takeover_rimage__3 (253:32) to centipede2-takeover_rimage_3 Problem reactivating logical volume centipede2/takeover. Jun 12 14:02:02 harding-03 qarshd[5845]: Running cmdline: lvconvert --yes --type raid0 centipede2/takeover Jun 12 14:02:03 harding-03 kernel: Jun 12 14:02:03 harding-03 dmeventd[3774]: No longer monitoring RAID device centipede2-takeover for events. Jun 12 14:02:03 harding-03 kernel: device-mapper: ioctl: Unable to change name on mapped device centipede2-takeover_rimage__3 to one that already exists: centipede2-takeover_rimage_3 Created attachment 1287100 [details]
verbose lvconvert attempt
(In reply to Corey Marthaler from comment #4) > Error locking on node 2: Failed to rename centipede2-takeover_rimage__3 > (253:32) to centipede2-takeover_rimage_3 I thought there'd been a patch already to avoid consecutive underscores appearing in names. Perhaps there's another instance of it remaining somewhere? Created attachment 1287439 [details]
requested dm information
Here are the requested dm states for both the single node and cluster node case, both before and after the takeover attempt.
(In reply to Alasdair Kergon from comment #6) > (In reply to Corey Marthaler from comment #4) > > > Error locking on node 2: Failed to rename centipede2-takeover_rimage__3 > > (253:32) to centipede2-takeover_rimage_3 > > I thought there'd been a patch already to avoid consecutive underscores > appearing in names. Perhaps there's another instance of it remaining > somewhere? raid code uses it for temporary renames during e.g. raid10 -> raid0(_meta) conversions (takeover) in order to avoid collisions on area shifts (e.g. *_rimage_2 has to be shifted to *_rimage_1). Upstream commit 08079ec42062944eadcd4e5625edafe45b36c55c (In reply to Heinz Mauelshagen from comment #9) > Upstream commit 08079ec42062944eadcd4e5625edafe45b36c55c This is an LVM2 fix - not kernel. Need to change component. clearing blocker flag Disallowing reshape/takeover while LV is in cluster VG until future release Note, that this means this bug won't be able to be verified until rhel 7.5! The bug will be prevented from happening because it is disallowed on 7.4. Marking "verified" with the caveat that this bug still exists, it's just not possible now to takeover/reshape when residing on a clustered VG (see bug 1462929) 3.10.0-685.el7.x86_64 lvm2-2.02.171-7.el7 BUILT: Thu Jun 22 08:35:15 CDT 2017 lvm2-libs-2.02.171-7.el7 BUILT: Thu Jun 22 08:35:15 CDT 2017 lvm2-cluster-2.02.171-7.el7 BUILT: Thu Jun 22 08:35:15 CDT 2017 device-mapper-1.02.140-7.el7 BUILT: Thu Jun 22 08:35:15 CDT 2017 device-mapper-libs-1.02.140-7.el7 BUILT: Thu Jun 22 08:35:15 CDT 2017 device-mapper-event-1.02.140-7.el7 BUILT: Thu Jun 22 08:35:15 CDT 2017 device-mapper-event-libs-1.02.140-7.el7 BUILT: Thu Jun 22 08:35:15 CDT 2017 device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 10:15:46 CDT 2017 [root@harding-03 ~]# vgs VG #PV #LV #SN Attr VSize VFree centipede2 8 1 0 wz--nc 1.95t <1.95t [root@harding-03 ~]# lvs -a -o +devices LV VG Attr LSize Cpy%Sync Devices takeover centipede2 rwi-a-r--- 4.00g 100.00 takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0),takeover_rimage_4(0),takeover_rimage_5(0) [takeover_rimage_0] centipede2 iwi-aor--- 1.00g /dev/mapper/mpatha1(1) [takeover_rimage_1] centipede2 iwi-aor--- 1.00g /dev/mapper/mpathb1(1) [takeover_rimage_2] centipede2 iwi-aor--- 1.00g /dev/mapper/mpathc1(1) [takeover_rimage_3] centipede2 iwi-aor--- 1.00g /dev/mapper/mpathd1(1) [takeover_rimage_4] centipede2 iwi-aor--- 1.00g /dev/mapper/mpathe1(1) [takeover_rimage_5] centipede2 iwi-aor--- 1.00g /dev/mapper/mpathf1(1) [takeover_rmeta_0] centipede2 ewi-aor--- 4.00m /dev/mapper/mpatha1(0) [takeover_rmeta_1] centipede2 ewi-aor--- 4.00m /dev/mapper/mpathb1(0) [takeover_rmeta_2] centipede2 ewi-aor--- 4.00m /dev/mapper/mpathc1(0) [takeover_rmeta_3] centipede2 ewi-aor--- 4.00m /dev/mapper/mpathd1(0) [takeover_rmeta_4] centipede2 ewi-aor--- 4.00m /dev/mapper/mpathe1(0) [takeover_rmeta_5] centipede2 ewi-aor--- 4.00m /dev/mapper/mpathf1(0) [root@harding-03 ~]# lvconvert --yes --type raid6_rs_6 centipede2/takeover Using default stripesize 64.00 KiB. Conversion of centipede2/takeover not supported in the cluster. [root@harding-03 ~]# lvconvert --yes --stripes 5 centipede2/takeover Using default stripesize 64.00 KiB. Conversion of centipede2/takeover not supported in the cluster. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2222 |
Description of problem: This happened on a machine w/ 16 X 4 cpu cores, on an exclusively activated raid on a clustered VG. ================================================================================ Iteration 0.2 started at Wed May 3 18:22:57 CDT 2017 ================================================================================ Scenario raid10_near: Convert Striped raid10_near volume ********* Take over hash info for this scenario ********* * from type: raid10_near * to type: raid0 * from legs: 4 * to legs: 5 * from region: 512.00k * to region: 0 * contiguous: 1 * snapshot: 0 ****************************************************** Creating original volume on mckinley-02... mckinley-02: lvcreate -aye --type raid10_near -R 512.00k -i 4 -n takeover -L 4G centipede2 Waiting until all mirror|raid volumes become fully syncd... 0/1 mirror(s) are fully synced: ( 43.76% ) 0/1 mirror(s) are fully synced: ( 94.21% ) 1/1 mirror(s) are fully synced: ( 100.00% ) Sleeping 15 sec Current volume device structure: LV Attr LSize Cpy%Sync Devices takeover rwi-a-r--- 4.00g 100.00 takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0),takeover_rimage_4(0),takeover_rimage_5(0),takeover_rimage_6(0),takeover_rimage_7(0) [takeover_rimage_0] iwi-aor--- 1.00g /dev/mapper/mpatha1(1) [takeover_rimage_1] iwi-aor--- 1.00g /dev/mapper/mpathb1(1) [takeover_rimage_2] iwi-aor--- 1.00g /dev/mapper/mpathc1(1) [takeover_rimage_3] iwi-aor--- 1.00g /dev/mapper/mpathd1(1) [takeover_rimage_4] iwi-aor--- 1.00g /dev/mapper/mpathe1(1) [takeover_rimage_5] iwi-aor--- 1.00g /dev/mapper/mpathf1(1) [takeover_rimage_6] iwi-aor--- 1.00g /dev/mapper/mpathg1(1) [takeover_rimage_7] iwi-aor--- 1.00g /dev/mapper/mpathh1(1) [takeover_rmeta_0] ewi-aor--- 4.00m /dev/mapper/mpatha1(0) [takeover_rmeta_1] ewi-aor--- 4.00m /dev/mapper/mpathb1(0) [takeover_rmeta_2] ewi-aor--- 4.00m /dev/mapper/mpathc1(0) [takeover_rmeta_3] ewi-aor--- 4.00m /dev/mapper/mpathd1(0) [takeover_rmeta_4] ewi-aor--- 4.00m /dev/mapper/mpathe1(0) [takeover_rmeta_5] ewi-aor--- 4.00m /dev/mapper/mpathf1(0) [takeover_rmeta_6] ewi-aor--- 4.00m /dev/mapper/mpathg1(0) [takeover_rmeta_7] ewi-aor--- 4.00m /dev/mapper/mpathh1(0) Creating xfs on top of mirror(s) on mckinley-02... warning: device is not properly aligned /dev/centipede2/takeover Mounting mirrored xfs filesystems on mckinley-02... Writing verification files (checkit) to mirror(s) on... ---- mckinley-02 ---- Sleeping 15 seconds to get some outsanding I/O locks before the failure Verifying files (checkit) on mirror(s) on... ---- mckinley-02 ---- TAKEOVER: lvconvert --yes --type raid0 centipede2/takeover Error locking on node 2: Failed to rename centipede2-takeover_rimage__3 (253:32) to centipede2-takeover_rimage_3 Problem reactivating logical volume centipede2/takeover. May 3 18:24:16 mckinley-02 qarshd[9457]: Running cmdline: lvconvert --yes --type raid0 centipede2/takeover May 3 18:24:16 mckinley-02 kernel: May 3 18:24:17 mckinley-02 dmeventd[6406]: No longer monitoring RAID device centipede2-takeover for events. May 3 18:24:17 mckinley-02 kernel: device-mapper: ioctl: Unable to change name on mapped device centipede2-takeover_rimage__3 to one that already exists: centipede2-takeover_rimage_3 May 3 18:30:51 mckinley-02 kernel: INFO: task kworker/7:1H:1348 blocked for more than 120 seconds. May 3 18:30:51 mckinley-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 3 18:30:51 mckinley-02 kernel: kworker/7:1H D ffffffffffffffff 0 1348 2 0x00000000 May 3 18:30:51 mckinley-02 kernel: Workqueue: xfs-log/dm-35 xfs_log_worker [xfs] May 3 18:30:51 mckinley-02 kernel: ffffa0d276b2fb20 0000000000000046 ffffa0d276f6de20 ffffa0d276b2ffd8 May 3 18:30:51 mckinley-02 kernel: ffffa0d276b2ffd8 ffffa0d276b2ffd8 ffffa0d276f6de20 ffffa0d276b2fc78 May 3 18:30:51 mckinley-02 kernel: 7fffffffffffffff ffffa0d276b2fc70 ffffa0d276f6de20 ffffffffffffffff May 3 18:30:51 mckinley-02 kernel: Call Trace: May 3 18:30:51 mckinley-02 kernel: [<ffffffff9d0a4aa9>] schedule+0x29/0x70 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9d0a25b9>] schedule_timeout+0x239/0x2c0 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cac1285>] ? check_preempt_curr+0x85/0xa0 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cac12b9>] ? ttwu_do_wakeup+0x19/0xd0 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cac4543>] ? try_to_wake_up+0x183/0x340 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9d0a4e5d>] wait_for_completion+0xfd/0x140 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cac47e0>] ? wake_up_state+0x20/0x20 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9caa8ecd>] flush_work+0xfd/0x190 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9caa5db0>] ? move_linked_works+0x90/0x90 May 3 18:30:51 mckinley-02 kernel: [<ffffffffc0789aba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cacdc9c>] ? dequeue_entity+0x11c/0x5d0 May 3 18:30:51 mckinley-02 kernel: [<ffffffffc0787b75>] _xfs_log_force+0x85/0x2c0 [xfs] May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cac76c5>] ? sched_clock_cpu+0x85/0xc0 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9ca29557>] ? __switch_to+0xd7/0x510 May 3 18:30:51 mckinley-02 kernel: [<ffffffffc0787e44>] ? xfs_log_worker+0x24/0x50 [xfs] May 3 18:30:51 mckinley-02 kernel: [<ffffffffc0787ddc>] xfs_log_force+0x2c/0x70 [xfs] May 3 18:30:51 mckinley-02 kernel: [<ffffffffc0787e44>] xfs_log_worker+0x24/0x50 [xfs] May 3 18:30:51 mckinley-02 kernel: [<ffffffff9caa874a>] process_one_work+0x17a/0x440 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9caa97e6>] worker_thread+0x126/0x3c0 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9caa96c0>] ? manage_workers.isra.24+0x2a0/0x2a0 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cab093f>] kthread+0xcf/0xe0 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cab0870>] ? insert_kthread_work+0x40/0x40 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9d0b0518>] ret_from_fork+0x58/0x90 May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cab0870>] ? insert_kthread_work+0x40/0x40 May 3 18:30:51 mckinley-02 kernel: INFO: task kworker/7:13:8706 blocked for more than 120 seconds. May 3 18:30:51 mckinley-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 3 18:30:51 mckinley-02 kernel: kworker/7:13 D ffffa0d27e7d0900 0 8706 2 0x00000080 May 3 18:30:51 mckinley-02 kernel: Workqueue: xfs-cil/dm-35 xlog_cil_push_work [xfs] May 3 18:30:51 mckinley-02 kernel: ffffa0d27460bbe8 0000000000000046 ffffa0d27df5de20 ffffa0d27460bfd8 May 3 18:30:51 mckinley-02 kernel: ffffa0d27460bfd8 ffffa0d27460bfd8 ffffa0d27df5de20 ffffa0d27e7d0928 May 3 18:30:51 mckinley-02 kernel: ffffa0d2752b0ac8 ffffa0d27df5de20 ffffa0d27e7d0800 ffffa0d27e7d0900 May 3 18:30:51 mckinley-02 kernel: Call Trace: May 3 18:30:51 mckinley-02 kernel: [<ffffffff9d0a4aa9>] schedule+0x29/0x70 May 3 18:30:51 mckinley-02 kernel: [<ffffffffc0786f7d>] xlog_state_get_iclog_space+0x10d/0x320 [xfs] May 3 18:30:51 mckinley-02 kernel: [<ffffffff9cac47e0>] ? wake_up_state+0x20/0x20 May 3 18:30:51 mckinley-02 kernel: [<ffffffffc0787489>] xlog_write+0x1a9/0x720 [xfs] May 3 18:30:51 mckinley-02 kernel: [<ffffffffc07843f7>] ? kmem_zone_alloc+0x97/0x130 [xfs] May 3 18:30:52 mckinley-02 kernel: [<ffffffffc0789198>] xlog_cil_push+0x2a8/0x430 [xfs] May 3 18:30:52 mckinley-02 kernel: [<ffffffffc0789335>] xlog_cil_push_work+0x15/0x20 [xfs] May 3 18:30:52 mckinley-02 kernel: [<ffffffff9caa874a>] process_one_work+0x17a/0x440 May 3 18:30:52 mckinley-02 kernel: [<ffffffff9caa9938>] worker_thread+0x278/0x3c0 May 3 18:30:52 mckinley-02 kernel: [<ffffffff9caa96c0>] ? manage_workers.isra.24+0x2a0/0x2a0 May 3 18:30:52 mckinley-02 kernel: [<ffffffff9cab093f>] kthread+0xcf/0xe0 May 3 18:30:52 mckinley-02 kernel: [<ffffffff9cab0870>] ? insert_kthread_work+0x40/0x40 May 3 18:30:52 mckinley-02 kernel: [<ffffffff9d0b0518>] ret_from_fork+0x58/0x90 May 3 18:30:52 mckinley-02 kernel: [<ffffffff9cab0870>] ? insert_kthread_work+0x40/0x40 May 3 18:32:52 mckinley-02 kernel: INFO: task kworker/7:1H:1348 blocked for more than 120 seconds. May 3 18:32:52 mckinley-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 3 18:32:52 mckinley-02 kernel: kworker/7:1H D ffffffffffffffff 0 1348 2 0x00000000 May 3 18:32:52 mckinley-02 kernel: Workqueue: xfs-log/dm-35 xfs_log_worker [xfs] May 3 18:32:52 mckinley-02 kernel: ffffa0d276b2fb20 0000000000000046 ffffa0d276f6de20 ffffa0d276b2ffd8 May 3 18:32:52 mckinley-02 kernel: ffffa0d276b2ffd8 ffffa0d276b2ffd8 ffffa0d276f6de20 ffffa0d276b2fc78 May 3 18:32:52 mckinley-02 kernel: 7fffffffffffffff ffffa0d276b2fc70 ffffa0d276f6de20 ffffffffffffffff May 3 18:32:52 mckinley-02 kernel: Call Trace: May 3 18:32:52 mckinley-02 kernel: [<ffffffff9d0a4aa9>] schedule+0x29/0x70 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9d0a25b9>] schedule_timeout+0x239/0x2c0 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cac1285>] ? check_preempt_curr+0x85/0xa0 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cac12b9>] ? ttwu_do_wakeup+0x19/0xd0 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cac4543>] ? try_to_wake_up+0x183/0x340 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9d0a4e5d>] wait_for_completion+0xfd/0x140 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cac47e0>] ? wake_up_state+0x20/0x20 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9caa8ecd>] flush_work+0xfd/0x190 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9caa5db0>] ? move_linked_works+0x90/0x90 May 3 18:32:52 mckinley-02 kernel: [<ffffffffc0789aba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cacdc9c>] ? dequeue_entity+0x11c/0x5d0 May 3 18:32:52 mckinley-02 kernel: [<ffffffffc0787b75>] _xfs_log_force+0x85/0x2c0 [xfs] May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cac76c5>] ? sched_clock_cpu+0x85/0xc0 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9ca29557>] ? __switch_to+0xd7/0x510 May 3 18:32:52 mckinley-02 kernel: [<ffffffffc0787e44>] ? xfs_log_worker+0x24/0x50 [xfs] May 3 18:32:52 mckinley-02 kernel: [<ffffffffc0787ddc>] xfs_log_force+0x2c/0x70 [xfs] May 3 18:32:52 mckinley-02 kernel: [<ffffffffc0787e44>] xfs_log_worker+0x24/0x50 [xfs] May 3 18:32:52 mckinley-02 kernel: [<ffffffff9caa874a>] process_one_work+0x17a/0x440 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9caa97e6>] worker_thread+0x126/0x3c0 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9caa96c0>] ? manage_workers.isra.24+0x2a0/0x2a0 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cab093f>] kthread+0xcf/0xe0 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cab0870>] ? insert_kthread_work+0x40/0x40 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9d0b0518>] ret_from_fork+0x58/0x90 May 3 18:32:52 mckinley-02 kernel: [<ffffffff9cab0870>] ? insert_kthread_work+0x40/0x40 Version-Release number of selected component (if applicable): 3.10.0-657.el7.x86_64 lvm2-2.02.171-1.el7 BUILT: Wed May 3 07:05:13 CDT 2017 lvm2-libs-2.02.171-1.el7 BUILT: Wed May 3 07:05:13 CDT 2017 lvm2-cluster-2.02.171-1.el7 BUILT: Wed May 3 07:05:13 CDT 2017 device-mapper-1.02.140-1.el7 BUILT: Wed May 3 07:05:13 CDT 2017 device-mapper-libs-1.02.140-1.el7 BUILT: Wed May 3 07:05:13 CDT 2017 device-mapper-event-1.02.140-1.el7 BUILT: Wed May 3 07:05:13 CDT 2017 device-mapper-event-libs-1.02.140-1.el7 BUILT: Wed May 3 07:05:13 CDT 2017 device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 10:15:46 CDT 2017