Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1315417

Summary: Potential memory leak when stressing lvm activation operations
Product: Red Hat Enterprise Linux 6 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: LVM and device-mapper development team <lvm-team>
lvm2 sub component: Activating existing Logical Volumes (RHEL6) QA Contact: cluster-qe <cluster-qe>
Status: CLOSED WONTFIX Docs Contact:
Severity: low    
Priority: unspecified CC: agk, heinzm, jbrassow, msnitzer, prajnoha, prockai, zkabelac
Version: 6.8   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-04 01:06:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2016-03-07 16:49:52 UTC
Description of problem:
Basically this test just creates many different types of volumes, and then deactivates and reactivates them over and over. It takes many hours/days to hit this, so this is definitely very low priority.



./activator -i 10000

Iteration 4811 of 10000 started at Sat Mar  5 23:44:24 CST 2016
Deactivating volumes on VG activator1 on host-113.virt.lab.msp.redhat.com
Deactivating volumes on VG activator2 on host-113.virt.lab.msp.redhat.com
Deactivating volumes on VG activator3 on host-113.virt.lab.msp.redhat.com
Deactivating volumes on VG activator4 on host-113.virt.lab.msp.redhat.com
Verifying that all volumes are deactivated
Verifying that all volumes no longer exist in device-mapper
Reactivating volumes on VG activator1 on host-113.virt.lab.msp.redhat.com
Reactivating volumes on VG activator2 on host-113.virt.lab.msp.redhat.com
Reactivating volumes on VG activator3 on host-113.virt.lab.msp.redhat.com
Reactivating volumes on VG activator4 on host-113.virt.lab.msp.redhat.com
recv_packet() returned NULL!
Remote command on host-113.virt.lab.msp.redhat.com exited with unknown state
couldn't reactivate on host-113.virt.lab.msp.redhat.com




device-mapper: thin: Data device (dm-293) discard unsupported: Disabling discard passdown.
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-313) discard unsupported: Disabling discard passdown.
md/raid1:mdX: active with 4 out of 4 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-337) discard unsupported: Disabling discard passdown.
kthreadd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
kthreadd cpuset=/ mems_allowed=0
Pid: 2, comm: kthreadd Not tainted 2.6.32-616.el6.x86_64 #1
Call Trace:
 [<ffffffff81131620>] ? dump_header+0x90/0x1b0
 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30
 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff811319e1>] ? select_bad_process+0xe1/0x120
 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950
 [<ffffffff8107a438>] ? copy_process+0x168/0x1520
 [<ffffffff8106b4e3>] ? perf_event_task_sched_out+0x33/0x70
 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0
 [<ffffffff81065990>] ? __dequeue_entity+0x30/0x50
 [<ffffffff8100970a>] ? __switch_to+0xea/0x340
 [<ffffffff8112e640>] ? sync_page_killable+0x0/0x40
 [<ffffffff8112e640>] ? sync_page_killable+0x0/0x40
 [<ffffffff8100c222>] ? kernel_thread+0x82/0xe0
 [<ffffffff810a6590>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
 [<ffffffff810a6536>] ? kthreadd+0x196/0x1f0
 [<ffffffff8100c28a>] ? child_rip+0xa/0x20
 [<ffffffff810a63a0>] ? kthreadd+0x0/0x1f0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:   1
active_anon:2 inactive_anon:27 isolated_anon:0
 active_file:19 inactive_file:0 isolated_file:0
 unevictable:45543 dirty:0 writeback:0 unstable:0
 free:15605 slab_reclaimable:2620 slab_unreclaimable:34494
 mapped:1929 shmem:10 pagetables:1468 bounce:0
Node 0 DMA free:4656kB min:668kB low:832kB high:1000kB active_anon:8kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:6644kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:6644kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:2612kB kernel_stack:608kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 994 994 994
Node 0 DMA32 free:57764kB min:44384kB low:55480kB high:66576kB active_anon:0kB inactive_anon:108kB active_file:72kB inactive_file:0kB unevictable:175528kB isolated(anon):0kB isolated(file):0kB present:1018068kB mlocked:151000kB dirty:0kB writeback:0kB mapped:7716kB shmem:40kB slab_reclaimable:10424kB slab_unreclaimable:135364kB kernel_stack:10112kB pagetables:5876kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 30*4kB 23*8kB 24*16kB 26*32kB 15*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4656kB
Node 0 DMA32: 11553*4kB 1442*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 57764kB
1992 total pagecache pages
34 pages in swap cache
Swap cache stats: add 1626302, delete 1626268, find 2823798/3043135
Free swap  = 753060kB
Total swap = 835580kB
262140 pages RAM
7108 pages reserved
12016 pages shared
228581 pages non-shared
[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[  455]     0   455     2772       67   0     -17         -1000 udevd
[  984]     0   984    76003     1528   0     -17         -1000 multipathd
[ 1030]     0  1030     7240     5702   0       0           -17 iscsiuio
[ 1036]     0  1036     1235       99   0       0             0 iscsid
[ 1037]     0  1037     2530     1510   0       0           -17 iscsid
[ 1247]     0  1247     2281       66   0       0             0 dhclient
[ 1337]     0  1337     2281       68   0       0             0 dhclient
[ 1397]     0  1397     6900      118   0     -17         -1000 auditd
[ 1431]     0  1431    62369      153   0       0             0 rsyslogd
[ 1668]    32  1668     4746      121   0       0             0 rpcbind
[ 1690]    29  1690     5839      168   0       0             0 rpc.statd
[ 1724]    81  1724     7953      135   0       0             0 dbus-daemon
[ 1746]     0  1746    47244      206   0       0             0 cupsd
[ 1778]     0  1778     1021      127   0       0             0 acpid
[ 1790]    68  1790     9580      198   0       0             0 hald
[ 1791]     0  1791     5101      127   0       0             0 hald-runner
[ 1823]     0  1823     5631      114   0       0             0 hald-addon-inpu
[ 1848]    68  1848     4503      156   0       0             0 hald-addon-acpi
[ 1880]     0  1880    96537      173   0       0             0 automount
[ 1936]     0  1936    16559       86   0     -17         -1000 sshd
[ 1947]     0  1947     5430      148   0       0             0 xinetd
[ 1964]     0  1964     1014      130   0       0             0 btimed
[ 1975]    38  1975     7685      209   0       0             0 ntpd
[ 1996]     0  1996    23261       96   0       0             0 sendmail
[ 2006]    51  2006    20080       93   0       0             0 sendmail
[ 2034]     0  2034    45759      189   0       0             0 abrtd
[ 2046]     0  2046    29220      166   0       0             0 crond
[ 2061]     0  2061     5278       71   0       0             0 atd
[ 2077]     0  2077    27088       88   0       0             0 rhsmcertd
[ 2094]     0  2094    59461      247   0       0             0 beah-srv
[ 2111]     0  2111    81769      254   0       0             0 beah-beaker-bac
[ 2128]     0  2128    55652      217   0       0             0 beah-fwd-backen
[ 2154]     0  2154     1021      126   0       0             0 agetty
[ 2157]     0  2157     1017      111   0       0             0 mingetty
[ 2159]     0  2159     1017      111   0       0             0 mingetty
[ 2161]     0  2161     1017      111   0       0             0 mingetty
[ 2163]     0  2163     1017      111   0       0             0 mingetty
[ 2165]     0  2165     1017      111   0       0             0 mingetty
[ 2167]     0  2167     1017      111   0       0             0 mingetty
[ 2193]     0  2193    37543      242   0       0             0 beah-rhts-task
[ 2317]     0  2317    25518      247   0       0             0 sshd
[ 2321]     0  2321    27089      189   0       0             0 bash
[19773]     0 19773   147034    31718   0     -17         -1000 dmeventd
[ 8876]     0  8876     2837      118   0     -17         -1000 udevd
[ 8879]     0  8879     2837      118   0     -17         -1000 udevd
[11921]     0 11921     2804      118   0     -17         -1000 udevd
[11922]     0 11922     2804      118   0     -17         -1000 udevd
[11944]     0 11944     2804      118   0     -17         -1000 udevd
[11948]     0 11948     2804      118   0     -17         -1000 udevd
[11954]     0 11954     2804      118   0     -17         -1000 udevd
[12012]     0 12012     2837      118   0     -17         -1000 udevd
[12016]     0 12016     2837      118   0     -17         -1000 udevd
[12039]     0 12039     2837      126   0     -17         -1000 udevd
[13481]     0 13481     1607      156   0       0             0 qarshd
[13482]     0 13482    38533     7515   0       0             0 vgchange
[13949]     0 13949     8149      255   0     -17         -1000 multipath
[13953]     0 13953     5519      170   0     -17         -1000 dmsetup
[13954]     0 13954     4980       40   0     -17         -1000 dmsetup
[13956]     0 13956     5497       41   0     -17         -1000 dmsetup
[13957]     0 13957     4336       37   0     -17         -1000 dmsetup
[13958]     0 13958     2804        0   0     -17         -1000 udevd
Out of memory: Kill process 1030 (iscsiuio) score 1 or sacrifice child
Killed process 1030, UID 0, (iscsiuio) total-vm:28960kB, anon-rss:20812kB, file-rss:1996kB
bio: create slab <bio-1> at 1
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-12) discard unsupported: Disabling discard passdown.
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-32) discard unsupported: Disabling discard passdown.
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX

[...]

md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-313) discard unsupported: Disabling discard passdown.
udevd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=-17, oom_score_adj=-1000
udevd cpuset=/ mems_allowed=0
Pid: 9831, comm: udevd Not tainted 2.6.32-616.el6.x86_64 #1
Call Trace:
 [<ffffffff81131620>] ? dump_header+0x90/0x1b0
 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30
 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff8113199e>] ? select_bad_process+0x9e/0x120
 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950
 [<ffffffff8107a438>] ? copy_process+0x168/0x1520
 [<ffffffff81052204>] ? __do_page_fault+0x1f4/0x500
 [<ffffffff81242f46>] ? selinux_file_alloc_security+0x46/0x70
 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0
 [<ffffffff811ba0e2>] ? alloc_fd+0x92/0x160
 [<ffffffff81196807>] ? fd_install+0x47/0x90
 [<ffffffff81009598>] ? sys_clone+0x28/0x30
 [<ffffffff8100b3f3>] ? stub_clone+0x13/0x20
 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
active_anon:0 inactive_anon:27 isolated_anon:0
 active_file:2 inactive_file:24 isolated_file:0
 unevictable:40378 dirty:0 writeback:20 unstable:0
 free:16597 slab_reclaimable:2605 slab_unreclaimable:33862
 mapped:1895 shmem:4 pagetables:1450 bounce:0
Node 0 DMA free:4656kB min:668kB low:832kB high:1000kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:0kB unevictable:5904kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:5904kB dirty:4kB writeback:0kB mapped:0kB shmem:8kB slab_reclaimable:32kB slab_unreclaimable:3120kB kernel_stack:352kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 994 994 994
Node 0 DMA32 free:61732kB min:44384kB low:55480kB high:66576kB active_anon:0kB inactive_anon:108kB active_file:0kB inactive_file:96kB unevictable:155608kB isolated(anon):0kB isolated(file):0kB present:1018068kB mlocked:147432kB dirty:0kB writeback:80kB mapped:7580kB shmem:8kB slab_reclaimable:10388kB slab_unreclaimable:132328kB kernel_stack:10080kB pagetables:5800kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:113 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 60*4kB 40*8kB 30*16kB 31*32kB 9*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4656kB
Node 0 DMA32: 12165*4kB 1632*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61732kB
1946 total pagecache pages
30 pages in swap cache
Swap cache stats: add 2924824, delete 2924794, find 4003741/4406241
Free swap  = 753148kB
Total swap = 835580kB
262140 pages RAM
7108 pages reserved
11389 pages shared
227750 pages non-shared
[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[  455]     0   455     2772       71   0     -17         -1000 udevd
[  984]     0   984    76003     1528   0     -17         -1000 multipathd
[ 1036]     0  1036     1235      100   0       0             0 iscsid
[ 1037]     0  1037     2530     1510   0       0           -17 iscsid
[ 1247]     0  1247     2281       66   0       0             0 dhclient
[ 1337]     0  1337     2281       68   0       0             0 dhclient
[ 1397]     0  1397     6900      118   0     -17         -1000 auditd
[ 1431]     0  1431    62369      156   0       0             0 rsyslogd
[ 1668]    32  1668     4746      121   0       0             0 rpcbind
[ 1690]    29  1690     5839      168   0       0             0 rpc.statd
[ 1724]    81  1724     7953      135   0       0             0 dbus-daemon
[ 1746]     0  1746    47244      206   0       0             0 cupsd
[ 1778]     0  1778     1021      127   0       0             0 acpid
[ 1790]    68  1790     9580      196   0       0             0 hald
[ 1791]     0  1791     5101      127   0       0             0 hald-runner
[ 1823]     0  1823     5631      114   0       0             0 hald-addon-inpu
[ 1848]    68  1848     4503      156   0       0             0 hald-addon-acpi
[ 1880]     0  1880    96537      173   0       0             0 automount
[ 1936]     0  1936    16559       86   0     -17         -1000 sshd
[ 1947]     0  1947     5430      148   0       0             0 xinetd
[ 1964]     0  1964     1014      130   0       0             0 btimed
[ 1975]    38  1975     7685      209   0       0             0 ntpd
[ 1996]     0  1996    23261       96   0       0             0 sendmail
[ 2006]    51  2006    20080       93   0       0             0 sendmail
[ 2034]     0  2034    45759      189   0       0             0 abrtd
[ 2046]     0  2046    29220      166   0       0             0 crond
[ 2061]     0  2061     5278       71   0       0             0 atd
[ 2077]     0  2077    27088       88   0       0             0 rhsmcertd
[ 2094]     0  2094    59461      247   0       0             0 beah-srv
[ 2111]     0  2111    81769      254   0       0             0 beah-beaker-bac
[ 2128]     0  2128    55652      217   0       0             0 beah-fwd-backen
[ 2154]     0  2154     1021      126   0       0             0 agetty
[ 2157]     0  2157     1017      111   0       0             0 mingetty
[ 2159]     0  2159     1017      111   0       0             0 mingetty
[ 2161]     0  2161     1017      111   0       0             0 mingetty
[ 2163]     0  2163     1017      111   0       0             0 mingetty
[ 2165]     0  2165     1017      111   0       0             0 mingetty
[ 2167]     0  2167     1017      111   0       0             0 mingetty
[ 2193]     0  2193    37543      242   0       0             0 beah-rhts-task
[ 2317]     0  2317    25518      247   0       0             0 sshd
[ 2321]     0  2321    27089      189   0       0             0 bash
[19773]     0 19773   147034    31804   0     -17         -1000 dmeventd
[ 6768]     0  6768     2837      122   0     -17         -1000 udevd
[ 6785]     0  6785     2837      122   0     -17         -1000 udevd
[ 9831]     0  9831     2804      122   0     -17         -1000 udevd
[ 9832]     0  9832     2804      122   0     -17         -1000 udevd
[ 9856]     0  9856     2804      122   0     -17         -1000 udevd
[ 9862]     0  9862     2804      122   0     -17         -1000 udevd
[ 9866]     0  9866     2804      122   0     -17         -1000 udevd
[ 9932]     0  9932     2837      122   0     -17         -1000 udevd
[ 9951]     0  9951     2837      122   0     -17         -1000 udevd
[10423]     0 10423     2836      122   0     -17         -1000 udevd
[11389]     0 11389     1607      156   0       0             0 qarshd
[11390]     0 11390    38533     7515   0       0             0 vgchange
[11786]     0 11786     6608      137   0     -17         -1000 multipath
[11789]     0 11789     6608      191   0     -17         -1000 multipath
[11790]     0 11790     6051       46   0     -17         -1000 multipath
[11791]     0 11791     2837        0   0     -17         -1000 udevd
[11792]     0 11792     2837        0   0     -17         -1000 udevd
Out of memory: Kill process 1036 (iscsid) score 1 or sacrifice child
Killed process 1036, UID 0, (iscsid) total-vm:4940kB, anon-rss:16kB, file-rss:384kB
kthreadd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
kthreadd cpuset=/ mems_allowed=0
Pid: 2, comm: kthreadd Not tainted 2.6.32-616.el6.x86_64 #1
Call Trace:
 [<ffffffff81131620>] ? dump_header+0x90/0x1b0
 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30
 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff811319e1>] ? select_bad_process+0xe1/0x120
 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950
 [<ffffffff8107a438>] ? copy_process+0x168/0x1520
 [<ffffffff8106b4e3>] ? perf_event_task_sched_out+0x33/0x70
 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0
 [<ffffffff81065990>] ? __dequeue_entity+0x30/0x50
 [<ffffffff8100969d>] ? __switch_to+0x7d/0x340
 [<ffffffffa0004570>] ? dm_wq_work+0x0/0xe0 [dm_mod]
 [<ffffffffa0004570>] ? dm_wq_work+0x0/0xe0 [dm_mod]
 [<ffffffff8100c222>] ? kernel_thread+0x82/0xe0
 [<ffffffff810a6590>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
 [<ffffffff810a6536>] ? kthreadd+0x196/0x1f0
 [<ffffffff8100c28a>] ? child_rip+0xa/0x20
 [<ffffffff810a63a0>] ? kthreadd+0x0/0x1f0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20



Version-Release number of selected component (if applicable):
2.6.32-616.el6.x86_64

lvm2-2.02.143-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
lvm2-libs-2.02.143-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
lvm2-cluster-2.02.143-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
udev-147-2.71.el6    BUILT: Wed Feb 10 07:07:17 CST 2016
device-mapper-1.02.117-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
device-mapper-libs-1.02.117-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
device-mapper-event-1.02.117-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
device-mapper-event-libs-1.02.117-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
device-mapper-persistent-data-0.6.2-0.1.rc5.el6    BUILT: Wed Feb 24 07:07:09 CST 2016
cmirror-2.02.143-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016