Bug 1315417 - Potential memory leak when stressing lvm activation operations
Potential memory leak when stressing lvm activation operations
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.8
x86_64 Linux
unspecified Severity low
: rc
: ---
Assigned To: LVM and device-mapper development team
cluster-qe@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-07 11:49 EST by Corey Marthaler
Modified: 2017-10-03 21:06 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-03 21:06:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2016-03-07 11:49:52 EST
Description of problem:
Basically this test just creates many different types of volumes, and then deactivates and reactivates them over and over. It takes many hours/days to hit this, so this is definitely very low priority.



./activator -i 10000

Iteration 4811 of 10000 started at Sat Mar  5 23:44:24 CST 2016
Deactivating volumes on VG activator1 on host-113.virt.lab.msp.redhat.com
Deactivating volumes on VG activator2 on host-113.virt.lab.msp.redhat.com
Deactivating volumes on VG activator3 on host-113.virt.lab.msp.redhat.com
Deactivating volumes on VG activator4 on host-113.virt.lab.msp.redhat.com
Verifying that all volumes are deactivated
Verifying that all volumes no longer exist in device-mapper
Reactivating volumes on VG activator1 on host-113.virt.lab.msp.redhat.com
Reactivating volumes on VG activator2 on host-113.virt.lab.msp.redhat.com
Reactivating volumes on VG activator3 on host-113.virt.lab.msp.redhat.com
Reactivating volumes on VG activator4 on host-113.virt.lab.msp.redhat.com
recv_packet() returned NULL!
Remote command on host-113.virt.lab.msp.redhat.com exited with unknown state
couldn't reactivate on host-113.virt.lab.msp.redhat.com




device-mapper: thin: Data device (dm-293) discard unsupported: Disabling discard passdown.
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-313) discard unsupported: Disabling discard passdown.
md/raid1:mdX: active with 4 out of 4 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-337) discard unsupported: Disabling discard passdown.
kthreadd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
kthreadd cpuset=/ mems_allowed=0
Pid: 2, comm: kthreadd Not tainted 2.6.32-616.el6.x86_64 #1
Call Trace:
 [<ffffffff81131620>] ? dump_header+0x90/0x1b0
 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30
 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff811319e1>] ? select_bad_process+0xe1/0x120
 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950
 [<ffffffff8107a438>] ? copy_process+0x168/0x1520
 [<ffffffff8106b4e3>] ? perf_event_task_sched_out+0x33/0x70
 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0
 [<ffffffff81065990>] ? __dequeue_entity+0x30/0x50
 [<ffffffff8100970a>] ? __switch_to+0xea/0x340
 [<ffffffff8112e640>] ? sync_page_killable+0x0/0x40
 [<ffffffff8112e640>] ? sync_page_killable+0x0/0x40
 [<ffffffff8100c222>] ? kernel_thread+0x82/0xe0
 [<ffffffff810a6590>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
 [<ffffffff810a6536>] ? kthreadd+0x196/0x1f0
 [<ffffffff8100c28a>] ? child_rip+0xa/0x20
 [<ffffffff810a63a0>] ? kthreadd+0x0/0x1f0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:   1
active_anon:2 inactive_anon:27 isolated_anon:0
 active_file:19 inactive_file:0 isolated_file:0
 unevictable:45543 dirty:0 writeback:0 unstable:0
 free:15605 slab_reclaimable:2620 slab_unreclaimable:34494
 mapped:1929 shmem:10 pagetables:1468 bounce:0
Node 0 DMA free:4656kB min:668kB low:832kB high:1000kB active_anon:8kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:6644kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:6644kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:2612kB kernel_stack:608kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 994 994 994
Node 0 DMA32 free:57764kB min:44384kB low:55480kB high:66576kB active_anon:0kB inactive_anon:108kB active_file:72kB inactive_file:0kB unevictable:175528kB isolated(anon):0kB isolated(file):0kB present:1018068kB mlocked:151000kB dirty:0kB writeback:0kB mapped:7716kB shmem:40kB slab_reclaimable:10424kB slab_unreclaimable:135364kB kernel_stack:10112kB pagetables:5876kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 30*4kB 23*8kB 24*16kB 26*32kB 15*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4656kB
Node 0 DMA32: 11553*4kB 1442*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 57764kB
1992 total pagecache pages
34 pages in swap cache
Swap cache stats: add 1626302, delete 1626268, find 2823798/3043135
Free swap  = 753060kB
Total swap = 835580kB
262140 pages RAM
7108 pages reserved
12016 pages shared
228581 pages non-shared
[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[  455]     0   455     2772       67   0     -17         -1000 udevd
[  984]     0   984    76003     1528   0     -17         -1000 multipathd
[ 1030]     0  1030     7240     5702   0       0           -17 iscsiuio
[ 1036]     0  1036     1235       99   0       0             0 iscsid
[ 1037]     0  1037     2530     1510   0       0           -17 iscsid
[ 1247]     0  1247     2281       66   0       0             0 dhclient
[ 1337]     0  1337     2281       68   0       0             0 dhclient
[ 1397]     0  1397     6900      118   0     -17         -1000 auditd
[ 1431]     0  1431    62369      153   0       0             0 rsyslogd
[ 1668]    32  1668     4746      121   0       0             0 rpcbind
[ 1690]    29  1690     5839      168   0       0             0 rpc.statd
[ 1724]    81  1724     7953      135   0       0             0 dbus-daemon
[ 1746]     0  1746    47244      206   0       0             0 cupsd
[ 1778]     0  1778     1021      127   0       0             0 acpid
[ 1790]    68  1790     9580      198   0       0             0 hald
[ 1791]     0  1791     5101      127   0       0             0 hald-runner
[ 1823]     0  1823     5631      114   0       0             0 hald-addon-inpu
[ 1848]    68  1848     4503      156   0       0             0 hald-addon-acpi
[ 1880]     0  1880    96537      173   0       0             0 automount
[ 1936]     0  1936    16559       86   0     -17         -1000 sshd
[ 1947]     0  1947     5430      148   0       0             0 xinetd
[ 1964]     0  1964     1014      130   0       0             0 btimed
[ 1975]    38  1975     7685      209   0       0             0 ntpd
[ 1996]     0  1996    23261       96   0       0             0 sendmail
[ 2006]    51  2006    20080       93   0       0             0 sendmail
[ 2034]     0  2034    45759      189   0       0             0 abrtd
[ 2046]     0  2046    29220      166   0       0             0 crond
[ 2061]     0  2061     5278       71   0       0             0 atd
[ 2077]     0  2077    27088       88   0       0             0 rhsmcertd
[ 2094]     0  2094    59461      247   0       0             0 beah-srv
[ 2111]     0  2111    81769      254   0       0             0 beah-beaker-bac
[ 2128]     0  2128    55652      217   0       0             0 beah-fwd-backen
[ 2154]     0  2154     1021      126   0       0             0 agetty
[ 2157]     0  2157     1017      111   0       0             0 mingetty
[ 2159]     0  2159     1017      111   0       0             0 mingetty
[ 2161]     0  2161     1017      111   0       0             0 mingetty
[ 2163]     0  2163     1017      111   0       0             0 mingetty
[ 2165]     0  2165     1017      111   0       0             0 mingetty
[ 2167]     0  2167     1017      111   0       0             0 mingetty
[ 2193]     0  2193    37543      242   0       0             0 beah-rhts-task
[ 2317]     0  2317    25518      247   0       0             0 sshd
[ 2321]     0  2321    27089      189   0       0             0 bash
[19773]     0 19773   147034    31718   0     -17         -1000 dmeventd
[ 8876]     0  8876     2837      118   0     -17         -1000 udevd
[ 8879]     0  8879     2837      118   0     -17         -1000 udevd
[11921]     0 11921     2804      118   0     -17         -1000 udevd
[11922]     0 11922     2804      118   0     -17         -1000 udevd
[11944]     0 11944     2804      118   0     -17         -1000 udevd
[11948]     0 11948     2804      118   0     -17         -1000 udevd
[11954]     0 11954     2804      118   0     -17         -1000 udevd
[12012]     0 12012     2837      118   0     -17         -1000 udevd
[12016]     0 12016     2837      118   0     -17         -1000 udevd
[12039]     0 12039     2837      126   0     -17         -1000 udevd
[13481]     0 13481     1607      156   0       0             0 qarshd
[13482]     0 13482    38533     7515   0       0             0 vgchange
[13949]     0 13949     8149      255   0     -17         -1000 multipath
[13953]     0 13953     5519      170   0     -17         -1000 dmsetup
[13954]     0 13954     4980       40   0     -17         -1000 dmsetup
[13956]     0 13956     5497       41   0     -17         -1000 dmsetup
[13957]     0 13957     4336       37   0     -17         -1000 dmsetup
[13958]     0 13958     2804        0   0     -17         -1000 udevd
Out of memory: Kill process 1030 (iscsiuio) score 1 or sacrifice child
Killed process 1030, UID 0, (iscsiuio) total-vm:28960kB, anon-rss:20812kB, file-rss:1996kB
bio: create slab <bio-1> at 1
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-12) discard unsupported: Disabling discard passdown.
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-32) discard unsupported: Disabling discard passdown.
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX

[...]

md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (1 pages) for device mdX
mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits
device-mapper: thin: Data device (dm-313) discard unsupported: Disabling discard passdown.
udevd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=-17, oom_score_adj=-1000
udevd cpuset=/ mems_allowed=0
Pid: 9831, comm: udevd Not tainted 2.6.32-616.el6.x86_64 #1
Call Trace:
 [<ffffffff81131620>] ? dump_header+0x90/0x1b0
 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30
 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff8113199e>] ? select_bad_process+0x9e/0x120
 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950
 [<ffffffff8107a438>] ? copy_process+0x168/0x1520
 [<ffffffff81052204>] ? __do_page_fault+0x1f4/0x500
 [<ffffffff81242f46>] ? selinux_file_alloc_security+0x46/0x70
 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0
 [<ffffffff811ba0e2>] ? alloc_fd+0x92/0x160
 [<ffffffff81196807>] ? fd_install+0x47/0x90
 [<ffffffff81009598>] ? sys_clone+0x28/0x30
 [<ffffffff8100b3f3>] ? stub_clone+0x13/0x20
 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
active_anon:0 inactive_anon:27 isolated_anon:0
 active_file:2 inactive_file:24 isolated_file:0
 unevictable:40378 dirty:0 writeback:20 unstable:0
 free:16597 slab_reclaimable:2605 slab_unreclaimable:33862
 mapped:1895 shmem:4 pagetables:1450 bounce:0
Node 0 DMA free:4656kB min:668kB low:832kB high:1000kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:0kB unevictable:5904kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:5904kB dirty:4kB writeback:0kB mapped:0kB shmem:8kB slab_reclaimable:32kB slab_unreclaimable:3120kB kernel_stack:352kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 994 994 994
Node 0 DMA32 free:61732kB min:44384kB low:55480kB high:66576kB active_anon:0kB inactive_anon:108kB active_file:0kB inactive_file:96kB unevictable:155608kB isolated(anon):0kB isolated(file):0kB present:1018068kB mlocked:147432kB dirty:0kB writeback:80kB mapped:7580kB shmem:8kB slab_reclaimable:10388kB slab_unreclaimable:132328kB kernel_stack:10080kB pagetables:5800kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:113 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 60*4kB 40*8kB 30*16kB 31*32kB 9*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4656kB
Node 0 DMA32: 12165*4kB 1632*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61732kB
1946 total pagecache pages
30 pages in swap cache
Swap cache stats: add 2924824, delete 2924794, find 4003741/4406241
Free swap  = 753148kB
Total swap = 835580kB
262140 pages RAM
7108 pages reserved
11389 pages shared
227750 pages non-shared
[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[  455]     0   455     2772       71   0     -17         -1000 udevd
[  984]     0   984    76003     1528   0     -17         -1000 multipathd
[ 1036]     0  1036     1235      100   0       0             0 iscsid
[ 1037]     0  1037     2530     1510   0       0           -17 iscsid
[ 1247]     0  1247     2281       66   0       0             0 dhclient
[ 1337]     0  1337     2281       68   0       0             0 dhclient
[ 1397]     0  1397     6900      118   0     -17         -1000 auditd
[ 1431]     0  1431    62369      156   0       0             0 rsyslogd
[ 1668]    32  1668     4746      121   0       0             0 rpcbind
[ 1690]    29  1690     5839      168   0       0             0 rpc.statd
[ 1724]    81  1724     7953      135   0       0             0 dbus-daemon
[ 1746]     0  1746    47244      206   0       0             0 cupsd
[ 1778]     0  1778     1021      127   0       0             0 acpid
[ 1790]    68  1790     9580      196   0       0             0 hald
[ 1791]     0  1791     5101      127   0       0             0 hald-runner
[ 1823]     0  1823     5631      114   0       0             0 hald-addon-inpu
[ 1848]    68  1848     4503      156   0       0             0 hald-addon-acpi
[ 1880]     0  1880    96537      173   0       0             0 automount
[ 1936]     0  1936    16559       86   0     -17         -1000 sshd
[ 1947]     0  1947     5430      148   0       0             0 xinetd
[ 1964]     0  1964     1014      130   0       0             0 btimed
[ 1975]    38  1975     7685      209   0       0             0 ntpd
[ 1996]     0  1996    23261       96   0       0             0 sendmail
[ 2006]    51  2006    20080       93   0       0             0 sendmail
[ 2034]     0  2034    45759      189   0       0             0 abrtd
[ 2046]     0  2046    29220      166   0       0             0 crond
[ 2061]     0  2061     5278       71   0       0             0 atd
[ 2077]     0  2077    27088       88   0       0             0 rhsmcertd
[ 2094]     0  2094    59461      247   0       0             0 beah-srv
[ 2111]     0  2111    81769      254   0       0             0 beah-beaker-bac
[ 2128]     0  2128    55652      217   0       0             0 beah-fwd-backen
[ 2154]     0  2154     1021      126   0       0             0 agetty
[ 2157]     0  2157     1017      111   0       0             0 mingetty
[ 2159]     0  2159     1017      111   0       0             0 mingetty
[ 2161]     0  2161     1017      111   0       0             0 mingetty
[ 2163]     0  2163     1017      111   0       0             0 mingetty
[ 2165]     0  2165     1017      111   0       0             0 mingetty
[ 2167]     0  2167     1017      111   0       0             0 mingetty
[ 2193]     0  2193    37543      242   0       0             0 beah-rhts-task
[ 2317]     0  2317    25518      247   0       0             0 sshd
[ 2321]     0  2321    27089      189   0       0             0 bash
[19773]     0 19773   147034    31804   0     -17         -1000 dmeventd
[ 6768]     0  6768     2837      122   0     -17         -1000 udevd
[ 6785]     0  6785     2837      122   0     -17         -1000 udevd
[ 9831]     0  9831     2804      122   0     -17         -1000 udevd
[ 9832]     0  9832     2804      122   0     -17         -1000 udevd
[ 9856]     0  9856     2804      122   0     -17         -1000 udevd
[ 9862]     0  9862     2804      122   0     -17         -1000 udevd
[ 9866]     0  9866     2804      122   0     -17         -1000 udevd
[ 9932]     0  9932     2837      122   0     -17         -1000 udevd
[ 9951]     0  9951     2837      122   0     -17         -1000 udevd
[10423]     0 10423     2836      122   0     -17         -1000 udevd
[11389]     0 11389     1607      156   0       0             0 qarshd
[11390]     0 11390    38533     7515   0       0             0 vgchange
[11786]     0 11786     6608      137   0     -17         -1000 multipath
[11789]     0 11789     6608      191   0     -17         -1000 multipath
[11790]     0 11790     6051       46   0     -17         -1000 multipath
[11791]     0 11791     2837        0   0     -17         -1000 udevd
[11792]     0 11792     2837        0   0     -17         -1000 udevd
Out of memory: Kill process 1036 (iscsid) score 1 or sacrifice child
Killed process 1036, UID 0, (iscsid) total-vm:4940kB, anon-rss:16kB, file-rss:384kB
kthreadd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
kthreadd cpuset=/ mems_allowed=0
Pid: 2, comm: kthreadd Not tainted 2.6.32-616.el6.x86_64 #1
Call Trace:
 [<ffffffff81131620>] ? dump_header+0x90/0x1b0
 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30
 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff811319e1>] ? select_bad_process+0xe1/0x120
 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950
 [<ffffffff8107a438>] ? copy_process+0x168/0x1520
 [<ffffffff8106b4e3>] ? perf_event_task_sched_out+0x33/0x70
 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0
 [<ffffffff81065990>] ? __dequeue_entity+0x30/0x50
 [<ffffffff8100969d>] ? __switch_to+0x7d/0x340
 [<ffffffffa0004570>] ? dm_wq_work+0x0/0xe0 [dm_mod]
 [<ffffffffa0004570>] ? dm_wq_work+0x0/0xe0 [dm_mod]
 [<ffffffff8100c222>] ? kernel_thread+0x82/0xe0
 [<ffffffff810a6590>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
 [<ffffffff810a6536>] ? kthreadd+0x196/0x1f0
 [<ffffffff8100c28a>] ? child_rip+0xa/0x20
 [<ffffffff810a63a0>] ? kthreadd+0x0/0x1f0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20



Version-Release number of selected component (if applicable):
2.6.32-616.el6.x86_64

lvm2-2.02.143-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
lvm2-libs-2.02.143-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
lvm2-cluster-2.02.143-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
udev-147-2.71.el6    BUILT: Wed Feb 10 07:07:17 CST 2016
device-mapper-1.02.117-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
device-mapper-libs-1.02.117-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
device-mapper-event-1.02.117-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
device-mapper-event-libs-1.02.117-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016
device-mapper-persistent-data-0.6.2-0.1.rc5.el6    BUILT: Wed Feb 24 07:07:09 CST 2016
cmirror-2.02.143-1.el6    BUILT: Wed Feb 24 07:59:50 CST 2016

Note You need to log in before you can comment on or make changes to this bug.