Hide Forgot
Description of problem: Basically this test just creates many different types of volumes, and then deactivates and reactivates them over and over. It takes many hours/days to hit this, so this is definitely very low priority. ./activator -i 10000 Iteration 4811 of 10000 started at Sat Mar 5 23:44:24 CST 2016 Deactivating volumes on VG activator1 on host-113.virt.lab.msp.redhat.com Deactivating volumes on VG activator2 on host-113.virt.lab.msp.redhat.com Deactivating volumes on VG activator3 on host-113.virt.lab.msp.redhat.com Deactivating volumes on VG activator4 on host-113.virt.lab.msp.redhat.com Verifying that all volumes are deactivated Verifying that all volumes no longer exist in device-mapper Reactivating volumes on VG activator1 on host-113.virt.lab.msp.redhat.com Reactivating volumes on VG activator2 on host-113.virt.lab.msp.redhat.com Reactivating volumes on VG activator3 on host-113.virt.lab.msp.redhat.com Reactivating volumes on VG activator4 on host-113.virt.lab.msp.redhat.com recv_packet() returned NULL! Remote command on host-113.virt.lab.msp.redhat.com exited with unknown state couldn't reactivate on host-113.virt.lab.msp.redhat.com device-mapper: thin: Data device (dm-293) discard unsupported: Disabling discard passdown. md/raid1:mdX: active with 2 out of 2 mirrors created bitmap (1 pages) for device mdX mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits device-mapper: thin: Data device (dm-313) discard unsupported: Disabling discard passdown. md/raid1:mdX: active with 4 out of 4 mirrors created bitmap (1 pages) for device mdX mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits device-mapper: thin: Data device (dm-337) discard unsupported: Disabling discard passdown. kthreadd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0 kthreadd cpuset=/ mems_allowed=0 Pid: 2, comm: kthreadd Not tainted 2.6.32-616.el6.x86_64 #1 Call Trace: [<ffffffff81131620>] ? dump_header+0x90/0x1b0 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0 [<ffffffff811319e1>] ? select_bad_process+0xe1/0x120 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950 [<ffffffff8107a438>] ? copy_process+0x168/0x1520 [<ffffffff8106b4e3>] ? perf_event_task_sched_out+0x33/0x70 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0 [<ffffffff81065990>] ? __dequeue_entity+0x30/0x50 [<ffffffff8100970a>] ? __switch_to+0xea/0x340 [<ffffffff8112e640>] ? sync_page_killable+0x0/0x40 [<ffffffff8112e640>] ? sync_page_killable+0x0/0x40 [<ffffffff8100c222>] ? kernel_thread+0x82/0xe0 [<ffffffff810a6590>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 [<ffffffff810a6536>] ? kthreadd+0x196/0x1f0 [<ffffffff8100c28a>] ? child_rip+0xa/0x20 [<ffffffff810a63a0>] ? kthreadd+0x0/0x1f0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 1 active_anon:2 inactive_anon:27 isolated_anon:0 active_file:19 inactive_file:0 isolated_file:0 unevictable:45543 dirty:0 writeback:0 unstable:0 free:15605 slab_reclaimable:2620 slab_unreclaimable:34494 mapped:1929 shmem:10 pagetables:1468 bounce:0 Node 0 DMA free:4656kB min:668kB low:832kB high:1000kB active_anon:8kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:6644kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:6644kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:2612kB kernel_stack:608kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 994 994 994 Node 0 DMA32 free:57764kB min:44384kB low:55480kB high:66576kB active_anon:0kB inactive_anon:108kB active_file:72kB inactive_file:0kB unevictable:175528kB isolated(anon):0kB isolated(file):0kB present:1018068kB mlocked:151000kB dirty:0kB writeback:0kB mapped:7716kB shmem:40kB slab_reclaimable:10424kB slab_unreclaimable:135364kB kernel_stack:10112kB pagetables:5876kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 30*4kB 23*8kB 24*16kB 26*32kB 15*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4656kB Node 0 DMA32: 11553*4kB 1442*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 57764kB 1992 total pagecache pages 34 pages in swap cache Swap cache stats: add 1626302, delete 1626268, find 2823798/3043135 Free swap = 753060kB Total swap = 835580kB 262140 pages RAM 7108 pages reserved 12016 pages shared 228581 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 455] 0 455 2772 67 0 -17 -1000 udevd [ 984] 0 984 76003 1528 0 -17 -1000 multipathd [ 1030] 0 1030 7240 5702 0 0 -17 iscsiuio [ 1036] 0 1036 1235 99 0 0 0 iscsid [ 1037] 0 1037 2530 1510 0 0 -17 iscsid [ 1247] 0 1247 2281 66 0 0 0 dhclient [ 1337] 0 1337 2281 68 0 0 0 dhclient [ 1397] 0 1397 6900 118 0 -17 -1000 auditd [ 1431] 0 1431 62369 153 0 0 0 rsyslogd [ 1668] 32 1668 4746 121 0 0 0 rpcbind [ 1690] 29 1690 5839 168 0 0 0 rpc.statd [ 1724] 81 1724 7953 135 0 0 0 dbus-daemon [ 1746] 0 1746 47244 206 0 0 0 cupsd [ 1778] 0 1778 1021 127 0 0 0 acpid [ 1790] 68 1790 9580 198 0 0 0 hald [ 1791] 0 1791 5101 127 0 0 0 hald-runner [ 1823] 0 1823 5631 114 0 0 0 hald-addon-inpu [ 1848] 68 1848 4503 156 0 0 0 hald-addon-acpi [ 1880] 0 1880 96537 173 0 0 0 automount [ 1936] 0 1936 16559 86 0 -17 -1000 sshd [ 1947] 0 1947 5430 148 0 0 0 xinetd [ 1964] 0 1964 1014 130 0 0 0 btimed [ 1975] 38 1975 7685 209 0 0 0 ntpd [ 1996] 0 1996 23261 96 0 0 0 sendmail [ 2006] 51 2006 20080 93 0 0 0 sendmail [ 2034] 0 2034 45759 189 0 0 0 abrtd [ 2046] 0 2046 29220 166 0 0 0 crond [ 2061] 0 2061 5278 71 0 0 0 atd [ 2077] 0 2077 27088 88 0 0 0 rhsmcertd [ 2094] 0 2094 59461 247 0 0 0 beah-srv [ 2111] 0 2111 81769 254 0 0 0 beah-beaker-bac [ 2128] 0 2128 55652 217 0 0 0 beah-fwd-backen [ 2154] 0 2154 1021 126 0 0 0 agetty [ 2157] 0 2157 1017 111 0 0 0 mingetty [ 2159] 0 2159 1017 111 0 0 0 mingetty [ 2161] 0 2161 1017 111 0 0 0 mingetty [ 2163] 0 2163 1017 111 0 0 0 mingetty [ 2165] 0 2165 1017 111 0 0 0 mingetty [ 2167] 0 2167 1017 111 0 0 0 mingetty [ 2193] 0 2193 37543 242 0 0 0 beah-rhts-task [ 2317] 0 2317 25518 247 0 0 0 sshd [ 2321] 0 2321 27089 189 0 0 0 bash [19773] 0 19773 147034 31718 0 -17 -1000 dmeventd [ 8876] 0 8876 2837 118 0 -17 -1000 udevd [ 8879] 0 8879 2837 118 0 -17 -1000 udevd [11921] 0 11921 2804 118 0 -17 -1000 udevd [11922] 0 11922 2804 118 0 -17 -1000 udevd [11944] 0 11944 2804 118 0 -17 -1000 udevd [11948] 0 11948 2804 118 0 -17 -1000 udevd [11954] 0 11954 2804 118 0 -17 -1000 udevd [12012] 0 12012 2837 118 0 -17 -1000 udevd [12016] 0 12016 2837 118 0 -17 -1000 udevd [12039] 0 12039 2837 126 0 -17 -1000 udevd [13481] 0 13481 1607 156 0 0 0 qarshd [13482] 0 13482 38533 7515 0 0 0 vgchange [13949] 0 13949 8149 255 0 -17 -1000 multipath [13953] 0 13953 5519 170 0 -17 -1000 dmsetup [13954] 0 13954 4980 40 0 -17 -1000 dmsetup [13956] 0 13956 5497 41 0 -17 -1000 dmsetup [13957] 0 13957 4336 37 0 -17 -1000 dmsetup [13958] 0 13958 2804 0 0 -17 -1000 udevd Out of memory: Kill process 1030 (iscsiuio) score 1 or sacrifice child Killed process 1030, UID 0, (iscsiuio) total-vm:28960kB, anon-rss:20812kB, file-rss:1996kB bio: create slab <bio-1> at 1 md/raid1:mdX: active with 2 out of 2 mirrors created bitmap (1 pages) for device mdX mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits device-mapper: thin: Data device (dm-12) discard unsupported: Disabling discard passdown. md/raid1:mdX: active with 2 out of 2 mirrors created bitmap (1 pages) for device mdX mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits device-mapper: thin: Data device (dm-32) discard unsupported: Disabling discard passdown. md/raid1:mdX: active with 2 out of 2 mirrors created bitmap (1 pages) for device mdX [...] md/raid1:mdX: active with 2 out of 2 mirrors created bitmap (1 pages) for device mdX mdX: bitmap initialized from disk: read 1 pages, set 0 of 304 bits device-mapper: thin: Data device (dm-313) discard unsupported: Disabling discard passdown. udevd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=-17, oom_score_adj=-1000 udevd cpuset=/ mems_allowed=0 Pid: 9831, comm: udevd Not tainted 2.6.32-616.el6.x86_64 #1 Call Trace: [<ffffffff81131620>] ? dump_header+0x90/0x1b0 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0 [<ffffffff8113199e>] ? select_bad_process+0x9e/0x120 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950 [<ffffffff8107a438>] ? copy_process+0x168/0x1520 [<ffffffff81052204>] ? __do_page_fault+0x1f4/0x500 [<ffffffff81242f46>] ? selinux_file_alloc_security+0x46/0x70 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0 [<ffffffff811ba0e2>] ? alloc_fd+0x92/0x160 [<ffffffff81196807>] ? fd_install+0x47/0x90 [<ffffffff81009598>] ? sys_clone+0x28/0x30 [<ffffffff8100b3f3>] ? stub_clone+0x13/0x20 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:27 isolated_anon:0 active_file:2 inactive_file:24 isolated_file:0 unevictable:40378 dirty:0 writeback:20 unstable:0 free:16597 slab_reclaimable:2605 slab_unreclaimable:33862 mapped:1895 shmem:4 pagetables:1450 bounce:0 Node 0 DMA free:4656kB min:668kB low:832kB high:1000kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:0kB unevictable:5904kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:5904kB dirty:4kB writeback:0kB mapped:0kB shmem:8kB slab_reclaimable:32kB slab_unreclaimable:3120kB kernel_stack:352kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 994 994 994 Node 0 DMA32 free:61732kB min:44384kB low:55480kB high:66576kB active_anon:0kB inactive_anon:108kB active_file:0kB inactive_file:96kB unevictable:155608kB isolated(anon):0kB isolated(file):0kB present:1018068kB mlocked:147432kB dirty:0kB writeback:80kB mapped:7580kB shmem:8kB slab_reclaimable:10388kB slab_unreclaimable:132328kB kernel_stack:10080kB pagetables:5800kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:113 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 60*4kB 40*8kB 30*16kB 31*32kB 9*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4656kB Node 0 DMA32: 12165*4kB 1632*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61732kB 1946 total pagecache pages 30 pages in swap cache Swap cache stats: add 2924824, delete 2924794, find 4003741/4406241 Free swap = 753148kB Total swap = 835580kB 262140 pages RAM 7108 pages reserved 11389 pages shared 227750 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 455] 0 455 2772 71 0 -17 -1000 udevd [ 984] 0 984 76003 1528 0 -17 -1000 multipathd [ 1036] 0 1036 1235 100 0 0 0 iscsid [ 1037] 0 1037 2530 1510 0 0 -17 iscsid [ 1247] 0 1247 2281 66 0 0 0 dhclient [ 1337] 0 1337 2281 68 0 0 0 dhclient [ 1397] 0 1397 6900 118 0 -17 -1000 auditd [ 1431] 0 1431 62369 156 0 0 0 rsyslogd [ 1668] 32 1668 4746 121 0 0 0 rpcbind [ 1690] 29 1690 5839 168 0 0 0 rpc.statd [ 1724] 81 1724 7953 135 0 0 0 dbus-daemon [ 1746] 0 1746 47244 206 0 0 0 cupsd [ 1778] 0 1778 1021 127 0 0 0 acpid [ 1790] 68 1790 9580 196 0 0 0 hald [ 1791] 0 1791 5101 127 0 0 0 hald-runner [ 1823] 0 1823 5631 114 0 0 0 hald-addon-inpu [ 1848] 68 1848 4503 156 0 0 0 hald-addon-acpi [ 1880] 0 1880 96537 173 0 0 0 automount [ 1936] 0 1936 16559 86 0 -17 -1000 sshd [ 1947] 0 1947 5430 148 0 0 0 xinetd [ 1964] 0 1964 1014 130 0 0 0 btimed [ 1975] 38 1975 7685 209 0 0 0 ntpd [ 1996] 0 1996 23261 96 0 0 0 sendmail [ 2006] 51 2006 20080 93 0 0 0 sendmail [ 2034] 0 2034 45759 189 0 0 0 abrtd [ 2046] 0 2046 29220 166 0 0 0 crond [ 2061] 0 2061 5278 71 0 0 0 atd [ 2077] 0 2077 27088 88 0 0 0 rhsmcertd [ 2094] 0 2094 59461 247 0 0 0 beah-srv [ 2111] 0 2111 81769 254 0 0 0 beah-beaker-bac [ 2128] 0 2128 55652 217 0 0 0 beah-fwd-backen [ 2154] 0 2154 1021 126 0 0 0 agetty [ 2157] 0 2157 1017 111 0 0 0 mingetty [ 2159] 0 2159 1017 111 0 0 0 mingetty [ 2161] 0 2161 1017 111 0 0 0 mingetty [ 2163] 0 2163 1017 111 0 0 0 mingetty [ 2165] 0 2165 1017 111 0 0 0 mingetty [ 2167] 0 2167 1017 111 0 0 0 mingetty [ 2193] 0 2193 37543 242 0 0 0 beah-rhts-task [ 2317] 0 2317 25518 247 0 0 0 sshd [ 2321] 0 2321 27089 189 0 0 0 bash [19773] 0 19773 147034 31804 0 -17 -1000 dmeventd [ 6768] 0 6768 2837 122 0 -17 -1000 udevd [ 6785] 0 6785 2837 122 0 -17 -1000 udevd [ 9831] 0 9831 2804 122 0 -17 -1000 udevd [ 9832] 0 9832 2804 122 0 -17 -1000 udevd [ 9856] 0 9856 2804 122 0 -17 -1000 udevd [ 9862] 0 9862 2804 122 0 -17 -1000 udevd [ 9866] 0 9866 2804 122 0 -17 -1000 udevd [ 9932] 0 9932 2837 122 0 -17 -1000 udevd [ 9951] 0 9951 2837 122 0 -17 -1000 udevd [10423] 0 10423 2836 122 0 -17 -1000 udevd [11389] 0 11389 1607 156 0 0 0 qarshd [11390] 0 11390 38533 7515 0 0 0 vgchange [11786] 0 11786 6608 137 0 -17 -1000 multipath [11789] 0 11789 6608 191 0 -17 -1000 multipath [11790] 0 11790 6051 46 0 -17 -1000 multipath [11791] 0 11791 2837 0 0 -17 -1000 udevd [11792] 0 11792 2837 0 0 -17 -1000 udevd Out of memory: Kill process 1036 (iscsid) score 1 or sacrifice child Killed process 1036, UID 0, (iscsid) total-vm:4940kB, anon-rss:16kB, file-rss:384kB kthreadd invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0 kthreadd cpuset=/ mems_allowed=0 Pid: 2, comm: kthreadd Not tainted 2.6.32-616.el6.x86_64 #1 Call Trace: [<ffffffff81131620>] ? dump_header+0x90/0x1b0 [<ffffffff810dd981>] ? cpuset_mems_allowed_intersects+0x21/0x30 [<ffffffff81131aa2>] ? oom_kill_process+0x82/0x2a0 [<ffffffff811319e1>] ? select_bad_process+0xe1/0x120 [<ffffffff81131ee0>] ? out_of_memory+0x220/0x3c0 [<ffffffff8113e8bc>] ? __alloc_pages_nodemask+0x93c/0x950 [<ffffffff8107a438>] ? copy_process+0x168/0x1520 [<ffffffff8106b4e3>] ? perf_event_task_sched_out+0x33/0x70 [<ffffffff8107b946>] ? do_fork+0x96/0x4c0 [<ffffffff81065990>] ? __dequeue_entity+0x30/0x50 [<ffffffff8100969d>] ? __switch_to+0x7d/0x340 [<ffffffffa0004570>] ? dm_wq_work+0x0/0xe0 [dm_mod] [<ffffffffa0004570>] ? dm_wq_work+0x0/0xe0 [dm_mod] [<ffffffff8100c222>] ? kernel_thread+0x82/0xe0 [<ffffffff810a6590>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 [<ffffffff810a6536>] ? kthreadd+0x196/0x1f0 [<ffffffff8100c28a>] ? child_rip+0xa/0x20 [<ffffffff810a63a0>] ? kthreadd+0x0/0x1f0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 Version-Release number of selected component (if applicable): 2.6.32-616.el6.x86_64 lvm2-2.02.143-1.el6 BUILT: Wed Feb 24 07:59:50 CST 2016 lvm2-libs-2.02.143-1.el6 BUILT: Wed Feb 24 07:59:50 CST 2016 lvm2-cluster-2.02.143-1.el6 BUILT: Wed Feb 24 07:59:50 CST 2016 udev-147-2.71.el6 BUILT: Wed Feb 10 07:07:17 CST 2016 device-mapper-1.02.117-1.el6 BUILT: Wed Feb 24 07:59:50 CST 2016 device-mapper-libs-1.02.117-1.el6 BUILT: Wed Feb 24 07:59:50 CST 2016 device-mapper-event-1.02.117-1.el6 BUILT: Wed Feb 24 07:59:50 CST 2016 device-mapper-event-libs-1.02.117-1.el6 BUILT: Wed Feb 24 07:59:50 CST 2016 device-mapper-persistent-data-0.6.2-0.1.rc5.el6 BUILT: Wed Feb 24 07:07:09 CST 2016 cmirror-2.02.143-1.el6 BUILT: Wed Feb 24 07:59:50 CST 2016