Description of problem: New Fedora 12 install using two BTRFS filesystems and dmcrypt on x61 thinkpad with 2gb ram. I'm seeing somewhat frequent page allocation failure. order:2, mode:0x4020 in dmesg. I'm connected to an 802.11a network using wpa-psk. I can't find a particular trigger condition but they are happening during heavy network and disk activity when the system has plenty of memory free. The system seems to be working okay in any case. I am unsure of how best to proceed with memory fragmentation troubleshooting. [root@helmholtz tmp]# lsmod Module Size Used by fuse 62048 2 sunrpc 191864 1 ip6t_REJECT 5856 2 nf_conntrack_ipv6 21880 2 ip6table_filter 4016 1 ip6_tables 19664 1 ip6table_filter ipv6 298864 30 ip6t_REJECT,nf_conntrack_ipv6 cpufreq_ondemand 7824 2 acpi_cpufreq 10528 0 freq_table 4864 2 cpufreq_ondemand,acpi_cpufreq cryptd 7640 0 aes_x86_64 8800 6 aes_generic 27776 1 aes_x86_64 lrw 3936 3 gf128mul 8000 1 lrw dm_crypt 13528 3 dm_multipath 17304 0 uinput 9248 0 snd_hda_codec_analog 75872 1 arc4 2160 2 ecb 3264 2 snd_hda_intel 30344 3 snd_hda_codec 72832 2 snd_hda_codec_analog,snd_hda_intel iwlagn 144576 0 snd_hwdep 9224 1 snd_hda_codec snd_seq 58080 0 snd_seq_device 7620 1 snd_seq sierra 22072 0 usbserial 37984 1 sierra iwlcore 171748 1 iwlagn sdhci_pci 9008 0 sdhci 20676 1 sdhci_pci firewire_ohci 23348 0 snd_pcm 83144 2 snd_hda_intel,snd_hda_codec firewire_core 47880 1 firewire_ohci mmc_core 58832 1 sdhci thinkpad_acpi 69276 0 mac80211 181512 2 iwlagn,iwlcore crc_itu_t 2080 1 firewire_core snd_timer 22608 2 snd_seq,snd_pcm cfg80211 87800 3 iwlagn,iwlcore,mac80211 snd 67592 14 snd_hda_codec_analog,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer rfkill 20488 2 thinkpad_acpi,cfg80211 iTCO_wdt 13008 0 iTCO_vendor_support 3588 1 iTCO_wdt soundcore 7328 1 snd e1000e 124624 0 snd_page_alloc 9568 2 snd_hda_intel,snd_pcm i2c_i801 12800 0 btrfs 402912 2 zlib_deflate 19384 1 btrfs libcrc32c 1976 1 btrfs yenta_socket 28412 1 rsrc_nonstatic 10432 1 yenta_socket i915 219952 2 drm_kms_helper 25360 1 i915 drm 171168 3 i915,drm_kms_helper i2c_algo_bit 6020 1 i915 i2c_core 28608 4 i2c_i801,i915,drm,i2c_algo_bit video 23476 1 i915 output 3360 1 video yum: page allocation failure. order:2, mode:0x4020 Pid: 4947, comm: yum Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c176d>] iwl_rx_handle+0x38e/0x3a7 [iwlagn] [<ffffffff813a842d>] ? ip_rcv+0x289/0x2c9 [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffff8106e897>] ? clocksource_read+0xf/0x11 [<ffffffff8106eb1f>] ? getnstimeofday+0x5b/0xaf [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa025a875>] ? iwl_isr_legacy+0x1c7/0x1fe [iwlcore] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffffa00d6dad>] ? btrfs_drop_pages+0x2b/0x56 [btrfs] [<ffffffffa00d6dcb>] ? btrfs_drop_pages+0x49/0x56 [btrfs] [<ffffffffa00d8a85>] ? btrfs_file_write+0x49c/0x6e7 [btrfs] [<ffffffff811c1f3c>] ? selinux_file_permission+0x58/0x5d [<ffffffff810fc922>] ? vfs_write+0xae/0x10b [<ffffffff810fca3f>] ? sys_write+0x4a/0x6e [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 64 CPU 1: hi: 186, btch: 31 usd: 1 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491649 unevictable:5 dirty:28088 writeback:0 unstable:0 free:23241 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:80984kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1964084kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19918*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81064kB 632606 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751473 pages shared 110358 pages non-shared iwlagn 0000:03:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining. yum: page allocation failure. order:2, mode:0x4020 Pid: 4947, comm: yum Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c1748>] iwl_rx_handle+0x369/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa01afec0>] ? ieee80211_tasklet_handler+0xe9/0x104 [mac80211] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffffa00d6dad>] ? btrfs_drop_pages+0x2b/0x56 [btrfs] [<ffffffffa00d6dcb>] ? btrfs_drop_pages+0x49/0x56 [btrfs] [<ffffffffa00d8a85>] ? btrfs_file_write+0x49c/0x6e7 [btrfs] [<ffffffff811c1f3c>] ? selinux_file_permission+0x58/0x5d [<ffffffff810fc922>] ? vfs_write+0xae/0x10b [<ffffffff810fca3f>] ? sys_write+0x4a/0x6e [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 29 CPU 1: hi: 186, btch: 31 usd: 0 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491515 unevictable:5 dirty:28125 writeback:0 unstable:0 free:23267 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:81088kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1963548kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19964*4kB 4*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81088kB 632458 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751477 pages shared 110522 pages non-shared iwlagn 0000:03:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining. yum: page allocation failure. order:2, mode:0x4020 Pid: 4947, comm: yum Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c1748>] iwl_rx_handle+0x369/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa01afec0>] ? ieee80211_tasklet_handler+0xe9/0x104 [mac80211] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffffa00d6dad>] ? btrfs_drop_pages+0x2b/0x56 [btrfs] [<ffffffffa00d6dcb>] ? btrfs_drop_pages+0x49/0x56 [btrfs] [<ffffffffa00d8a85>] ? btrfs_file_write+0x49c/0x6e7 [btrfs] [<ffffffff811c1f3c>] ? selinux_file_permission+0x58/0x5d [<ffffffff810fc922>] ? vfs_write+0xae/0x10b [<ffffffff810fca3f>] ? sys_write+0x4a/0x6e [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 29 CPU 1: hi: 186, btch: 31 usd: 0 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491515 unevictable:5 dirty:28125 writeback:0 unstable:0 free:23267 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:81088kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1963548kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19964*4kB 4*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81088kB 632458 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751477 pages shared 110521 pages non-shared iwlagn 0000:03:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining. yum: page allocation failure. order:2, mode:0x4020 Pid: 4947, comm: yum Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c1748>] iwl_rx_handle+0x369/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa01afec0>] ? ieee80211_tasklet_handler+0xe9/0x104 [mac80211] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffffa00d6dad>] ? btrfs_drop_pages+0x2b/0x56 [btrfs] [<ffffffffa00d6dcb>] ? btrfs_drop_pages+0x49/0x56 [btrfs] [<ffffffffa00d8a85>] ? btrfs_file_write+0x49c/0x6e7 [btrfs] [<ffffffff811c1f3c>] ? selinux_file_permission+0x58/0x5d [<ffffffff810fc922>] ? vfs_write+0xae/0x10b [<ffffffff810fca3f>] ? sys_write+0x4a/0x6e [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 30 CPU 1: hi: 186, btch: 31 usd: 0 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491515 unevictable:5 dirty:28125 writeback:0 unstable:0 free:23267 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:81088kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1963548kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19964*4kB 4*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81088kB 632458 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751477 pages shared 110521 pages non-shared iwlagn 0000:03:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining. yum: page allocation failure. order:2, mode:0x4020 Pid: 4947, comm: yum Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c1748>] iwl_rx_handle+0x369/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa01afec0>] ? ieee80211_tasklet_handler+0xe9/0x104 [mac80211] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffffa00d6dad>] ? btrfs_drop_pages+0x2b/0x56 [btrfs] [<ffffffffa00d6dcb>] ? btrfs_drop_pages+0x49/0x56 [btrfs] [<ffffffffa00d8a85>] ? btrfs_file_write+0x49c/0x6e7 [btrfs] [<ffffffff811c1f3c>] ? selinux_file_permission+0x58/0x5d [<ffffffff810fc922>] ? vfs_write+0xae/0x10b [<ffffffff810fca3f>] ? sys_write+0x4a/0x6e [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 30 CPU 1: hi: 186, btch: 31 usd: 0 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491515 unevictable:5 dirty:28125 writeback:0 unstable:0 free:23267 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:81088kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1963548kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19964*4kB 4*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81088kB 632458 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751477 pages shared 110521 pages non-shared yum: page allocation failure. order:2, mode:0x4020 Pid: 4947, comm: yum Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c1748>] iwl_rx_handle+0x369/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa01afec0>] ? ieee80211_tasklet_handler+0xe9/0x104 [mac80211] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffffa00d6dad>] ? btrfs_drop_pages+0x2b/0x56 [btrfs] [<ffffffffa00d6dcb>] ? btrfs_drop_pages+0x49/0x56 [btrfs] [<ffffffffa00d8a85>] ? btrfs_file_write+0x49c/0x6e7 [btrfs] [<ffffffff811c1f3c>] ? selinux_file_permission+0x58/0x5d [<ffffffff810fc922>] ? vfs_write+0xae/0x10b [<ffffffff810fca3f>] ? sys_write+0x4a/0x6e [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 30 CPU 1: hi: 186, btch: 31 usd: 0 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491515 unevictable:5 dirty:28125 writeback:0 unstable:0 free:23267 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:81088kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1963548kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19964*4kB 4*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81088kB 632458 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751477 pages shared 110521 pages non-shared yum: page allocation failure. order:2, mode:0x4020 Pid: 4947, comm: yum Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c176d>] iwl_rx_handle+0x38e/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa01afec0>] ? ieee80211_tasklet_handler+0xe9/0x104 [mac80211] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffffa00d6dad>] ? btrfs_drop_pages+0x2b/0x56 [btrfs] [<ffffffffa00d6dcb>] ? btrfs_drop_pages+0x49/0x56 [btrfs] [<ffffffffa00d8a85>] ? btrfs_file_write+0x49c/0x6e7 [btrfs] [<ffffffff811c1f3c>] ? selinux_file_permission+0x58/0x5d [<ffffffff810fc922>] ? vfs_write+0xae/0x10b [<ffffffff810fca3f>] ? sys_write+0x4a/0x6e [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 30 CPU 1: hi: 186, btch: 31 usd: 0 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491515 unevictable:5 dirty:28125 writeback:0 unstable:0 free:23267 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:81088kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1963548kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19964*4kB 4*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81088kB 632458 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751477 pages shared 110521 pages non-shared swapper: page allocation failure. order:2, mode:0x4020 Pid: 0, comm: swapper Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff812b737d>] ? scsi_done+0x0/0x17 [<ffffffff812b737d>] ? scsi_done+0x0/0x17 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c1748>] iwl_rx_handle+0x369/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffff8141eab2>] ? smp_apic_timer_interrupt+0x86/0x94 [<ffffffff81012873>] ? apic_timer_interrupt+0x13/0x20 [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffff8126796a>] ? acpi_idle_enter_simple+0x111/0x145 [<ffffffff81267963>] ? acpi_idle_enter_simple+0x10a/0x145 [<ffffffff813538a3>] ? cpuidle_idle_call+0x8d/0xc2 [<ffffffff81010c60>] ? cpu_idle+0xa6/0xe9 [<ffffffff81405ad7>] ? rest_init+0x6b/0x6d [<ffffffff81714dc9>] ? start_kernel+0x3ef/0x3fa [<ffffffff817142a1>] ? x86_64_start_reservations+0xac/0xb0 [<ffffffff8171439d>] ? x86_64_start_kernel+0xf8/0x107 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 30 CPU 1: hi: 186, btch: 31 usd: 0 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491515 unevictable:5 dirty:28125 writeback:0 unstable:0 free:23267 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:81088kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1963548kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19964*4kB 4*8kB 2*16kB 1*32kB 0*64kB 1*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81104kB 632458 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751477 pages shared 110517 pages non-shared swapper: page allocation failure. order:2, mode:0x4020 Pid: 0, comm: swapper Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff812b737d>] ? scsi_done+0x0/0x17 [<ffffffff812b737d>] ? scsi_done+0x0/0x17 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c1748>] iwl_rx_handle+0x369/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffff8141eab2>] ? smp_apic_timer_interrupt+0x86/0x94 [<ffffffff81012873>] ? apic_timer_interrupt+0x13/0x20 [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffff8126796a>] ? acpi_idle_enter_simple+0x111/0x145 [<ffffffff81267963>] ? acpi_idle_enter_simple+0x10a/0x145 [<ffffffff813538a3>] ? cpuidle_idle_call+0x8d/0xc2 [<ffffffff81010c60>] ? cpu_idle+0xa6/0xe9 [<ffffffff81405ad7>] ? rest_init+0x6b/0x6d [<ffffffff81714dc9>] ? start_kernel+0x3ef/0x3fa [<ffffffff817142a1>] ? x86_64_start_reservations+0xac/0xb0 [<ffffffff8171439d>] ? x86_64_start_kernel+0xf8/0x107 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 30 CPU 1: hi: 186, btch: 31 usd: 0 Active_anon:42425 active_file:129157 inactive_anon:11709 inactive_file:491515 unevictable:5 dirty:28125 writeback:0 unstable:0 free:23267 slab:37212 mapped:19711 pagetables:4829 bounce:0 Node 0 DMA free:11980kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:2512kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:81088kB min:6976kB low:8720kB high:10464kB active_anon:169700kB inactive_anon:46836kB active_file:516396kB inactive_file:1963548kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 1*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 3*2048kB 0*4096kB = 11980kB Node 0 DMA32: 19964*4kB 4*8kB 2*16kB 1*32kB 0*64kB 1*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 81104kB 632458 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 2097144kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 751477 pages shared 110517 pages non-shared CE: hpet increasing min_delta_ns to 15000 nsec __ratelimit: 48 callbacks suppressed __ratelimit: 19 callbacks suppressed gdm-user-switch: page allocation failure. order:2, mode:0x4020 Pid: 3466, comm: gdm-user-switch Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c176d>] iwl_rx_handle+0x38e/0x3a7 [iwlagn] [<ffffffff812bd599>] ? scsi_next_command+0x43/0x4b [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffff8106e897>] ? clocksource_read+0xf/0x11 [<ffffffff8106eb1f>] ? getnstimeofday+0x5b/0xaf [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa025a875>] ? iwl_isr_legacy+0x1c7/0x1fe [iwlcore] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> [<ffffffffa00e78db>] ? extent_readpages+0xb2/0x164 [btrfs] [<ffffffffa00e78bc>] ? extent_readpages+0x93/0x164 [btrfs] [<ffffffffa00d07de>] ? btrfs_get_extent+0x0/0x75b [btrfs] [<ffffffffa00cfef2>] ? btrfs_readpages+0x1f/0x21 [btrfs] [<ffffffff810c92a0>] ? __do_page_cache_readahead+0x103/0x176 [<ffffffff810c9334>] ? ra_submit+0x21/0x25 [<ffffffff810c1abd>] ? filemap_fault+0x193/0x317 [<ffffffff810d6641>] ? __do_fault+0x54/0x3c4 [<ffffffff810d881a>] ? handle_mm_fault+0x2f6/0x705 [<ffffffff811118f5>] ? mntput_no_expire+0x29/0xec [<ffffffff810ffc54>] ? cp_new_stat+0xe3/0xf0 [<ffffffff8141cfe1>] ? do_page_fault+0x281/0x299 [<ffffffff8141af75>] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 48 CPU 1: hi: 186, btch: 31 usd: 52 Active_anon:87450 active_file:262189 inactive_anon:23290 inactive_file:263065 unevictable:5 dirty:24939 writeback:0 unstable:0 free:63330 slab:32939 mapped:19010 pagetables:5009 bounce:0 Node 0 DMA free:11956kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:241364kB min:6976kB low:8720kB high:10464kB active_anon:349800kB inactive_anon:93160kB active_file:1048756kB inactive_file:1052260kB unevictable:20kB present:3060992kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 3*4kB 5*8kB 3*16kB 3*32kB 6*64kB 5*128kB 4*256kB 3*512kB 2*1024kB 3*2048kB 0*4096kB = 11972kB Node 0 DMA32: 36791*4kB 11697*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 241364kB 543285 total pagecache pages 82 pages in swap cache Swap cache stats: add 1209, delete 1127, find 3/10 Free swap = 2092536kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 665307 pages shared 159838 pages non-shared iwlagn 0000:03:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining. firefox: page allocation failure. order:2, mode:0x4020 Pid: 17515, comm: firefox Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: <IRQ> [<ffffffff810c7672>] __alloc_pages_nodemask+0x542/0x584 [<ffffffff810f2532>] alloc_pages_node+0x48/0x4a [<ffffffff810f255e>] kmalloc_large_node+0x2a/0x67 [<ffffffff810f37ee>] __kmalloc_node_track_caller+0x31/0x123 [<ffffffffa025f1c1>] ? iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffff81374b9c>] __alloc_skb+0x80/0x170 [<ffffffffa025f1c1>] iwl_rx_allocate+0xa9/0x2b2 [iwlcore] [<ffffffffa0260784>] iwl_rx_replenish_now+0x1b/0x27 [iwlcore] [<ffffffffa02c176d>] iwl_rx_handle+0x38e/0x3a7 [iwlagn] [<ffffffffa02c24fb>] ? __iwl_write32.clone.2+0xad/0xbc [iwlagn] [<ffffffff8132941d>] ? psmouse_process_byte+0x25c/0x26a [<ffffffff8106e897>] ? clocksource_read+0xf/0x11 [<ffffffff8106eb1f>] ? getnstimeofday+0x5b/0xaf [<ffffffffa02c383e>] iwl_irq_tasklet_legacy+0x4e6/0x721 [iwlagn] [<ffffffffa025a875>] ? iwl_isr_legacy+0x1c7/0x1fe [iwlcore] [<ffffffff81056097>] tasklet_action+0x85/0xe4 [<ffffffff810575b4>] __do_softirq+0xdd/0x1ad [<ffffffff81026976>] ? apic_write+0x16/0x18 [<ffffffff81012eac>] call_softirq+0x1c/0x30 [<ffffffff810143fb>] do_softirq+0x47/0x8d [<ffffffff810572c6>] irq_exit+0x44/0x86 [<ffffffff8141ea15>] do_IRQ+0xa5/0xbc [<ffffffff810126d3>] ret_from_intr+0x0/0x11 <EOI> Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 32 CPU 1: hi: 186, btch: 31 usd: 15 Active_anon:92922 active_file:161034 inactive_anon:22778 inactive_file:143340 unevictable:5 dirty:2819 writeback:0 unstable:0 free:117849 slab:197187 mapped:18883 pagetables:5075 bounce:0 Node 0 DMA free:11960kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:0kB unevictable:0kB present:15348kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 2989 2989 2989 Node 0 DMA32 free:459436kB min:6976kB low:8720kB high:10464kB active_anon:371688kB inactive_anon:91240kB active_file:644116kB inactive_file:573252kB unevictable:20kB present:3060992kB pages_scanned:197 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 4*4kB 9*8kB 9*16kB 7*32kB 8*64kB 4*128kB 5*256kB 2*512kB 2*1024kB 3*2048kB 0*4096kB = 11976kB Node 0 DMA32: 114551*4kB 20*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 459468kB 327427 total pagecache pages 1991 pages in swap cache Swap cache stats: add 20651, delete 18660, find 3191/4296 Free swap = 2081300kB Total swap = 2097144kB 779952 pages RAM 13130 pages reserved 508020 pages shared 324222 pages non-shared iwlagn 0000:03:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining. The below were captured now, not at the time of any of the failure events: [root@helmholtz ~]# free total used free shared buffers cached Mem: 3067292 1963152 1104140 0 40 995196 -/+ buffers/cache: 967916 2099376 Swap: 2097144 55416 2041728 [root@helmholtz ~]# cat /proc/slabinfo slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> fuse_request 50 50 632 25 4 : tunables 0 0 0 : slabdata 2 2 0 fuse_inode 42 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 rpc_inode_cache 19 19 832 19 4 : tunables 0 0 0 : slabdata 1 1 0 ip6_dst_cache 35 36 320 12 1 : tunables 0 0 0 : slabdata 3 3 0 UDPLITEv6 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 32 32 1024 16 4 : tunables 0 0 0 : slabdata 2 2 0 tw_sock_TCPv6 0 0 320 12 1 : tunables 0 0 0 : slabdata 0 0 0 TCPv6 34 34 1856 17 8 : tunables 0 0 0 : slabdata 2 2 0 dm_crypt_io 100 130 152 26 1 : tunables 0 0 0 : slabdata 5 5 0 extent_state 38536 104704 128 32 1 : tunables 0 0 0 : slabdata 3272 3272 0 btrfs_transaction_cache 34 34 232 17 1 : tunables 0 0 0 : slabdata 2 2 0 btrfs_trans_handle_cache 128 128 64 64 1 : tunables 0 0 0 : slabdata 2 2 0 btrfs_inode_cache 17995 31605 1040 15 4 : tunables 0 0 0 : slabdata 2107 2107 0 iint_cache 24789 46944 128 32 1 : tunables 0 0 0 : slabdata 1467 1467 0 nf_conntrack_expect 0 0 240 17 1 : tunables 0 0 0 : slabdata 0 0 0 nf_conntrack 31 78 312 13 1 : tunables 0 0 0 : slabdata 6 6 0 dm_raid1_read_record 0 0 1064 15 4 : tunables 0 0 0 : slabdata 0 0 0 kcopyd_job 0 0 368 22 2 : tunables 0 0 0 : slabdata 0 0 0 dm_uevent 0 0 2608 12 8 : tunables 0 0 0 : slabdata 0 0 0 dm_rq_target_io 0 0 376 21 2 : tunables 0 0 0 : slabdata 0 0 0 cfq_queue 228 240 168 24 1 : tunables 0 0 0 : slabdata 10 10 0 bsg_cmd 0 0 312 13 1 : tunables 0 0 0 : slabdata 0 0 0 mqueue_inode_cache 18 18 896 18 4 : tunables 0 0 0 : slabdata 1 1 0 isofs_inode_cache 0 0 640 12 2 : tunables 0 0 0 : slabdata 0 0 0 hugetlbfs_inode_cache 13 13 608 13 2 : tunables 0 0 0 : slabdata 1 1 0 journal_handle 340 340 24 170 1 : tunables 0 0 0 : slabdata 2 2 0 journal_head 36 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 revoke_table 256 256 16 256 1 : tunables 0 0 0 : slabdata 1 1 0 revoke_record 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 0 0 888 18 4 : tunables 0 0 0 : slabdata 0 0 0 ext4_free_block_extents 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_alloc_context 43229 105448 144 28 1 : tunables 0 0 0 : slabdata 3766 3766 0 ext4_prealloc_space 0 0 104 39 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_system_zone 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 ext3_inode_cache 40 40 784 20 4 : tunables 0 0 0 : slabdata 2 2 0 ext3_xattr 24257 36754 88 46 1 : tunables 0 0 0 : slabdata 799 799 0 dquot 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_mm_slot 146 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 shmem_inode_cache 2767 3150 800 20 4 : tunables 0 0 0 : slabdata 159 159 0 pid_namespace 0 0 2112 15 8 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 832 19 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 534 556 768 21 4 : tunables 0 0 0 : slabdata 28 28 0 UDP 38 38 832 19 4 : tunables 0 0 0 : slabdata 2 2 0 tw_sock_TCP 32 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 TCP 44 57 1664 19 8 : tunables 0 0 0 : slabdata 3 3 0 blkdev_queue 45 45 2072 15 8 : tunables 0 0 0 : slabdata 3 3 0 blkdev_requests 36 48 336 12 1 : tunables 0 0 0 : slabdata 4 4 0 fsnotify_event_holder 334477 334900 24 170 1 : tunables 0 0 0 : slabdata 1970 1970 0 fsnotify_event 78 78 104 39 1 : tunables 0 0 0 : slabdata 2 2 0 bip-256 105 105 4224 7 8 : tunables 0 0 0 : slabdata 15 15 0 bip-128 0 0 2176 15 8 : tunables 0 0 0 : slabdata 0 0 0 bip-64 0 0 1152 14 4 : tunables 0 0 0 : slabdata 0 0 0 bip-16 67 126 384 21 2 : tunables 0 0 0 : slabdata 6 6 0 sock_inode_cache 623 644 704 23 4 : tunables 0 0 0 : slabdata 28 28 0 file_lock_cache 44 44 184 22 1 : tunables 0 0 0 : slabdata 2 2 0 net_namespace 0 0 1984 16 8 : tunables 0 0 0 : slabdata 0 0 0 task_delay_info 538 612 112 36 1 : tunables 0 0 0 : slabdata 17 17 0 taskstats 24 24 328 12 1 : tunables 0 0 0 : slabdata 2 2 0 proc_inode_cache 1255 1272 640 12 2 : tunables 0 0 0 : slabdata 106 106 0 sigqueue 50 50 160 25 1 : tunables 0 0 0 : slabdata 2 2 0 radix_tree_node 47813 139573 560 14 2 : tunables 0 0 0 : slabdata 9970 9970 0 bdev_cache 54 54 896 18 4 : tunables 0 0 0 : slabdata 3 3 0 sysfs_dir_cache 12351 12393 80 51 1 : tunables 0 0 0 : slabdata 243 243 0 inode_cache 2845 2847 592 13 2 : tunables 0 0 0 : slabdata 219 219 0 dentry 20247 26355 192 21 1 : tunables 0 0 0 : slabdata 1255 1255 0 selinux_inode_security 28643 56504 72 56 1 : tunables 0 0 0 : slabdata 1009 1009 0 buffer_head 93 360 112 36 1 : tunables 0 0 0 : slabdata 10 10 0 vm_area_struct 17620 18032 176 23 1 : tunables 0 0 0 : slabdata 784 784 0 mm_struct 122 126 896 18 4 : tunables 0 0 0 : slabdata 7 7 0 files_cache 126 138 704 23 4 : tunables 0 0 0 : slabdata 6 6 0 signal_cache 222 238 960 17 4 : tunables 0 0 0 : slabdata 14 14 0 sighand_cache 218 225 2112 15 8 : tunables 0 0 0 : slabdata 15 15 0 task_struct 315 320 6016 5 8 : tunables 0 0 0 : slabdata 64 64 0 anon_vma 5512 5632 32 128 1 : tunables 0 0 0 : slabdata 44 44 0 shared_policy_node 170 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0 numa_policy 30 30 136 30 1 : tunables 0 0 0 : slabdata 1 1 0 idr_layer_cache 695 705 544 15 2 : tunables 0 0 0 : slabdata 47 47 0 kmalloc-8192 101 104 8192 4 8 : tunables 0 0 0 : slabdata 26 26 0 kmalloc-4096 632 664 4096 8 8 : tunables 0 0 0 : slabdata 83 83 0 kmalloc-2048 575 704 2048 16 8 : tunables 0 0 0 : slabdata 44 44 0 kmalloc-1024 599 624 1024 16 4 : tunables 0 0 0 : slabdata 39 39 0 kmalloc-512 4639 4704 512 16 2 : tunables 0 0 0 : slabdata 294 294 0 kmalloc-256 1545 1616 256 16 1 : tunables 0 0 0 : slabdata 101 101 0 kmalloc-128 1221 2816 128 32 1 : tunables 0 0 0 : slabdata 88 88 0 kmalloc-64 9095 10624 64 64 1 : tunables 0 0 0 : slabdata 166 166 0 kmalloc-32 425440 425600 32 128 1 : tunables 0 0 0 : slabdata 3325 3325 0 kmalloc-16 205210 205824 16 256 1 : tunables 0 0 0 : slabdata 804 804 0 kmalloc-8 8714 9728 8 512 1 : tunables 0 0 0 : slabdata 19 19 0 kmalloc-192 13884 14784 192 21 1 : tunables 0 0 0 : slabdata 704 704 0 kmalloc-96 7046 9576 96 42 1 : tunables 0 0 0 : slabdata 228 228 0 kmem_cache_node 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 [root@helmholtz ~]# cat /proc/vmstat nr_free_pages 273925 nr_inactive_anon 41517 nr_active_anon 159542 nr_inactive_file 100336 nr_active_file 104892 nr_unevictable 5 nr_mlock 5 nr_anon_pages 153266 nr_mapped 15819 nr_file_pages 253719 nr_dirty 18 nr_writeback 0 nr_slab_reclaimable 38263 nr_slab_unreclaimable 14784 nr_page_table_pages 6108 nr_unstable 0 nr_bounce 0 nr_vmscan_write 23697 nr_writeback_temp 0 numa_hit 125926407 numa_miss 0 numa_foreign 0 numa_interleave 13537 numa_local 125926407 numa_other 0 pgpgin 13212460 pgpgout 23432227 pswpin 11417 pswpout 23061 pgalloc_dma 5124507 pgalloc_dma32 169704394 pgalloc_normal 0 pgalloc_movable 0 pgfree 175102986 pgactivate 3543571 pgdeactivate 2245648 pgfault 69046738 pgmajfault 9127 pgrefill_dma 403 pgrefill_dma32 2331179 pgrefill_normal 0 pgrefill_movable 0 pgsteal_dma 0 pgsteal_dma32 7041913 pgsteal_normal 0 pgsteal_movable 0 pgscan_kswapd_dma 0 pgscan_kswapd_dma32 6736514 pgscan_kswapd_normal 0 pgscan_kswapd_movable 0 pgscan_direct_dma 0 pgscan_direct_dma32 486467 pgscan_direct_normal 0 pgscan_direct_movable 0 zone_reclaim_failed 0 pginodesteal 53966 slabs_scanned 3919744 kswapd_steal 6566562 kswapd_inodesteal 1282948 pageoutrun 77801 allocstall 4956 pgrotated 22994 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 101529 unevictable_pgs_scanned 0 unevictable_pgs_rescued 104600 unevictable_pgs_mlocked 104605 unevictable_pgs_munlocked 104595 unevictable_pgs_cleared 5 unevictable_pgs_stranded 0 unevictable_pgs_mlockfreed 0 [root@helmholtz tmp]# lspci 00:00.0 Host bridge: Intel Corporation Mobile PM965/GM965/GL960 Memory Controller Hub (rev 0c) 00:02.0 VGA compatible controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c) 00:02.1 Display controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c) 00:19.0 Ethernet controller: Intel Corporation 82566MM Gigabit Network Connection (rev 03) 00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 03) 00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 03) 00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 03) 00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 03) 00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 03) 00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 03) 00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 03) 00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 03) 00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 03) 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f3) 00:1f.0 ISA bridge: Intel Corporation 82801HBM (ICH8M-E) LPC Interface Controller (rev 03) 00:1f.1 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) IDE Controller (rev 03) 00:1f.2 SATA controller: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) SATA AHCI Controller (rev 03) 00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 03) 03:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN [Kedron] Network Connection (rev 61) 05:00.0 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev ba) 05:00.1 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 04) 05:00.2 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 21)
Ah, I should have mentioned that I have /tmp on tmpfs: tmpfs 1.5G 119M 1.4G 8% /tmp So I don't think the swap in use shouldn't have been indicative of any prior exhaustion. (nor have I seen any userspace evidence of memory exaustion)
Hi Gregory The bug is not fixed upstream, fixing it in not so easy. We have upstream bug entry for it here: http://bugzilla.kernel.org/show_bug.cgi?id=14141 Are you able to reproduce? There are some upstream patches to test, could you test 2.6.33-rcX vanilla and help solving this bug on bugzilla.kernel.org ?
Gregory, please note we have now koji build for 2.6.33 based kernels (see http://koji.fedoraproject.org/koji/packageinfo?packageID=8) If issue is not present in 2.6.33 and you can reproduce it in 2.6.32, I think we have to backport iwlwifi "rx paged" patch, fixing that memory management can be _very_ hard.
I am seeing something very similar tho I do not use BTRFS ... just iwlagn. https://bugzilla.redhat.com/show_bug.cgi?id=569412 Is that a dup of this one ?
Hi Gregory Please tell us if you are able to reproduce that bug. If so, I will prepare kernel for test.
I'd just like to add a "me too". I'm running F12 x86_64 in a default install (no BTRFS or anything) on a Thinkpad T61p with 3 GB ram. Yesterday I got page allocation failures like the ones shown in the attached log file - I was pushing the memory limits and beginning to use some swap when the network connection started dropping out. I could apparently reproduce the problem by making sure that running processes took up a lot of memory and then doing something that causes network traffic like checking out a source tree from SVN.
Created attachment 399277 [details] Snippet from /var/log/messages
Could you please test kernel from. http://koji.fedoraproject.org/koji/taskinfo?taskID=2045788 Would be nice if you can assure that bug is reproducible fedora released kernel and not reproducible on kernel from above link.
Bug should be fixed in F-13.