Bug 860933 - alloc_page is failing to allocate memory due to memoryless node
alloc_page is failing to allocate memory due to memoryless node
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.3
i686 Linux
unspecified Severity high
: rc
: ---
Assigned To: Larry Woodman
Red Hat Kernel QE team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-27 02:36 EDT by Muppana Prasad
Modified: 2014-02-26 12:29 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-26 12:29:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
The attachment has the code for reproducing the problem. (10.00 KB, application/x-tar)
2012-09-27 02:36 EDT, Muppana Prasad
no flags Details

  None (edit)
Description Muppana Prasad 2012-09-27 02:36:16 EDT
Created attachment 617906 [details]
The attachment has the code for reproducing the problem.

Description of problem:
alloc_page is failed to allocate a page with the flags "GFP_KERNEL | __GFP_NORETRY | __GFP_HIGHMEM". Compiled the module on RHEL 6 and loading the module on RHEL 6.3. And in the init_module, the module has a call to alloc_page.

Version-Release number of selected component (if applicable):
2.6.32-279.el6.i686

How reproducible:
Always

Steps to Reproduce:
1. Write a module and in the init_module function, call alloc_page with one of the follwoing flags.
   a) GFP_KERNEL | __GFP_NORETRY | __GFP_HIGHMEM
   b) GFP_KERNEL | __GFP_HIGHMEM
   c) GFP_KERNEL
2. Compile the module on RHEL 6
3. Load the module on RHEL 6.3
  
Actual results:
alloc_page is failing to allocate the page.

Expected results:
alloc_page should allocate page as the RHEL supports KABI compatibility.

Additional info:
Attached a simple module code.
Comment 2 Larry Woodman 2012-09-28 09:41:08 EDT
Not sure what the problem is here, this module loads and succeeds on my system which is a 2-node 16BG system but with 8GB on each node.  

1.) Are you saying the  "alloc_page(GFP_KERNEL | __GFP_NORETRY | __GFP_HIGHMEM)" fails when there is no memory on one and you are unlucky enough to run on that node???  

2.) What is /proc/sys/vm/zone_reclaim_mode set to ???  

3.) Finally, can you attach a AltSysrq-M output???


Thanks, Larry Woodman
Comment 3 Muppana Prasad 2012-10-03 06:44:00 EDT
> Not sure what the problem is here, this module loads and succeeds on my
> system which is a 2-node 16BG system but with 8GB on each node.  

As Redhat supports KABI compatibility, we built our driver on RHEL6 and able to load the driver on RHEL6 & even on its updates. It is working fine with 64-bit and 32-bit systems upto RHEL6.2. On RHEL6.3, the 64-bit version of driver compiled on RHEL6 (x86_64) is loading on 64-bit system but the 32-bit version of driver compiled on RHEL6 (i686) not loading on 32-bit system due to alloc_page is failing to allocate the page.

So do you mean the attached module code is compiled on RHEL6 (2.6.32-71.el6.i686) and is loaded successfully on RHEL6.3 (2.6.32-279.el6.i686)?

> 1.) Are you saying the  "alloc_page(GFP_KERNEL | __GFP_NORETRY |
> __GFP_HIGHMEM)" fails when there is no memory on one and you are unlucky
> enough to run on that node???  

This setup has one node "Node 0" only.

> 2.) What is /proc/sys/vm/zone_reclaim_mode set to ???  

This file doesn't exist on my setup (32-bit one). CONFIG_NUMA is not set.

> 3.) Finally, can you attach a AltSysrq-M output???

Oct  3 15:18:48 TST-RHEL6U3-32 kernel: SysRq : Show Memory
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: Mem-Info:
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: DMA per-cpu:
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: Normal per-cpu:
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: CPU    0: hi:  186, btch:  31 usd:  59
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: HighMem per-cpu:
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: CPU    0: hi:   42, btch:   7 usd:  12
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: active_anon:2957 inactive_anon:1 isolated_anon:0
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: active_file:5366 inactive_file:13970 isolated_file:0
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: unevictable:0 dirty:2 writeback:0 unstable:0
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: free:223124 slab_reclaimable:1762 slab_unreclaimable:6839
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: mapped:2653 shmem:38 pagetables:351 bounce:0
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: DMA free:7872kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: lowmem_reserve[]: 0 863 1000 1000
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: Normal free:820728kB min:3724kB low:4652kB high:5584kB active_anon:208kB inactive_anon:0kB active_file:4880kB inactive_file:10056kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:883912kB mlocked:0kB dirty:8kB writeback:0kB mapped:4kB shmem:32kB slab_reclaimable:7048kB slab_unreclaimable:27356kB kernel_stack:736kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: lowmem_reserve[]: 0 0 1094 1094
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: HighMem free:63896kB min:136kB low:280kB high:428kB active_anon:11620kB inactive_anon:4kB active_file:16584kB inactive_file:45824kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:140148kB mlocked:0kB dirty:0kB writeback:0kB mapped:10608kB shmem:120kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1404kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: lowmem_reserve[]: 0 0 0 0
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: DMA: 6*4kB 3*8kB 3*16kB 3*32kB 2*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 2*2048kB 0*4096kB = 7872kB
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: Normal: 0*4kB 33*8kB 83*16kB 60*32kB 39*64kB 27*128kB 15*256kB 11*512kB 9*1024kB 1*2048kB 193*4096kB = 820728kB
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: HighMem: 214*4kB 132*8kB 114*16kB 40*32kB 14*64kB 17*128kB 56*256kB 35*512kB 13*1024kB 5*2048kB 0*4096kB = 63896kB
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: 19374 total pagecache pages
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: 0 pages in swap cache
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: Free swap  = 1780728kB
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: Total swap = 1780728kB
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: 262128 pages RAM
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: 35330 pages HighMem
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: 4492 pages reserved
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: 11906 pages shared
Oct  3 15:18:48 TST-RHEL6U3-32 kernel: 28449 pages non-shared

Thanks,
Prasad
Comment 4 Larry Woodman 2012-10-18 15:30:33 EDT
Not sure whats happening here, I dont get the error I assume you are seeing loading and running the module.  I assume the page_alloc() is failing???



 	page = alloc_page(GFP_KERNEL | __GFP_NORETRY | __GFP_HIGHMEM);
	//page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM);
	//page = alloc_page(GFP_KERNEL);
	if(!page){
		printk("init_module: alloc_page failed\n");
		return -ENOMEM;
	}


Is your system, under a heavy memory load when this happens?


Larry
Comment 5 Muppana Prasad 2013-01-10 03:35:59 EST
(In reply to comment #4)
> Not sure whats happening here, I dont get the error I assume you are seeing
> loading and running the module.  I assume the page_alloc() is failing???

Yes, alloc_page is failing always.

>  	page = alloc_page(GFP_KERNEL | __GFP_NORETRY | __GFP_HIGHMEM);
> 	//page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM);
> 	//page = alloc_page(GFP_KERNEL);
> 	if(!page){
> 		printk("init_module: alloc_page failed\n");
> 		return -ENOMEM;
> 	}
> 
> 
> Is your system, under a heavy memory load when this happens?

No. And is always reproducible. And the same module compiled on RHEL6.3 (32-bit) and is loading without any error. Only when compiled on RHEL6 (32-bit), the module is faailing to load.

Thanks,
Prasad
Comment 6 Muppana Prasad 2013-01-10 03:36:33 EST
(In reply to comment #4)
> Not sure whats happening here, I dont get the error I assume you are seeing
> loading and running the module.  I assume the page_alloc() is failing???

Yes, alloc_page is failing always.

>  	page = alloc_page(GFP_KERNEL | __GFP_NORETRY | __GFP_HIGHMEM);
> 	//page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM);
> 	//page = alloc_page(GFP_KERNEL);
> 	if(!page){
> 		printk("init_module: alloc_page failed\n");
> 		return -ENOMEM;
> 	}
> 
> 
> Is your system, under a heavy memory load when this happens?

No. And is always reproducible. And the same module compiled on RHEL6.3 (32-bit) and is loading without any error. Only when compiled on RHEL6 (32-bit), the module is faailing to load.

Thanks,
Prasad
Comment 7 Muppana Prasad 2013-02-01 02:15:03 EST
Any update please.

Thanks,
Prasad
Comment 8 Larry Woodman 2013-02-01 07:34:05 EST
Prasad, for some reason I the alloc_page() succeeds on the systems I have tested on, both 64 and 32 bit systems.  I suspect I am missing something here since the subject line says "alloc_page is failing to allocate memory due to memoryless node" and I dont have a memoryless node.  Still digging into this.

Larry
Comment 9 Larry Woodman 2013-02-01 07:54:19 EST
Prasad, BTW do you think this is a KABI compatibility issue?  If you recompile the module on the system you are loading it on does the alloc_pages call succeed for you?

Larry
Comment 10 Muppana Prasad 2013-02-04 05:26:06 EST
(In reply to comment #9)
> Prasad, BTW do you think this is a KABI compatibility issue?  If you
> recompile the module on the system you are loading it on does the
> alloc_pages call succeed for you?
> 

Yes, this is a KABI issue with 32-bit. But when compiling for RHEL6U3 (32-bit), able to load the driver. The test system for both the modules is same.

With same test box of RHEL6U3 (32-bit), the driver compiled on RHEL6 (32-bit) is failing to load whereas the driver compiled on RHEL6U3 (32-bit) is loading. And this is happening for me always.
Comment 11 Muppana Prasad 2013-03-11 06:13:53 EDT
The same driver is loading on RHEL6U4 (32-bit) when compiled on RHEL6 (32-bit). So it is a clear indication of KABI compatibility issue.
Comment 12 Muppana Prasad 2013-03-12 03:06:26 EDT
Hi Larry,

Any update on this issue.

~~
Prasad
Comment 13 Larry Woodman 2013-03-12 16:13:31 EDT
Muppana, sorry but I have not been able to reproduce this problem on an internal x86 system.  Does the module even load for you?  It does load for me.  Can you attach the error message you get while loading the module?

Thank You, Larry
Comment 14 Muppana Prasad 2013-04-19 07:53:27 EDT
Following is the output from RHEL6U3 test box.

---------------------------------------8<----------------------------------------
# modinfo alloc_page.ko 
filename:       alloc_page.ko
....
....
vermagic:       2.6.32-71.el6.i686 SMP mod_unload modversions 686 
#
# uname -a
Linux TST-RHEL6U3-32 2.6.32-279.el6.i686 #1 SMP Wed Jun 13 18:23:32 EDT 2012 i686 i686 i386 GNU/Linux
#
# insmod alloc_page.ko 
insmod: error inserting 'alloc_page.ko': -1 Cannot allocate memory
#
---------------------------------------8<----------------------------------------

Following message is logged in /var/log/messages.

> 		printk("init_module: alloc_page failed\n");

Now the same module is loaded successfully for 2.6.32-279.19.1.el6.i686 (the patched kernel over RHEL6U3 base).
Comment 15 Linda Wang 2014-02-26 12:29:11 EST
per Reporter's feedback in comment#14,  
this x86 issue has been fixed with 2.6.32-279.19.1.el6.i686. 

Close as CURRENTRELEASE. Please feel free to reopen it if 
the issue persist.  Many thanks.

Note You need to log in before you can comment on or make changes to this bug.