Bug 179098 - 'out of vmalloc space' caused by iptables on kernel-smp
Summary: 'out of vmalloc space' caused by iptables on kernel-smp
Keywords:
Status: CLOSED DUPLICATE of bug 173193
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel
Version: 4.0
Hardware: i686
OS: Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Larry Woodman
QA Contact: Brian Brock
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2006-01-27 13:54 UTC by dieter
Modified: 2007-11-30 22:07 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2007-07-10 19:28:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Complete strace log of a failing iptables command (5.43 KB, text/plain)
2006-01-27 14:00 UTC, dieter
no flags Details
iptables.save file to reproduce 'out of vmalloc space' problem with iptables-restore (3.64 MB, text/plain)
2006-01-27 14:16 UTC, dieter
no flags Details
iptables.save file to reproduce 'out of vmalloc space' problem with iptables-restore (157.33 KB, application/octet-stream)
2006-01-27 14:19 UTC, dieter
no flags Details

Description dieter 2006-01-27 13:54:57 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/417.9 (KHTML, like Gecko) Safari/417.8

Description of problem:
We are experiencing vmalloc space problems with 2.6.9 SMP kernels when stress testing 
systems by installing large amounts of iptables rules (about 50000). This happens both
on single- and multi-processor systems using the SMP kernel.

The same operations works without problems and as expected on the same systems when 
running the same non-SMP kernel and with every RAM configuration we tested 
(<1GB and >=1GB RAM).

When installing the iptables rules using separate iptables commands, we see the command
aborting with 'iptables: Memory allocation problem' after about 8000 rules were installed.

With 'dmesg' we see the kernel reporting:
allocation failed: out of vmalloc space - use vmalloc=<size> to increase size. 

Running strace while installing an iptables rule reports a setsockopt() call 
causing the error:

setsockopt(3, SOL_IP, 0x40 /* IP_??? */, "filter\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
\0"..., 580388) = -1 ENOMEM (Cannot allocate memory)
munmap(0xb7f61000, 581632)              = 0
brk(0x8e7c000)                          = 0x8e7c000
write(2, "iptables: Memory allocation prob"..., 36iptables: Memory allocation problem
) = 36
exit_group(1)                           = ?
Process 11962 detached  

It seems to be following section of libiptc/libiptc.c in function TC_COMMIT:
...
        if (setsockopt(sockfd, TC_IPPROTO, SO_SET_REPLACE, repl,
                       sizeof(*repl) + (*handle)->entries.size) < 0) {
                free(repl->counters);
                free(repl);
                free(newcounters);
                return 0;
        }           
...

We also tried inserting the iptables rules in bulk using iptables-restore. This was partially
successful on the SMP kernel with 384MB RAM and 512RAM. Inserting initially worked, but
repeating the operation froze the system (384MB) or failed with 'out of vmalloc space' (512MB).
All other RAM settings (<1GB and >=1GB) already failed initially with 'out of vmalloc space'.

Adapting VmallocTotal using vmalloc=<size> is not an option since GRUB prevents 
booting the system as mentioned in bug 164497 and bug 149092.

Details of our tests:
kernel-smp-2.6.9-22.0.2, 256MB RAM --> iptables-restore: line 48080 failed --> system freezes 
sometimes
kernel-smp-2.6.9-22.0.2, 384MB RAM --> OK (1/2 consecutive times), second run: system freezes
kernel-smp-2.6.9-22.0.2, 512MB RAM --> OK (2/3 consecutive times), second run: iptables-restore: 
line 48080 failed, third run works after 'service iptables restart'
kernel-smp-2.6.9-22.0.2, 768MB RAM --> iptables-restore: line 48080 failed
kernel-smp-2.6.9-22.0.2, 896MB RAM --> iptables-restore: line 18025 failed 
kernel-smp-2.6.9-22.0.2, >=1GB RAM--> iptables-restore: line 18025 failed

We were not sure if this is a kernel or iptables problem. But we see the problem occuring only
if we are booting an SMP kernel, therefore we reported this as a kernel problem.


Version-Release number of selected component (if applicable):
kernel-smp-2.6.9-22.EL kernel-smp-2.6.9-22.0.2.EL

How reproducible:
Always

Steps to Reproduce:
1. Boot the system using the reported SMP kernel
2. Run 'iptables-restore < iptables.save' using the file attached to this bug


Actual Results:  We saw the iptables error message and the kernel message 'out of vmalloc space' in 'dmesg'.
  

Expected Results:  The command should have been completed without error and all rules were expected to be installed.

Additional info:

Comment 1 dieter 2006-01-27 14:00:50 UTC
Created attachment 123771 [details]
Complete strace log of a failing iptables command

The attached logfile shows the complete output of strace for a failing iptables
command.

Comment 2 dieter 2006-01-27 14:16:28 UTC
Created attachment 123774 [details]
iptables.save file to reproduce 'out of vmalloc space' problem with iptables-restore

You should be able to reproduce the problem with the attached file by running
$ iptables-restore < iptables.save

Comment 3 dieter 2006-01-27 14:19:01 UTC
Created attachment 123775 [details]
iptables.save file to reproduce 'out of vmalloc space' problem with iptables-restore

You should be able to reproduce the problem with the attached compressed file
by running
$ iptables-restore < iptables.save

Comment 4 Larry Woodman 2006-12-01 19:09:12 UTC
Is this causing a problem in the real world or is it just toi illustrate that
iptables-restore can cause the system to exhaust vmalloc space?  The reason I
ask it that the kernel's virtual address window used for vmalloc() is smaller on
an SMP kernel than it is on a UP kernel.  The reason for this is there are
per-cpu mapping windows allocated out of that 128MB virtual window when running
the SMP kernel but not the UP kernel.  These per-cpu mapping windows are used
for temporary mapping of highmem pages but they do use and therefore decrease
the remaining kernel virtual window size that vmalloc uses.

Larry Woodman


Comment 5 dieter 2006-12-04 10:47:27 UTC
This is actually a real world problem. We have an application which dynamically
inserts/removes lots of iptables rules based on trouble tickets. As a workaround
we had to switch to the UP kernel at the customer site.

This issue seems to be fixed in recent SPM kernels as we are not able to
reproduce the problem again with kernel-smp-2.6.9-42.0.3.EL and the iptables.save
file attached to this bug report.

Comment 6 Larry Woodman 2007-07-10 19:28:45 UTC
This Bug was also fixed by linux-2.6.9-vmalloc.patch

Larry Woodman


*** This bug has been marked as a duplicate of 173193 ***


Note You need to log in before you can comment on or make changes to this bug.