Bug 570491

Created attachment 397820 [details]
reproducer

Client is stress testing iptables for their requirement on smp system. They have provided program(attached) which adds thousands of rules in iptables. In rhel5.x, once limit goes to ~50000, program exits on getting error code with error "iptables: Unknown error 18446744073709551615". iptables version is iptables-1.3.5-5.3.el5_4.1

How reproducible:
Always

Steps to Reproduce:
I'll attach the reproducer. 

run the program
#./a.out

Actual result:
iptables: Unknown error 18446744073709551615 

This is really an ENOMEM that doesn't get propagated to iptables. The iptables code sets errno to ret val (-1) of setsockopt. There is a simple fix for this and is not the focus of this bug report. 

setsockopt(3, SOL_IP, 0x40 /* IP_??? */, "filter\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 6127868) = -1 ENOMEM (Cannot allocate memory)
munmap(0xb728c000, 512000)              = 0
munmap(0xb7309000, 512000)              = 0
munmap(0xb7386000, 6131712)             = 0
write(2, "iptables: Unknown error 42949672"..., 35iptables: Unknown error 4294967295
) = 35
exit_group(1)


Expected result:
should run successfully

Should not be an ENOMEM as was fixed in rhel4.7


Additional info:
Once this limit is reached, one can not insert even single rule. These rules are persistent in iptables service restart. They are destroyed when you remove all iptables kernel modules(mostly x_tables). 'VmallocUsed' field changes drastically after the program. Difference is more than 125MB. When the program is straced, setsockopt() returns -ENOMEM in the end and program stops.  On rhel 4, It gave error "iptables: Memory allocation problem". After upgrading to rhel4.7 , the issue is resolved. 


Appears to be a duplicate of a rhel4 bug

https://bugzilla.redhat.com/show_bug.cgi?id=179098

that was a dup of 

https://bugzilla.redhat.com/show_bug.cgi?id=173193

it looks like the rhel4 patch was applied to rhel5 kernel, but then some other updates have been applied since.