Red Hat Bugzilla – Bug 46665
Slowly rising CPU load when using REDIRECT with ipchains and iptables
Last modified: 2007-04-18 12:34:19 EDT
From Bugzilla Helper: User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Description of problem: I configured a RedHat 7.1 server as a transparent proxy server, using Squid and ipchains or iptables. Some time after the installation, I noticed that the server vas very slow when I worked with the console. So I checked the CPU load and found that system was using 95-99% of the CPU time on the Intel PIII 866 CPU. The server was rebooted, and the same thing happened again after 2-3 hours. To see if it was a bug in the 2.4.2-2 kernal in the original RH7.1, I tried upgrading to 2.4.3-12 from updates.redhat.com. After 2-3 hours, system was using 95-99% CPU time again. Then we tried downloading and compiling 2.4.5-ac21 with the same results. before the next reboot, we removed the IPChains rule to redirect calls to port 80 to 8080 (-j REDIRECT 8080) and manually configured the clients to use the proxy server. After this reboot the server has been up for 3hours and system is still at about 0% CPU usage. How reproducible: Always Steps to Reproduce: 1. Configure Squid to function as a transparent proxy (http://www.squid- cache.org/Doc/FAQ/FAQ-17.html) 2. add the rule: ipchains -A input -p TCP -s x.x.x.x --dport 80 -j REDIRECT 8080 3. use vstat 2 and see the system CPU usage slowly rise to 99%. Actual Results: The server became too slow to work on and the internet transfers through the server became slower. Additional info: System CPU load went down to about 60-70% when disconnecting the server from the network.
We have encoutered similar problems when upgrading from RH 6.2 to 7.1 on a single CPU AMD K6-2 450Mhz. 'top' shows an unexplainable high 'system' CPU usage (30-50%). The integer index of the NBench benchmark from http://www.tux.org/~mayer/linux/bmark.html gives about 0.5 on 7.1 about 3 time faster on 6.2 on the same machine. There is no disk activity, swap, or network activity during the benchmark. We had ipchained REDIRECT but taking it off does not seems to change much. Kernel update to 'Kernel 2.4.3-12 on an i586' did not help either. Looks like we need to downgrade our production system back to 6.2 until 7.X will become usable.
Our problem is solved. It was due to a defective AMD K6-2 450 that had 25% of the normal performance (using nbench benchmark program) even though it reported the normal bogus mips. We have several of these CPU's (purchased about 18 months ago) and replacing them with a new ones solved the problem. There are several news discussions about this issue (try to search www.deja.com for 'amd cpu slow p5a'). Redhat 7.1 is off the hook ;-) Tal
Rusty Russel from the Netfilter project has confirmed the bug. I quote from the reply we got from him: "Just looked through the code, and sure enough, there is a leak in the redirect compatibility code, as I suspected from Lutz's earlier report. Please try the attached patch, and thanks! Rusty. -- Premature optmztion is rt of all evl. --DK diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.7-official/net/ipv4/netfilter/ip_fw_compat_redir.c working-2.4.7-module/net/ipv4/netfilter/ip_fw_compat_redir.c --- linux-2.4.7-official/net/ipv4/netfilter/ip_fw_compat_redir.c Sat Aug 5 06:07:24 2000 +++ working-2.4.7-module/net/ipv4/netfilter/ip_fw_compat_redir.c Wed Aug 15 20:45:02 2001 @@ -206,6 +206,8 @@ } list_prepend(&redirs, redir); init_timer(&redir->destroyme); + redir->destroyme.expires = jiffies + 75*HZ; + add_timer(&redir->destroyme); } /* In case mangling has changed, rewrite this part. */ redir->core = ((struct redir_core) " We have not been able to test the patch, since we don't have any hardware to test the patch with so I cannot confirm if this patch fixed the problem.