Created attachment 345856 [details] Test libevent server Description of problem: When creating many concurrent connections over a host2host ipsec session, connect fails with ENOBUFS (105). The same two machines have no issue when ipsec is not enabled. A demonstration libevent based server is attached. ab us used to create the concurrent connections. This is a recreation of something we are seeing in the field with ipsec thin clients with many server/X connections. Reproducible on 2.6.28 as well. I suspect a bug in the IPSec SA replay window logic. Both machines are up to date F10 on 5/28/09 Version-Release number of selected component (if applicable): kernel-2.6.27.24-170.2.68.fc10.x86_64 ipsec-tools-0.7.2-1.fc10.x86_64 How reproducible: 100% Steps to Reproduce: 1. Raise nofiles to 10000 in /etc/security/limits.conf 2. Establish IPSec session between two machines 3. Run attached simple web server on one machine 4. Open port 8080/disable iptables 5. Run ab on the other machine Actual results: [joe@f10-2 ~]$ ab -c 2500 -n5000 http://f10:8080/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking f10 (be patient) Test aborted after 10 failures apr_socket_connect(): No buffer space available (105) Expected results: Before starting IPSec session [joe@f10-2 ~]$ ab -c 2500 -n5000 http://f10:8080/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking f10 (be patient) Completed 500 requests Completed 1000 requests Completed 1500 requests Completed 2000 requests Completed 2500 requests Completed 3000 requests Completed 3500 requests Completed 4000 requests Completed 4500 requests Completed 5000 requests Finished 5000 requests Server Software: Server Hostname: f10 Server Port: 8080 Document Path: / Document Length: 13 bytes Concurrency Level: 2500 Time taken for tests: 2.088 seconds Complete requests: 5000 Failed requests: 0 Write errors: 0 Total transferred: 401247 bytes HTML transferred: 67743 bytes Requests per second: 2395.19 [#/sec] (mean) Time per request: 1043.759 [ms] (mean) Time per request: 0.418 [ms] (mean, across all concurrent requests) Transfer rate: 187.71 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 64 195 65.7 226 287 Processing: 79 319 131.7 291 574 Waiting: 48 246 121.2 200 526 Total: 213 514 179.0 538 805 Percentage of the requests served within a certain time (ms) 50% 538 66% 631 75% 689 80% 697 90% 743 95% 789 98% 799 99% 802 100% 805 (longest request) Additional info:
Can you provide a sosreport of the system in question? I'd like to see how much memory you have on board, and what your core networking memory sysctls are set to. Thanks!
Created attachment 345910 [details] sosreport sosreport from VM from machine ab was run from. If was a clone of the server machine in this test. Problem has been duplicated between 8GB/Quad-core ab client and 32GB/24 Core server so I don't think it is a physical resource issue.
Thanks for that. Lookin at the code, my first guess would be that you're simply not able to allocate buffers under load on this system (although that would seem unlikely). I need to get two systems to set an ipsec tunnel up when I have time, but I tried the test on localhost here and it worked just fine (not that that proves anything). Looking at your sosreport shows you seem to be using a good chunk of memory, but by no means all in your system. Unfortunately for some reason the skbuff slab caches aren't appearing in /proc/slabinfo, so I can't tell how much you have allocated there (I'll need to look into that further). While I'm getting systems rounded up here, can you tell me if removing the ipsec tunnel on your setup and simply running the benchmark provides the same faliing results? Knowing that will help narrow down the amount of code to look through for this problem. Thanks
Same two machines and loopback have no issues without ipsec. See 'Expected results: " above.
Updated to reflect that this affects Fedora 10 and 11
XfrmInError 0 XfrmInBufferError 0 XfrmInHdrError 0 XfrmInNoStates 0 XfrmInStateProtoError 0 XfrmInStateModeError 0 XfrmInStateSeqError 207 XfrmInStateExpired 0 XfrmInStateMismatch 0 XfrmInStateInvalid 0 XfrmInTmplMismatch 0 XfrmInNoPols 0 XfrmInPolBlock 0 XfrmInPolError 0 XfrmOutError 0 XfrmOutBundleGenError 1144 XfrmOutBundleCheckError 0 XfrmOutNoStates 6000 XfrmOutStateProtoError 0 XfrmOutStateModeError 0 XfrmOutStateSeqError 0 XfrmOutStateExpired 0 XfrmOutPolBlock 0 XfrmOutPolDead 0 XfrmOutPolError 0
Ok, I think I've found the problem, or at least part of it. The error is likely stemming from the XfrmOutBundleGenError stats you see increasing. Those stats get incremented when the xfrm code goes to allocate a dst_entry for the route cache, but winds up not being able to. In that case the GenError stat gets incremented and we return ENOBUFS. There are two reasons this happens, either we are above 2x the garbage collection threshold, or we simply can't get any more ram out of heap. Since this appears to be your only issue with this system, I'm going to assume the former is happening. It would be good to monitor your free memory count when the problem happens though, just to be certain. That said, we need to make this tunable, I think, i.e. we need to let the gc thresh be adjustable for the xfrm code the same way its adjustable for the route cache. Currently its hard-coded to 1024 entries. I can start working on this patch soon, but in the interim, would you mind attaching the below patch? Its not a final solution, but it will tell us if we're on the right track. It just bumps up the size of the gc threshold to 8192 from 1024. If that solves the problem we'll know we're on the right path
Created attachment 354568 [details] patch to increase gc threshold on xfrm code heres the patch. This isn't a permanent solution, just something to make sure that we've identified the problem. If this fixes the issue, I'll go ahead and write the larger patch to make xfrm gc limits tunable.
I see the same error when setting up a he ipv6 tunnel. No buffer space available using Linux-net tools works and using linux iproute2 give the No buffer space available
I patched the fedora 10 2.6.29 kernel in updates-testing and it looks good so far.
Ok, cool, I'll start working on a more appropriate port for upstream shortly. Thanks!
I've sent the patch upstream, and cc'd you on it. As soon as its accepted, I'll backport it to F-11: http://marc.info/?l=linux-netdev&m=124871899201909&w=2
If possible a backport to F10 would be appreciated.
This message is a reminder that Fedora 11 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 11. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '11'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 11's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 11 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.