Bug 503124 - No buffer space available with many concurrent sockets over ipsec connection
No buffer space available with many concurrent sockets over ipsec connection
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
11
All Linux
low Severity high
: ---
: ---
Assigned To: Neil Horman
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-28 21:48 EDT by Joe Nall
Modified: 2010-06-28 08:44 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-06-28 08:44:31 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Test libevent server (845 bytes, text/plain)
2009-05-28 21:48 EDT, Joe Nall
no flags Details
sosreport (553.08 KB, application/x-bzip2)
2009-05-29 10:40 EDT, Joe Nall
no flags Details
patch to increase gc threshold on xfrm code (779 bytes, patch)
2009-07-21 15:40 EDT, Neil Horman
no flags Details | Diff

  None (edit)
Description Joe Nall 2009-05-28 21:48:11 EDT
Created attachment 345856 [details]
Test libevent server

Description of problem:

When creating many concurrent connections over a host2host ipsec session, connect fails with ENOBUFS (105). The same two machines have no issue when ipsec is not enabled. A demonstration libevent based server is attached. ab us used to create the concurrent connections.

This is a recreation of something we are seeing in the field with ipsec thin clients with many server/X connections. Reproducible on 2.6.28 as well.

I suspect a bug in the IPSec SA replay window logic.

Both machines are up to date F10 on 5/28/09

Version-Release number of selected component (if applicable):

kernel-2.6.27.24-170.2.68.fc10.x86_64
ipsec-tools-0.7.2-1.fc10.x86_64

How reproducible:

100%

Steps to Reproduce:
1. Raise nofiles to 10000 in /etc/security/limits.conf
2. Establish IPSec session between two machines
3. Run attached simple web server on one machine
4. Open port 8080/disable iptables
5. Run ab on the other machine

Actual results:

[joe@f10-2 ~]$ ab -c 2500 -n5000 http://f10:8080/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking f10 (be patient)

Test aborted after 10 failures

apr_socket_connect(): No buffer space available (105)

Expected results:

Before starting IPSec session

[joe@f10-2 ~]$ ab -c 2500 -n5000 http://f10:8080/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking f10 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests


Server Software:        
Server Hostname:        f10
Server Port:            8080

Document Path:          /
Document Length:        13 bytes

Concurrency Level:      2500
Time taken for tests:   2.088 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      401247 bytes
HTML transferred:       67743 bytes
Requests per second:    2395.19 [#/sec] (mean)
Time per request:       1043.759 [ms] (mean)
Time per request:       0.418 [ms] (mean, across all concurrent requests)
Transfer rate:          187.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       64  195  65.7    226     287
Processing:    79  319 131.7    291     574
Waiting:       48  246 121.2    200     526
Total:        213  514 179.0    538     805

Percentage of the requests served within a certain time (ms)
  50%    538
  66%    631
  75%    689
  80%    697
  90%    743
  95%    789
  98%    799
  99%    802
 100%    805 (longest request)

Additional info:
Comment 1 Neil Horman 2009-05-29 06:41:30 EDT
Can you provide a sosreport of the system in question?  I'd like to see how much memory you have on board, and what your core networking memory sysctls are set to.  Thanks!
Comment 2 Joe Nall 2009-05-29 10:40:24 EDT
Created attachment 345910 [details]
sosreport

sosreport from VM from machine ab was run from. If was a clone of the server machine in this test. Problem has been duplicated between 8GB/Quad-core ab client and 32GB/24 Core server so I don't think it is a physical resource issue.
Comment 3 Neil Horman 2009-05-29 14:57:51 EDT
Thanks for that.  Lookin at the code, my first guess would be that you're simply not able to allocate buffers under load on this system (although that would seem unlikely).  I need to get two systems to set an ipsec tunnel up when I have time, but I tried the test on localhost here and it worked just fine (not that that proves anything). Looking at your sosreport shows you seem to be using a good chunk of memory, but by no means all in your system. Unfortunately for some reason the skbuff slab caches aren't appearing in /proc/slabinfo, so I can't tell how much you have allocated there (I'll need to look into that further).  While I'm getting systems rounded up here, can you tell me if removing the ipsec tunnel on your setup and simply running the benchmark provides the same faliing results?  Knowing that will help narrow down the amount of code to look through for this problem.

Thanks
Comment 4 Joe Nall 2009-05-29 15:38:04 EDT
Same two machines and loopback have no issues without ipsec. See 'Expected results:
" above.
Comment 5 Joe Nall 2009-07-19 23:45:14 EDT
Updated to reflect that this affects Fedora 10 and 11
Comment 6 Joe Nall 2009-07-20 00:54:13 EDT
XfrmInError                     0
XfrmInBufferError               0
XfrmInHdrError                  0
XfrmInNoStates                  0
XfrmInStateProtoError           0
XfrmInStateModeError            0
XfrmInStateSeqError             207
XfrmInStateExpired              0
XfrmInStateMismatch             0
XfrmInStateInvalid              0
XfrmInTmplMismatch              0
XfrmInNoPols                    0
XfrmInPolBlock                  0
XfrmInPolError                  0
XfrmOutError                    0
XfrmOutBundleGenError           1144
XfrmOutBundleCheckError         0
XfrmOutNoStates                 6000
XfrmOutStateProtoError          0
XfrmOutStateModeError           0
XfrmOutStateSeqError            0
XfrmOutStateExpired             0
XfrmOutPolBlock                 0
XfrmOutPolDead                  0
XfrmOutPolError                 0
Comment 7 Neil Horman 2009-07-21 15:35:54 EDT
Ok, I think I've found the problem, or at least part of it.  The error is likely stemming from the XfrmOutBundleGenError stats you see increasing.  Those stats get incremented when the xfrm code goes to allocate a dst_entry for the route cache, but winds up not being able to.  In that case the GenError stat gets incremented and we return ENOBUFS.  There are two reasons this happens, either we are above 2x the garbage collection threshold, or we simply can't get any more ram out of heap.  Since this appears to be your only issue with this system, I'm going to assume the former is happening.  It would be good to monitor your free memory count when the problem happens though, just to be certain.

That said, we need to make this tunable, I think, i.e. we need to let the gc thresh be adjustable for the xfrm code the same way its adjustable for the route cache.  Currently its hard-coded to 1024 entries.  I can start working on this patch soon, but in the interim, would you mind attaching the below patch?  Its not a final solution, but it will tell us if we're on the right track.  It just bumps up the size of the gc threshold to 8192 from 1024.  If that solves the problem we'll know we're on the right path
Comment 8 Neil Horman 2009-07-21 15:40:16 EDT
Created attachment 354568 [details]
patch to increase gc threshold on xfrm code

heres the patch.  This isn't a permanent solution, just something to make sure that we've identified the problem.  If this fixes the issue, I'll go ahead and write the larger patch to make xfrm gc limits tunable.
Comment 9 Itamar Reis Peixoto 2009-07-21 17:45:06 EDT
I see the same error when setting up a he ipv6 tunnel.

No buffer space available

using Linux-net tools works and using linux iproute2 give the No buffer space available
Comment 10 Joe Nall 2009-07-22 13:43:36 EDT
I patched the fedora 10 2.6.29 kernel in updates-testing and it looks good so far.
Comment 11 Neil Horman 2009-07-22 15:44:40 EDT
Ok, cool, I'll start working on a more appropriate port for upstream shortly.  Thanks!
Comment 12 Neil Horman 2009-07-27 14:24:25 EDT
I've sent the patch upstream, and cc'd you on it.  As soon as its accepted, I'll backport it to F-11:

http://marc.info/?l=linux-netdev&m=124871899201909&w=2
Comment 13 Joe Nall 2009-07-27 19:41:18 EDT
If possible a backport to F10 would be appreciated.
Comment 14 Bug Zapper 2010-04-27 10:33:49 EDT
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 15 Bug Zapper 2010-06-28 08:44:31 EDT
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.