Bug 607361 - pfkey socket buffer overflow
Summary: pfkey socket buffer overflow
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: ipsec-tools
Version: 5.5
Hardware: All
OS: Linux
Target Milestone: rc
: ---
Assignee: Tomas Mraz
QA Contact: Ondrej Moriš
Depends On:
Blocks: 609084
TreeView+ depends on / blocked
Reported: 2010-06-23 22:08 UTC by Bryan Mason
Modified: 2018-10-27 11:37 UTC (History)
8 users (show)

Fixed In Version: ipsec-tools-0.6.5-14.el5.4
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2013-10-02 10:55:44 UTC
Target Upstream Version:

Attachments (Terms of Use)
Proposed patch (1.15 KB, patch)
2010-06-23 22:09 UTC, Bryan Mason
no flags Details | Diff
Output of 'strace setkey -f setkey.conf-huge' (254.09 KB, text/plain)
2010-08-20 10:54 UTC, Ondrej Moriš
no flags Details
setkey input containing ~800 SPD rules (91.48 KB, text/plain)
2010-08-20 10:55 UTC, Ondrej Moriš
no flags Details
Output of 'strace /usr/sbin/racoon -l /var/log/racoon.log -d -d -d > /var/log/racoon.strace.log 2>&1' from ifup-ipsec (91.48 KB, application/octet-stream)
2010-08-20 11:08 UTC, Ondrej Moriš
no flags Details
Output of 'strace /usr/sbin/racoon -l /var/log/racoon.log -d -d -d > /var/log/racoon.strace.log 2>&1' from ifup-ipsec (28.46 KB, text/plain)
2010-08-20 11:09 UTC, Ondrej Moriš
no flags Details

Description Bryan Mason 2010-06-23 22:08:16 UTC
Description of problem:

    In ipsec-tools, pfkey_open() in libipsec/pfkey.c sets a 128K
    receive buffer size.  This only allows about 600 SPD entries to be
    dumped without buffer overflow.

    Customer needs ipsec-tools to be able to handle ~9000 SPD entries
    (400 targets with 24 SPD entries per target), which corresponds
    to a buffer size of 2 MB.

Version-Release number of selected component (if applicable):

    ipsec-tools 0.6.5-13.el5-13.1

How reproducible:

    100% with enough SPD entries 

Additional info:

Upstream code has a 1 MB buffer:


Upstream patch is here:


Comment 1 Bryan Mason 2010-06-23 22:09:23 UTC
Created attachment 426403 [details]
Proposed patch

Increased buffer size to 2 MB.  Based on upstream patch mentioned in Description.

Comment 2 Tomas Mraz 2010-06-24 06:58:01 UTC
I do not like this change as this has the consequence of higher memory requirements in kernel. But I've found that upstream CVS already contains a different patch that allows setting the pfkey receive buffer size in the racoon configuration. I would like to add this patch.

This bug should be proposed for 5.5.z so we can include the patch in the 5.5.z errata that is in the works.

Comment 7 Tomas Mraz 2010-06-29 08:05:47 UTC
Added a patch from upstream that allows to configure the pfkey socket buffer size.


Comment 9 Ondrej Moriš 2010-07-27 00:00:41 UTC
Tom, I can't verify this bug-fix since I see no difference between old (ipsec-tools-0.6.5-13.el5_3.1) and new (ipsec-tools-0.6.5-14.el5_5.4) version. 

My testing environment is as follows:

* two virtual machines (A and B)
* PSK IPsec connection from A to B and vice versa via ifcg/ifup
* racoon in configured with "pfkey_buffer 4000;" on both A and B
* SPD rules are as follows:

A (2 rules):

spdadd any -P out ipsec esp/transport//requ\
ire ;
spdadd any -P in ipsec esp/transport//requi\
re ;

B (552 rules):

spdadd any -P out ipsec esp/tunnel/\
[4500]-[30000]/require ;
... (548 different tunnel rules) ...
spdadd any -P out ipsec esp/tunnel/192.168.0\
.1[4500]-[30000]/require ;
spdadd any -P out ipsec esp/transport//requ\
ire ;
spdadd any -P in ipsec esp/transport//requi\
re ;

Starting IPsec via ifup correctly starts racoon. On A all SPD rules are loaded, on B only first ~512 rules are loaded and the others are missing (especially two last and most important) and hence 'ping B' from A does not work since there are no matching rules on B. 

Moreover, if this fix is intended for racoon, it should also fix setkey, since both old and new version on B give me the following: 

# setkey -DP | grep any | wc -l
recv: Resource temporarily unavailable

Is there something I missed? According to 'man racoon.conf' I should have allocated ~4MB for SPD rules (instead of default 252kB). Whenever I decrease number of rules to <512, it works fine. 

BTW: kill -9 `pidof racoon` causes removing 'include "A.B.C.D.conf" ;' from racoon.conf, it seems to be buggy behaviour to me, isn't it?

Comment 12 Ondrej Moriš 2010-07-27 08:36:38 UTC
(In reply to comment #9)

> BTW: kill -9 `pidof racoon` causes removing 'include "A.B.C.D.conf" ;' from
> racoon.conf, it seems to be buggy behaviour to me, isn't it?    

Sorry for silly question, this is correct behaviour.

Comment 22 Ondrej Moriš 2010-08-20 10:54:46 UTC
Created attachment 439907 [details]
Output of 'strace setkey -f setkey.conf-huge'

Comment 23 Ondrej Moriš 2010-08-20 10:55:33 UTC
Created attachment 439908 [details]
setkey input containing ~800 SPD rules

Comment 24 Ondrej Moriš 2010-08-20 11:08:49 UTC
Created attachment 439911 [details]
Output of 'strace /usr/sbin/racoon -l /var/log/racoon.log -d -d -d > /var/log/racoon.strace.log 2>&1' from ifup-ipsec

Comment 25 Ondrej Moriš 2010-08-20 11:09:47 UTC
Created attachment 439913 [details]
Output of 'strace /usr/sbin/racoon -l /var/log/racoon.log -d -d -d > /var/log/racoon.strace.log 2>&1' from ifup-ipsec

Comment 35 Tomas Mraz 2011-09-02 18:48:28 UTC
I am not sure why would you need that. Did the customer properly configured the buffer size in the racoon.conf?
If they have the problem with the setkey command, they can load the rules in more consecutive setkey calls.

Comment 36 Marcelo Ricardo Leitner 2011-09-05 13:08:33 UTC
Loading via consecutive calls would required adapting shell scripts and so on, as far as I'm aware of.

Customer didn't test it yet. It has been made available to him today. I'll keep you posted if it works.

Note You need to log in before you can comment on or make changes to this bug.