Bug 145424

Created attachment 109914 [details]
tcpdump between rhel3 and rhel4beta

Please let me know if you need the setup scripts, don't know if the customer
would  like if I put them up into a public BZ, or you may have a lookin in IT
alternatively.

Created attachment 109915 [details]
second attach

Created attachment 109916 [details]
third attach

Please try the ipsec-tools at:

http://people.redhat.com/notting/ipsec/

Does that help?

You can find actual a message in IT#59401, it is from the RHEL3 GW
form Mar 4.

Mar  4 07:12:38 adsl01 racoon: ERROR: pfkey.c:210:pfkey_handler():
pfkey ADD failed: File exists

The "pfkey ADD failed: File exists" is known from the RHEL3 -
RHEL4_BETA2 connections.

The new RPMS are build for RHEL4. I have installed the i386 now on two
RHEL4 GWs and I will see over 24 hours the result.

Can or should I also install this new version on the RHEL3 GWs or do
this break the RHEL3 versions management?

I am not sure, but I think the problem comes from RHEL3 if I see the
messages from RHEL3 and RHEL4 in IT#59401.

Created attachment 111698 [details]
ipsec-tools.spec patch

(In reply to comment #4)
> Please try the ipsec-tools at:
> 
> http://people.redhat.com/notting/ipsec/
> 
> Does that help?

With ipsec-tools-0.5-0.RHEL4 the racoon daemon can not start.

1. /var/log/messages:
- racoon: ERROR: glob found no matches for path
- racoon: failed to parse configuration file.

raccon was build with /etc/raccon.conf as default. This breaks the RHEL ipsec
configation generated with system-config-network and initscripts (ifup-ipsec and
ifdown-ipsec).

2. /var/log/messages:
- racoon: ERROR: bind(sockname:/var/racoon/racoon.sock): No such file or directory

The ipsec-tools-0.5-0.RHEL4 do not contain or create the /var/racoon directory.

After ln -s  /etc/racoon/racoon.conf /etc/racoon.conf and create /var/racoon the
racoon daemon can start and create automatically the "fwd" policies.

I add the ipsec-tools-0.5-0.RHEL4.patch, hope ist o.k.. Can you please build new
rpms and put them in http://people.redhat.com/notting/ipsec/ ?

Other than I say in additional Comment #5 the ipsec-tools-0.5-0.RHEL4
is only on one RHEL4 GW installed at this time. The other RHEL4 GW has
the ipsec-tools-0.3.3-2.1 because the ipsec-tools-0.5-0.RHEL4 can not
install without corections by hand.

Now I see on all GWs which do not have the new ipsec-tools-0.5-0.RHEL4
the error:

racoon: ERROR: pfkey.c:210:pfkey_handler(): pfkey ADD failed: File exists

In details:
adsl01.dyndomain.org has address 84.160.142.253
adslot.dyndomain.org has address 84.153.141.244
adslwu.dyndomain.org has address 217.236.65.8
adslub.dyndomain.org has address 217.93.253.48

adsl01 (RHEL3, ipsec-tools-0.2.5-0.5)
Mar  7 00:43:17 adsl01 racoon: ERROR: pfkey.c:210:pfkey_handler():
pfkey ADD failed: File existsMar  7 00:43:18 adsl01 racoon: INFO:
pfkey.c:1394:pk_recvexpire(): IPsec-SA expired: ESP/Tunnel
217.93.253.48->84.160.142.253 spi=13545290(0xceaf4a)
Mar  7 00:43:18 adsl01 racoon: INFO: pfkey.c:1394:pk_recvexpire():
IPsec-SA expired: ESP/Tunnel 84.160.142.253->217.93.253.48
spi=180184966(0xabd6786)
Mar  7 00:43:18 adsl01 racoon: INFO:
isakmp.c:1058:isakmp_ph2begin_r(): respond new phase 2 negotiation:
84.160.142.253[0]<=>217.93.253.48[0]
Mar  7 00:43:19 adsl01 modprobe: modprobe: Can't locate module ripemd160
Mar  7 00:43:19 adsl01 modprobe: modprobe: Can't locate module cast128
Mar  7 00:43:19 adsl01 modprobe: modprobe: Can't locate module lzs
Mar  7 00:43:19 adsl01 modprobe: modprobe: Can't locate module lzjh
Mar  7 00:43:19 adsl01 modprobe: modprobe: Can't locate module ripemd160
Mar  7 00:43:19 adsl01 modprobe: modprobe: Can't locate module cast128
Mar  7 00:43:19 adsl01 modprobe: modprobe: Can't locate module lzs
Mar  7 00:43:19 adsl01 modprobe: modprobe: Can't locate module lzjh
Mar  7 00:43:19 adsl01 racoon: INFO: pfkey.c:1127:pk_recvupdate():
IPsec-SA established: ESP/Tunnel 217.93.253.48->84.160.142.253
spi=24472347(0x1756b1b)
Mar  7 00:43:19 adsl01 racoon: ERROR: pfkey.c:210:pfkey_handler():
pfkey ADD failed: File exists

adslot (RHEL4, ipsec-tools-0.3.3-2.1)
Mar  7 01:41:29 adslot racoon: INFO: IPsec-SA expired: ESP/Tunnel
217.93.253.48->84.153.141.244 spi=200846609(0xbf8ad11)
Mar  7 01:41:29 adslot racoon: INFO: IPsec-SA expired: ESP/Tunnel
84.153.141.244->217.93.253.48 spi=132522918(0x7e623a6)
Mar  7 01:41:29 adslot racoon: INFO: respond new phase 2 negotiation:
84.153.141.244[0]<=>217.93.253.48[0]
Mar  7 01:41:29 adslot racoon: INFO: IPsec-SA established: ESP/Tunnel
217.93.253.48->84.153.141.244 spi=222512179(0xd434433)
Mar  7 01:41:29 adslot racoon: ERROR: pfkey ADD failed: File exists
Mar  7 01:41:31 adslot racoon: INFO: respond new phase 2 negotiation:
84.153.141.244[0]<=>217.93.253.48[0]
Mar  7 01:41:31 adslot racoon: INFO: IPsec-SA established: ESP/Tunnel
217.93.253.48->84.153.141.244 spi=86254291(0x52422d3)

adslwu (RHEL4, ipsec-tools-0.3.3-2.1)
Mar  7 01:41:17 adslwu racoon: INFO:
isakmp.c:1058:isakmp_ph2begin_r(): respond new phase 2 negotiation:
217.236.65.8[0]<=>217.93.253.48[0]
Mar  7 01:41:17 adslwu racoon: INFO: pfkey.c:1394:pk_recvexpire():
IPsec-SA expired: ESP/Tunnel 217.93.253.48->217.236.65.8
spi=162963483(0x9b6a01b)
Mar  7 01:41:17 adslwu racoon: INFO: pfkey.c:1394:pk_recvexpire():
IPsec-SA expired: ESP/Tunnel 217.236.65.8->217.93.253.48
spi=159235033(0x97dbbd9)
Mar  7 01:41:17 adslwu racoon: WARNING: pfkey.c:1422:pk_recvexpire():
the expire message is received but the handler has not been established.
Mar  7 01:41:17 adslwu modprobe: modprobe: Can't locate module
ripemd160Mar  7 01:41:17 adslwu modprobe: modprobe: Can't locate
module cast128
Mar  7 01:41:17 adslwu modprobe: modprobe: Can't locate module lzsMar
 7 01:41:17 adslwu modprobe: modprobe: Can't locate module lzjh
Mar  7 01:41:17 adslwu modprobe: modprobe: Can't locate module
ripemd160Mar  7 01:41:17 adslwu modprobe: modprobe: Can't locate
module cast128
Mar  7 01:41:17 adslwu modprobe: modprobe: Can't locate module lzsMar
 7 01:41:17 adslwu modprobe: modprobe: Can't locate module lzjh
Mar  7 01:41:17 adslwu racoon: INFO: pfkey.c:1127:pk_recvupdate():
IPsec-SA established: ESP/Tunnel 217.93.253.48->217.236.65.8
spi=176316142(0xa825eee)
Mar  7 01:41:17 adslwu racoon: ERROR: pfkey.c:210:pfkey_handler():
pfkey ADD failed: File exists 

adslub (RHEL4, ipsec-tools-0.5-0.RHEL4)
Mar  7 01:43:50 adslub racoon: INFO: IPsec-SA expired: ESP/Tunnel
84.160.142.253->217.93.253.48 spi=180184966(0xabd6786)
Mar  7 01:43:50 adslub racoon: INFO: initiate new phase 2 negotiation:
217.93.253.48[0]<=>84.160.142.253[0]
Mar  7 01:43:50 adslub racoon: INFO: IPsec-SA established: ESP/Tunnel
217.236.65.8->217.93.253.48 spi=159235033(0x97dbbd9)
Mar  7 01:43:50 adslub racoon: INFO: IPsec-SA established: ESP/Tunnel
217.93.253.48->217.236.65.8 spi=59603142(0x38d78c6)
Mar  7 01:43:51 adslub racoon: INFO: IPsec-SA expired: ESP/Tunnel
84.153.141.244->217.93.253.48 spi=132522918(0x7e623a6)
Mar  7 01:43:51 adslub racoon: INFO: initiate new phase 2 negotiation:
217.93.253.48[0]<=>84.153.141.244[0]
Mar  7 01:43:51 adslub racoon: INFO: IPsec-SA established: ESP/Tunnel
84.160.142.253->217.93.253.48 spi=180184966(0xabd6786)
Mar  7 01:43:51 adslub racoon: INFO: IPsec-SA established: ESP/Tunnel
217.93.253.48->84.160.142.253 spi=144866554(0x8a27cfa)

I install now the ipsec-tools-0.5-0.RHEL4 on the second RHEL4 GW
(adslot). But what should I do with the RHEL3 GWs?

It is possible to change this bugzilla report to Red Hat Enterprise
Linux Version 4 (begin was in beta)?

Patch added, ipsec-tools-0.5-1.RHEL4 updated.

The ipsec-tools-0.5-1.RHEL4 installation is correct now.

Not sure if I misunderstood you; does the installation of 0.5 on the
RHEL 4 gateways:

- solve the problem completely?
- work, but cause errors in the logs?
- not work at all?

I mean only the installation of the ipsec-tools-0.5-1.RHEL4 rpm. This
in now o.k..

Between RHEL3 and RHEL4 the key change after 384 minutes is critical.

IPsec-SA expired:
IPsec-SA established:
ISAKMP-SA expired
ISAKMP-SA deleted

If this do not work correct there are problems with the connections
over ipsec and racoon generate on over one hour new SAs. This SAs are
not generate on the other side. If this are over 500 SAs no new
connections can established.
Sometime the problem is directly at the first day when racoon starts
new. It can also be, that it is after some days.

So I can only wait and see in the next days if ipsec-tools-0.5-1.RHEL4
solve the problem. I hope so.

ipsec-tools-0.5-1.RHEL4 rpm do not solve the keychange problem between
the RHEL3 and RHEL4 gateways. During the raccon works on the keychange
problem traffic over the tunnel is possible now. With
ipsec-tools-0.3.3-2.1 this was not so.

In the last 18 hours there were two keychanges. You can find messages,
setkey -D from two RHEL3 and two RHLE4 gateways and tcpdump form a
RHEL4 gateway in issue tracker issue #59401. There are also some
additional informations - please contact Steffen Mann for it.

In my case I'm unable to connect RHEL3 and RHEL4 with a LAN-LAN setup
described in RH's Security guide (with shared secret). The connection
dies right after first packet was transmitted. I was able to pass
through 1% of ping. The tunnel was repeatedly builded an dropped, see
bug #150179 comment #13.

With no change my setup works with two RHEL3 gateways.

Doing some testing here, I'm noticing:

- manual mode - works fine

As for combinations:

- RHEL 3 <-> 2.6-based distro                    - FAILS (as stated)
- RHEL 3 <-> RHEL 3                              - WORKS
- RHEL 3 <-> RHEL 3 + 2.6 kernel + current tools - WORKS
- RHEL 3 + new ipsec tools <-> 2.6-based distro  - FAILS

This is not particularly enlightening as to discovering the source of the
problem. :/  Will do more testing.

Please try the patch for a 2.6 kernel at:

http://marc.theaimsgroup.com/?l=linux-netdev&m=111030409923301&q=p3

Reproduced here inline:

===== net/xfrm/xfrm_state.c 1.55 vs edited =====
--- 1.55/net/xfrm/xfrm_state.c	2005-03-07 06:23:53 +01:00
+++ edited/net/xfrm/xfrm_state.c	2005-03-08 18:42:13 +01:00
@@ -609,7 +609,7 @@
 
 	for (i = 0; i < XFRM_DST_HSIZE; i++) {
 		list_for_each_entry(x, xfrm_state_bydst+i, bydst) {
-			if (x->km.seq == seq) {
+			if (x->km.seq == seq && x->km.state == XFRM_STATE_ACQ) {
 				xfrm_state_hold(x);
 				return x;
 			}

http://marc.theaimsgroup.com/?l=linux-netdev&m=111030409923301&w=2

is the thread, if you want more info.

Interesting and I think this is what I see since RHEL4_BETA.

At the last 24 hours there were only this tunnel activ for me and
there was no problem with keychange after 384 minutes.

- adslwu (RHEL3) - adsl01 (RHEL3) - tunnel was automatically init
after ADSL reconnect and restart raccon bei systems in the networks

- adslub (RHEL4) - adsl01 (RHEL3) - I init the tunnel from adslub
(RHEL4) with ICMP if I need it

Important: in the last reports in issue tracker the tunnel adslub -
adsl01 works correct and was always init form adslub. But if I init
tunnels from RHEL3 to RHEL4 then I have the problems.
I think it is a good idea to test the patch.

What will you do now Bill? Test the patch first in your environment
and then put the kernel rpms in your people?
I use x86 (i686) up and smp kernels. This would be a easy way for me
and the test in the real environment with two RHEL3 and two RHEL4
gateways could start soon.

Test kernels uploaded to http://people.redhat.com/notting/ipsec/

FWIW, a version of ipsec-tools-0.3.3 with handling for FWD policies is
now in that dir as well; it is more likely that that will be released
for RHEL 4 rather than 0.5. But I don't think that's necessarily
related to your issue.

Milan: I'm fairly confident the new kernel will solve your issue; just switching
the kernel to this fixed the tunnel for me.

Uwe: Since your configuration is more extensive than mine, I'd love to hear
feedback as to whether this solves the issue for you.

Created attachment 111928 [details]
messages and tcpdumps duplicate SA

The new kernels are installed on two gateways.

The SA expired/established looks good now. But it is to early after
one change to say that it really solve the problem.

I see the following in messages:

from adslot (RHEL4) - adslub (RHEL4) with ipsec-tools-0.5-1.RHEL4:

Mar 12 23:34:41 adslub racoon: INFO: respond new phase 1 negotiation:
217.93.243.38[500]<=>84.153.168.100[500]
Mar 12 23:34:41 adslub racoon: INFO: begin Identity Protection mode.
Mar 12 23:34:41 adslub racoon: INFO: received Vendor ID: DPD
Mar 12 23:34:41 adslub racoon: INFO: ISAKMP-SA established
217.93.243.38[500]-84.153.168.100[500] spi:faad44618b8d3c82:361
463c9baf8d596
Mar 12 23:34:42 adslub racoon: INFO: respond new phase 2 negotiation:
217.93.243.38[0]<=>84.153.168.100[0]
Mar 12 23:34:43 adslub racoon: INFO: IPsec-SA established: ESP/Tunnel
84.153.168.100->217.93.243.38 spi=148964289(0x8e103c1)
Mar 12 23:34:43 adslub racoon: INFO: IPsec-SA established: ESP/Tunnel
217.93.243.38->84.153.168.100 spi=23504102(0x166a4e6)

--> racoon: INFO: received Vendor ID: DPD

from adslub (RHEL4) - adsl01 (RHEL3)

messages adslub:
Mar 12 23:30:37 adslub racoon: INFO: IPsec-SA request for
84.160.173.130 queued due to no phase1 found.
Mar 12 23:30:37 adslub racoon: INFO: initiate new phase 1 negotiation:
217.93.243.38[500]<=>84.160.173.130[500]
Mar 12 23:30:37 adslub racoon: INFO: begin Identity Protection mode.
Mar 12 23:30:37 adslub racoon: INFO: received Vendor ID: KAME/racoon
Mar 12 23:30:37 adslub racoon: INFO: received Vendor ID: KAME/racoon
Mar 12 23:30:38 adslub racoon: INFO: ISAKMP-SA established
217.93.243.38[500]-84.160.173.130[500]
spi:9d98f59b290cf3c3:e8e6f21bd55068dc
Mar 12 23:30:38 adslub racoon: INFO: initiate new phase 2 negotiation:
217.93.243.38[0]<=>84.160.173.130[0]
Mar 12 23:30:38 adslub racoon: INFO: IPsec-SA established: ESP/Tunnel
84.160.173.130->217.93.243.38 spi=92941571(0x58a2d03)
Mar 12 23:30:38 adslub racoon: INFO: IPsec-SA established: ESP/Tunnel
217.93.243.38->84.160.173.130 spi=92605049(0x5850a79)

2 x racoon: INFO: received Vendor ID: KAME/racoon ?
The tunnel is correct established.

from adsl01 (RHEL3) - adslub (RHEL4)

messages adsl01:
Mar 12 23:30:17 adsl01 racoon: INFO: isakmp.c:807:isakmp_ph1begin_i():
initiate new phase 1 negotiation: 84.160.173.130[50
0]<=>217.236.86.104[500]
Mar 12 23:30:17 adsl01 racoon: INFO: isakmp.c:812:isakmp_ph1begin_i():
begin Identity Protection mode.
Mar 12 23:30:17 adsl01 racoon: INFO: vendorid.c:128:check_vendorid():
received Vendor ID: KAME/racoon
Mar 12 23:30:17 adsl01 racoon: INFO: vendorid.c:128:check_vendorid():
received Vendor ID: KAME/racoon
Mar 12 23:30:18 adsl01 racoon: INFO:
isakmp.c:2443:log_ph1established(): ISAKMP-SA established
84.160.173.130[500]-217.236.86.104[500]
spi:5d6d830b4e5d4a72:637d43f3527d3102
Mar 12 23:30:18 adsl01 racoon: INFO: isakmp.c:951:isakmp_ph2begin_i():
initiate new phase 2 negotiation: 84.160.173.130[0]<=>217.236.86.104[0]
Mar 12 23:30:18 adsl01 racoon: INFO: pfkey.c:1127:pk_recvupdate():
IPsec-SA established: ESP/Tunnel 217.236.86.104->84.160.173.130
spi=19496869(0x1297fa5)
Mar 12 23:30:18 adsl01 racoon: INFO: pfkey.c:1348:pk_recvadd():
IPsec-SA established: ESP/Tunnel 84.160.173.130->217.236.86.104
spi=78054602(0x4a704ca)

2 x racoon: INFO: vendorid.c:128:check_vendorid(): received Vendor ID:
KAME/racoon ?
The tunnel is correct established.

Looks like a fault in ipsec-tools-0.2.5-0.6?

But I see also duplicate SAs for the same policy. Messages and
tcpdumps are in attachment duplicate-sa.tar.gz.
I see this problem only on two gateways and can reproduce it every
time. It is only if I init the connetion from adslwu (RHEL3) to adslub
(RHEL4). It is not in the other direction. The tunnel works but can
this be normal?
Configuration of ipsec and iptables are the same. What are the
differences between the gateways.
- adslwu P4 3,06 GHz
- adslub P3 600 MHz
- adslot P4 3,06 GHz
- adsl01 P2 266 MHz
adslwu -> adslot ok
adsl01 -> adslub ok
adslwu -> adslub ok but two tunnels

I don't have a live system anymore (I was forced to go back to RHEL3) so I'll
try to test patched kernel later in the lab. But - thank you for your fast response!

With kernel-2.6.9-5.0.3.EL.notting.ipsec.i686.rpm ipsec between RHEL3 and RHEL4
is not longer a problem. It works now correct.

The SA expired/established works also correct if there was duplicate SA
established at the init of the connection. I do not know why? Duplicate SA are
not everytime generated. I use ipsec-tools-0.5-1.RHEL4 on the RHEL4 gateways.
Some more informations you can find in IT#59401.
If you also do not see the problem then it is not longer relevant for this
bugzilla entry. I will open a now bugzilla if there are relevant problems in the
future.

It would be good to have an errata soon.

Yup. Too bad 2.6.9-5.0.3.EL is now obsolete, and the fix for this bug didn't go
in the ERRATA.

There are so many security fixes in 2.6.9-5.0.5.EL but the fix for the ipsec in
this bugzilla not. I need the 2.6.9-5.0.5.EL kernel with the fix from comment
#18 for some productiv systems because the kernel-2.6.9-5.0.3.EL.notting.ipsec
is now also obsolete. Since I use kernel-2.6.9-5.0.3.EL.notting.ipsec there was
no problems with ipsec in RHEL4 and also between RHEL3 and RHEL4.

Sorry, I do not understand Red Hat. Is ipsec not a security problem?

Uwe, IPSec not working is not a security problem, but a bug, nothing more.
The IPSec bug fix will be in RHEL4 Update 1, it will not be integrated in
security updates.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2005-420.html