Bug 467789

Summary: a race preventing VPN connection establishing
Product: [Fedora] Fedora Reporter: Need Real Name <mal>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 11CC: kernel-maint
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-06-28 10:46:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
panick
none
panick (2)
none
panick (3)
none
panick(4)
none
kernel-2.6.27.5-41.fc9 panic none

Description Need Real Name 2008-10-20 21:28:24 UTC
I use the script below to establish VPN link.
When I start it - typically everything starts OK.
but when in some other window there is a traffic to VPN network 
it may not get started.

To reproduce:
1. start VPN.
2. In a separate window start 
ping with short intervals
ping -i 0.05 192.168.vpn.host
3. As ping is working - do Ctl-C and the start again the VPN script.
eventually - VPN link will not be established.
The only way to establish it - Ctl-C the script 
and start it over. eventually VPN link will be established.

ipsec-tools-0.7-13.fc9.i386
kernel-2.6.26.5-45.fc9.i686



echo "flush; spdflush;
        spdadd $VIRTUAL_IP_ADDR/24 $RENOTE_NETW any
            -P out ipsec esp/tunnel/$LOCAL_PUBLIC_ADDR-$VSU_ADDR/require ;
        spdadd $RENOTE_NETW $VIRTUAL_IP_ADDR/24 any
            -P in ipsec esp/tunnel/$VSU_ADDR-$LOCAL_PUBLIC_ADDR/require ;" \
        | setkey -c

    # bring up the virtual interface
    #/sbin/ifconfig eth2:1 up $VIRTUAL_IP_ADDR netmask 255.255.255.0

    # remove old default gateway and set up a new one
    #/sbin/route del default
    #/sbin/route add default gw $VIRTUAL_IP_ADDR
	/sbin/route add -net $RENOTE_NETW gw $VIRTUAL_IP_ADDR


    racoon -F -d -d -d

Comment 1 Chuck Ebbert 2008-10-22 04:59:27 UTC
Lots of bugs in this area were fixed in kernel 2.6.26.6-79. Please try that one -- it's in updates-testing now. Either that version or a later one will be released soon.

Comment 2 Chuck Ebbert 2008-10-24 02:24:26 UTC
kernel-2.6.26.6-79.fc9 is released. Please test it.

Comment 3 Need Real Name 2008-10-28 20:39:38 UTC
the kernel-2.6.26.6-79.fc9 has some problem definitely fixed,
but it stil has at least one, 
which was a seldom problem before now shows up extremely often.
The problem is this:

IPSEC is established, everything is OK.
Then, say 7200 seconds later, the key has expired,
and the message that VPN key is expired printed by racoon.
Then is about 50% cases the VPN link
DOES NOT GET RE-ESTABLISHED.

The only way to restore the link - Ctl-C racoon and then start it over.
I remember this problem before, but not often.

This is huge problem now.
Some computers become unreachable from outside,
just because VPN link is not re-established when the key have expired.

I am not sure this is racoon or kernel problem, may be both.

Comment 4 Fedora Update System 2008-11-05 03:37:21 UTC
kernel-2.6.27.4-24.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/kernel-2.6.27.4-24.fc9

Comment 5 Need Real Name 2008-11-05 08:05:14 UTC
I just tried 
Linux comp 2.6.27.4-24.fc9.x86_64 #1 SMP Tue Nov 4 19:10:40 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

with two windows (ping in one and VPN script in a second one)
After continuous Ctl-C and restarting
looks no problem. Sometimes it may take up to a minute,
but eventually the connection gets established.

I check the second problem (no connection re-establishing after keys expiration)
in the next few days.

Comment 6 Need Real Name 2008-11-05 09:26:12 UTC
Some problems still exist.
I have VPN connection 1 established.
Then I ctl-C the script and start connection 2 with the same script.
It gets established OK,

but when probably a packet from old connection 1 is received
the connection 2 stalls and nothing happens.

2008-11-05 04:13:07: DEBUG: malformed cookie received or the spi expired.
2008-11-05 04:13:15: DEBUG: ===
2008-11-05 04:13:15: DEBUG: 405 bytes message received from 1.2.3.4[500] to 192.168.0.90[500]
2008-11-05 04:13:15: DEBUG: 

after this connection 2 is broken and not get re-established.
The only way to start it - Ctl-C the script and start over.
This may be an way to perform DOS attack by breaking VPN link
using IPSEC packets.

Comment 7 Need Real Name 2008-11-05 13:23:45 UTC
The problem still exist,
VPN connection key expires, connection re-established
but actual connection does not work.

Comment 8 Fedora Update System 2008-11-06 01:51:12 UTC
kernel-2.6.27.4-26.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/kernel-2.6.27.4-26.fc9

Comment 9 Need Real Name 2008-11-06 08:24:13 UTC
The 2.6.27.4-26.fc9.x86_64 panics on boot while running mdadm in mkdir syscall.
I do not have serial cable handy to get the complete trace.
2.6.27.4-24.fc9.x86_64 is OK

Comment 10 Fedora Update System 2008-11-07 02:56:21 UTC
kernel-2.6.27.4-26.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update kernel'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-9467

Comment 11 Need Real Name 2008-11-09 21:47:08 UTC
I will check later on.
Also, recent ipsec update broke VPN links:
https://bugzilla.redhat.com/show_bug.cgi?id=470738


and I am not sure where the problem is, 
but I started getting OOM kills on 8Gb machine 
when I start VPN.
See 
free
ps 
and 
/proc/meminfo in #470738

Comment 12 Fedora Update System 2008-11-10 13:15:48 UTC
kernel-2.6.27.5-32.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/kernel-2.6.27.5-32.fc9

Comment 13 Need Real Name 2008-11-10 16:03:28 UTC
The
Linux comp 2.6.27.5-32.fc9.x86_64 #1 SMP Sun Nov 9 18:00:34 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
re-establishes connection OK, when I constantly Ctl-C and restart racoon.
Sometimes this may take a while (1-2 minutes), I am not whether this is this OK, probably yes.


I will see how 2.6.27.5-32 handles key expiration. This was the most common problem in previous kernels.

Comment 14 Need Real Name 2008-11-11 22:08:26 UTC
The uname -a
Linux comp 2.6.27.5-32.fc9.x86_64 #1 SMP Sun Nov 9 18:00:34 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
has much more stable VPN.

I had VPN link stopped working only once. I am not sure what it was.
Previously VPN link typically stop working every 4-5 hours.

Comment 15 Fedora Update System 2008-11-12 02:57:51 UTC
kernel-2.6.27.5-32.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update kernel'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-9583

Comment 16 Fedora Update System 2008-11-13 07:42:58 UTC
kernel-2.6.27.5-37.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/kernel-2.6.27.5-37.fc9

Comment 17 Need Real Name 2008-11-14 08:11:14 UTC
Created attachment 323548 [details]
panick

I cannot boot this kernel. I did have similar problems with other test kerneles.
This problem go on and off sporadically.

Comment 18 Need Real Name 2008-11-14 08:12:11 UTC
Created attachment 323549 [details]
panick (2)

Comment 19 Need Real Name 2008-11-14 08:13:31 UTC
Created attachment 323550 [details]
panick (3)

Comment 20 Need Real Name 2008-11-14 08:15:07 UTC
Created attachment 323551 [details]
panick(4)

I do not have serial cable handy, so I submit images

Comment 21 Fedora Update System 2008-11-14 11:54:15 UTC
kernel-2.6.27.5-41.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/kernel-2.6.27.5-41.fc9

Comment 22 Need Real Name 2008-11-14 14:53:56 UTC
Created attachment 323584 [details]
kernel-2.6.27.5-41.fc9 panic

The 
Linux comp 2.6.27.5-41.fc9.x86_64 #1 SMP Thu Nov 13 20:29:07 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
has the same problem - periodic panic on boot in mdadm.
There was no such problem in old 2.6.26 kernels.

The strange thing - when I power computer off and then turn on -
it is most likely to boot OK.

When I do just 
shutdown -r now
I in 90% cases get panic on boot (see attached).

Again, there was no such problem in old 2.6.26.

Comment 23 Need Real Name 2008-11-18 07:58:49 UTC
The problem with VPN connection re-establishing still exist in 2.6.27.5-41. Seldom I get this:

VPN key expired.
Connection re-established, message about this printed.
But ping over VPN link does not work.

Restarting racoon establishes link OK.

Comment 24 Fedora Update System 2008-11-19 14:54:40 UTC
kernel-2.6.27.5-41.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 25 Need Real Name 2008-11-21 20:53:36 UTC
There is one more situation when VPN connection breaks.
An intermittent connectivity problem.
Even when TCP link become OK after all, the connection sometimes is not re-established.

Is it possible to review the code and fix all these popping-up 
dead VPN link problem once and for good.
Looking at the number of bugs in this part of code I do not believe 
that bug reports will be of any substantial help.

Only complete complete code review and possible rewrite can make a difference.

Comment 26 Need Real Name 2008-11-21 20:55:30 UTC
Also VPN key expiration sometimes (one in 10-20 times) lead to dead VPN link.

Comment 27 Need Real Name 2008-12-05 21:01:59 UTC
The problem still persists in
 Linux mobile 2.6.27.5-117.fc10.i686 #1 SMP Tue Nov 18 12:19:59 EST 2008 i686 i686 i386 GNU/Linux

ipsec-tools-0.7.1-6.fc10.i386

Comment 28 Need Real Name 2008-12-07 20:56:15 UTC
Everything works extremely poorly. Reboot a router and VPN link Linux client-> router will never be re-established

Comment 29 Need Real Name 2008-12-27 01:58:10 UTC
with 
Linux mobile 2.6.27.9-159.fc10.i686 #1 SMP Tue Dec 16 15:12:04 EST 2008 i686 i686 i386 GNU/Linux
VPN breaks every 5-10 minutes, no debug messages printed.

Comment 30 Need Real Name 2009-01-26 09:03:21 UTC
related bug.
Different problem with VPN
https://bugzilla.redhat.com/show_bug.cgi?id=481529

Comment 31 Need Real Name 2009-07-27 08:21:19 UTC
same problem with
ipsec-tools-0.7.2-2
from 
http://download.fedora.redhat.com/pub/fedora/linux/development/source/SRPMS/

Comment 32 Need Real Name 2009-10-20 13:18:27 UTC
VPN still work poorly in 
kernel-2.6.30.8-64.fc11.x86_64
ipsec-tools-0.7.2-1.fc11.x86_64

I am starting to think about getting rid of IPSEC 
completely and start using openvpn.
Looks like this convoluted IPSEC kernel code 
is impossible to fix at all or at least in reasonable time.

Comment 33 Bug Zapper 2010-04-27 12:17:13 UTC
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 34 Bug Zapper 2010-06-28 10:46:40 UTC
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.