Description of problem: In lspp60 kernel, when regular ipsec is configured to use ipv6, my ping6 to remote works for a few seconds and then stops. I can no longer reach remote via ipv6. I tried this using nc and netperf with same results. Works for about a minute and then stops working and remote can no longer be contacted. As soon as ipsec config is removed, I can then successfully contact remote over ipv6. Same behavior in lspp58 and 59 kernel. How reproducible: Happens all the time on lspp kernel. Steps to Reproduce: 1. Configure ipv6 addresses on 2 machines. ifconfig eth0 inet6 add fec0:0:0:105::64/64 on machine A ifconfig eth0 inet6 add fec0:0:0:105::35/64 on machine B 2. On machine B, create file named setkey.v6 containing: flush; spdflush; add -6 fec0:0:0:105::64 fec0:0:0:105::35 esp 45764 -m transport -E 3des-cbc "06183223c23a21e8b36c566b"; add -6 fec0:0:0:105::35 fec0:0:0:105::64 esp 84758 -m transport -E 3des-cbc "06183223c23a21e8b36c566b"; spdadd -6 fec0:0:0:105::35 fec0:0:0:105::64 any -P in ipsec esp/transport//require; spdadd -6 fec0:0:0:105::64 fec0:0:0:105::35 any -P out ipsec esp/transport//require; 3. Issue setkey -f setkey.v6 to configure ipsecv6. 4. Do steps 2 & 3 on Machine B. NOTE: remember to swap the "in and out" in the spdadd entry. 5. now ping6 one of the ipv6 addresses. it will work for a few seconds, maybe even a minute and then stop. When it stops, get out of the ping6, and then issue it again. It hangs. You can try this with nc too. 6. remove ipsec config setkey -F; setkey -FP 7. now try the ping6 after removing ipsecv6 config. It works for as long as you want. Actual results: ping6 hangs after a few seconds. Expected results: ping6 should run for as long as you want it to. Additional info: I tried this on an upstream kernel from kernel.org (2.6.20-rc1-git6 I think it was) and regular ipsecv6 worked. I could not get labeled ipsecv6 to work though. Will try again on kernel 2.6.20-rc3-git4 from kernel.org. I am using power5 lpars.
Where do I get a copy of the source of this kernel? Please get a stack backtrace while ping6 is hung. You can do it with CTRL-SCROLLLOCK or magic SysRq.
The LSPP kernels can be found here: http://people.redhat.com/sgrubb/files/lspp We have them in cvs, too.
Thanks, I've got the source. Now I just need that backtrace :)
The lspp 62 kernel with the patch from RIT 108432 appeared to solve some of the ipv6 problems, I was able to run regular ipsec ok with a manual set of policies and SAs. However, ipv6 has problems using racoon and we MUST use racoon for labeled ipsec. From some investigation, it appears the following happens when using racoon with ipv6. configure regular ipsec by adding policy and starting racoon. from machine A, send a ping6 to machine B. here is the tcpdump from machine B, [root@hvracer1 racoon]# tcpdump -i eth0 -f "host fc00:0:0:105::35" tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 10:47:28.697109 IP6 fc00:0:0:105::35 > ff02::1:ff00:64: ICMP6, neighbor solicitation, who has fc00:0:0:105::64, length 32 10:47:28.699480 IP6 fc00:0:0:105::64.isakmp > fc00:0:0:105::35.isakmp: isakmp: phase 1 I ident 10:47:29.697062 IP6 fc00:0:0:105::35 > ff02::1:ff00:64: ICMP6, neighbor solicitation, who has fc00:0:0:105::64, length 32 10:47:30.696984 IP6 fc00:0:0:105::35 > ff02::1:ff00:64: ICMP6, neighbor solicitation, who has fc00:0:0:105::64, length 32 10:47:33.699903 IP6 fe80::3c61:60ff:fe00:2002 > fc00:0:0:105::35: ICMP6, neighbor solicitation, who has fc00:0:0:105::35, length 32 10:47:33.700211 IP6 fc00:0:0:105::35 > fe80::3c61:60ff:fe00:2002: ICMP6, neighbor advertisement, tgt is fc00:0:0:105::35, length 24 10:47:38.739213 IP6 fc00:0:0:105::64.isakmp > fc00:0:0:105::35.isakmp: isakmp: phase 1 I ident 10:47:48.740210 IP6 fc00:0:0:105::64.isakmp > fc00:0:0:105::35.isakmp: isakmp: phase 1 I ident From this dump, it appears that a ping6 from Machine A to Machine B, results in 2 things happening: 1. kernel sends an acquire to racoon who then begins negotiations to B as the initiator. 2. kernel also puts a neighbor discovery packet on link, which B picks up. NOTE that the isakmp packet (negotiation packet) from racoon on machine A NEVER arrives on machine B. A tcpdump on machine A confirmed that this packet NEVER gets sent out. I don't know a lot about ipv6, but I THINK machine A's isakmp never gets sent out because of the neighbor discovery packet that is sent out instead. Machine B, picks up the neighbor discovery packet and from what I can tell from above tcpdump and racoon output, tries to begin negotiations with machine A, but as the initiator, not the responder. Remember, it never got machine A's isakmp packet. Thus the two racoons become confused believing each to be the initiator and the negotiations eventually time out resulting in NO SAs being established. I am not an ipv6 guru so I am wondering WHY does machine B picking up the neighbor discovery packet result in an acquire message being sent to machine B's racoon? Should a neighbor discovery packet result in racoon negotiations? I saw similar behaviour in the latest kernel.org kernel, 2.6.20-rc5. A colleague suggested I bypass icmpv6 traffic in racoon. So, I set up my ipsec policy using "icmp6 135,0" to bypass ONLY neighbor solicitations. I then used "tcp" or "udp" instead of "any" in the rest of the policy. i.e. spdadd ::/0 ::/0 icmp6 135,0 -P in none; spdadd ::/0 ::/0 icmp6 135,0 -P out none; This worked. I was able to get racoon to successfully establish SAs and my tcp apps talked to each other. Need to figure out how to specify policy to bypass neighbor solicitation packets and apply ipsec to all other icmp6 packets. Also, need to find out if this is acceptable for lspp. Also, I am still wondering do we ever want ipsec applied to neighbor solicitation packets?
I am assuming it is the neighbor solicitation packet that causes the acquire on machine B because it is the only packet I see being sent or received with an ipv6 address in the tcpdump.
oh, I forgot to also mention that I changed my ipv6 addresses from "fec0:0:..." to "fc00:0:.." in my config files and on my interfaces since fec0 addresses have been deprecated and want a scopeid. Thought I would mention in case anyone tries to replicate my setup.
With a config that specified bypassing ipsec for icmp neighbor discovery packets, things appeared to work fine for ipsec over ipv6. I was able to succesfully complete stress tests this weekend using labeled ipsec over ipv6 and regular ipsec over ipv6 in the lspp63 kernel. I usually verify my packets on the wire were "ipsec'd" but can do so later when tcpdump is working.
So we are saying this bug is fixed?
Yes. But I would like to try it one more time in a kernel that contains the fix for tcpdump so I can verify that the ipv6 packets on the wire or "ipsec'd".
Ok, let me know what you get the chance to try kernel-2.6.18-6.el5.lspp.64 from people.redhat.com/sgrubb/files/lspp
ran a 15 hour stress tests for ipv6 with labeled ipsec on 64 kernel and all completed very well. tcpdump verified my packets were being secured.