=Comment: #0================================================= Joy M. Latten <latten.com> - 2008-03-30 14:05 EDT ---Problem Description--- The kernel crashes when ipsec is setup to use aes-xcbc-mac algorithms and running "./netperf -l 60 -H fc00:0:0:105::64 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m 1472 -s 32768 -S 32768" Contact Information = Joy Latten/latten.com ---uname output--- Linux nachos.ltc.austin.ibm.com 2.6.18-87.el5 #1 SMP Tue Mar 25 17:34:09 EDT 2008 ppc64 ppc64 ppc64 GNU/Linux ---Debugger Data--- see above. this appears to ONLY happen on ppc64. Have not tried x86_64. Doesn't appear to happen on i386. Machine Type = ppc64 lpar ---System Hang--- Must reboot. ---Steps to Reproduce--- spdflush; flush; add fc00:0:0:105::35 fc00:0:0:105::64 esp 35590 -m transport -E 3des-cbc "ipv6readylogo3descbcin01" -A aes-xcbc-mac "ipv6readaesxin01"; add fc00:0:0:105::64 fc00:0:0:105::35 esp 12360 -m transport -E 3des-cbc "ipv6readylogo3descbcin01" -A aes-xcbc-mac "ipv6readaesxin01"; spdadd ::/0 ::/0 icmp6 135,0 -P in none; spdadd ::/0 ::/0 icmp6 135,0 -P out none; spdadd -6 fc00:0:0:105::35 fc00:0:0:105::64 any -P out ipsec esp/transport//require; spdadd -6 fc00:0:0:105::64 fc00:0:0:105::35 any -P in ipsec esp/transport//require; ---Kernel Component Data--- Stack trace output: ./netperf -l 60 -H fc00:0:0:105::64 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m 1472 -s 32768 -S 32768 UDPIPV6 UNIDIRECTIONAL SEND TEST to fc00:0:0:105::64 : +/-5.0% @ 99% conf. : interval Unable to handle kernel paging request for data at address 0x00000000 Faulting instruction address: 0xc00000000019dc20 cpu 0x0: Vector: 300 (Data Access) at [c000000076406f10] pc: c00000000019dc20: .crypto_xcbc_digest_update2+0xf8/0x304 lr: c00000000019dda0: .crypto_xcbc_digest_update2+0x278/0x304 sp: c000000076407190 msr: 8000000000009032 dar: 0 dsisr: 40000000 current = 0xc000000077c412d0 paca = 0xc0000000004d4e00 pid = 2171, comm = netperf enter ? for help 0:mon> 0:mon> t [c000000076407190] c00000000019dd70 .crypto_xcbc_digest_update2+0x248/0x304 (unreliable) [c000000076407270] c00000000019ebb4 .crypto_authenc_hash+0xf8/0x1ac [c000000076407330] c00000000019edc8 .crypto_authenc_genicv+0x160/0x1cc [c000000076407440] d000000000830f94 .esp6_output+0x424/0x498 [esp6] [c000000076407530] d0000000008d670c .xfrm6_output_finish2+0x2a4/0x3f8 [ipv6] [c0000000764075d0] d0000000008d69b8 .xfrm6_output+0x64/0x78 [ipv6] [c000000076407650] d0000000008a41e8 .ip6_push_pending_frames+0x56c/0x678 [ipv6] [c000000076407720] d0000000008bc800 .udp_v6_push_pending_frames+0x368/0x3b4 [ipv6] [c0000000764077c0] d0000000008bea7c .udpv6_sendmsg+0x73c/0x9cc [ipv6] [c000000076407970] c0000000003915c0 .inet_sendmsg+0x7c/0xa8 [c000000076407a10] c0000000003232f4 .sock_sendmsg+0x114/0x15c [c000000076407c10] c0000000003254dc .sys_sendto+0xec/0x13c [c000000076407d90] c000000000348060 .compat_sys_socketcall+0x148/0x214 [c000000076407e30] c0000000000086a4 syscall_exit+0x0/0x40 --- Exception: c00 (System Call) at 000000000ff22fa4 SP (ff9ff880) is in userspace 0:mon> This happened while running stress tests between ppc64 and i386. Stress tests consist of alternately sending streams of tcp and udp packets using netperf. I pretty much took the tcp_stream_script and udp_stream_script in netperf and modified them, TCP with TCPIPV6 and UDP with UDPIPV6 to send v6 packets. Have always done this in past to stress kernel. Always, during udp, crash occurs. UDP packets are sent in 3 ways/tests in netperf. So far, always crashes on 3rd one. 1. ./netperf -l 60 -H fc00:0:0:105::35 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m 64 -s 32768 -S 32768 2. ./netperf -l 60 -H fc00:0:0:105::35 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m 1024 -s 32768 -S 32768 3. ./netperf -l 60 -H fc00:0:0:105::35 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m 1472 -s 32768 -S 32768 Using an old version of netperf and probably should update to more recent version... but this should not matter. The stack trace indicates following: (gdb) l *0xc00000000019dc20 0xc00000000019dc20 is in crypto_xcbc_digest_update2 (include/linux/mmzone.h:581). 576 #define SECTION_MAP_MASK (~(SECTION_MAP_LAST_BIT-1)) 577 #define SECTION_NID_SHIFT 2 578 579 static inline struct page *__section_mem_map_addr(struct mem_section *section) 580 { 581 unsigned long map = section->section_mem_map; 582 map &= SECTION_MAP_MASK; 583 return (struct page *)map; 584 } 585 (gdb) l*0xc00000000019dda0 0xc00000000019dda0 is in crypto_xcbc_digest_update2 (crypto/xcbc.c:176). 171 len -= bs; 172 } 173 174 /* keeping the surplus of blocksize */ 175 if (len) { 176 memcpy(ctx->odds, p, len); 177 ctx->len = len; 178 } 179 ncrypto_kunmap(p, 0); 180 ncrypto_yield(pdesc->flags); (gdb) Not sure why this only seems to occur when sending udpipv6 packets and with "-m 1472" in netperf on ppc64? ok, looks like my i386 may have crashed while trying this out with v4.
------- Comment From latten.com 2008-03-31 17:23 EDT------- I think I know why xcbc crashes. esp6_output gives it data such that nbytes=1496 and slen=1440 first time in do loop in crypto_xcbc_digest_update2(). The first 1440 bytes are in first sg enrty. The last 56 bits are in next sg entry, only thing is we do not check to see if its length is zero and if so go look it up. I noticed in hmac.c, we do a scatterwalk_sg_next() which does this for us. Thus I changed xcbc to use scatterwalk_sg_next() like hmac.c I think this may be broken upstream too. Checking upstream right now. Will run an ipsec stress test overnight using aes-xcbc-mac with fix and also include aes-ctr, since these are both new algorithms.
Created attachment 299788 [details] Patch for xcbc. OK, this is a problem in upstream kernel too. I have patched an upstream kernel and RHEL5.2 87 kernel with following patch and will run a stress tests a couple of hours and see what happens. Red Hat guys, please let me know if patch looks ok. Will also post upstream once stress test completes successfully.
Looks good to me.
------- Comment From latten.com 2008-04-01 11:23 EDT------- The stress test has run for 15 hours with no problems! The test ran between an upstream kernel with patch and a RHEL5.2 kernel (87 from dzickus) with patch sending streams of tcp and udp packets. Will also submit this patch upstream.
in kernel-2.6.18-89.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5
------- Comment From latten.com 2008-04-23 17:20 EDT------- This verified successfully in the 90 kernel.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2008-0314.html