Bug 439874 - IPV6DOD: 2.6.18-87.el5 kernel crashes when using ipsec with aes-xcbc-mac and UDP on ppc64.
IPV6DOD: 2.6.18-87.el5 kernel crashes when using ipsec with aes-xcbc-mac and ...
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
ppc64 All
urgent Severity high
: rc
: ---
Assigned To: Thomas Graf
Martin Jenner
Depends On:
Blocks: 253764
  Show dependency treegraph
Reported: 2008-03-31 16:32 EDT by IBM Bug Proxy
Modified: 2014-06-18 04:29 EDT (History)
5 users (show)

See Also:
Fixed In Version: RHBA-2008-0314
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2008-05-21 11:13:48 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Patch for xcbc. (935 bytes, text/plain)
2008-03-31 19:16 EDT, IBM Bug Proxy
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
IBM Linux Technology Center 43640 None None None Never

  None (edit)
Description IBM Bug Proxy 2008-03-31 16:32:21 EDT
=Comment: #0=================================================
Joy M. Latten <latten@us.ibm.com> - 2008-03-30 14:05 EDT
---Problem Description---
The kernel crashes when ipsec is setup to use aes-xcbc-mac algorithms and
running "./netperf -l 60 -H fc00:0:0:105::64 -i 10,2 -I 99,10 -t UDPIPV6_STREAM
-- -m 1472 -s 32768 -S 32768" 
Contact Information = Joy Latten/latten@us.ibm.com
---uname output---
Linux nachos.ltc.austin.ibm.com 2.6.18-87.el5 #1 SMP Tue Mar 25 17:34:09 EDT
2008 ppc64 ppc64 ppc64 GNU/Linux
---Debugger Data---
see above. this appears to ONLY happen on ppc64. Have not tried x86_64. Doesn't
appear to happen on i386.
Machine Type = ppc64 lpar
---System Hang---
Must reboot.
---Steps to Reproduce---

add fc00:0:0:105::35 fc00:0:0:105::64 esp 35590
-m transport
-E 3des-cbc "ipv6readylogo3descbcin01"
-A aes-xcbc-mac "ipv6readaesxin01";

add fc00:0:0:105::64 fc00:0:0:105::35 esp 12360
-m transport
-E 3des-cbc "ipv6readylogo3descbcin01"
-A aes-xcbc-mac "ipv6readaesxin01";

spdadd ::/0 ::/0 icmp6 135,0 -P in none;
spdadd ::/0 ::/0 icmp6 135,0 -P out none;

spdadd -6 fc00:0:0:105::35 fc00:0:0:105::64 any
-P out ipsec

spdadd -6 fc00:0:0:105::64 fc00:0:0:105::35 any
-P in ipsec

---Kernel Component Data--- 
Stack trace output:
  ./netperf -l 60 -H fc00:0:0:105::64 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m
1472 -s 32768 -S 32768
UDPIPV6 UNIDIRECTIONAL SEND TEST to fc00:0:0:105::64 : +/-5.0% @ 99% conf. :
Unable to handle kernel paging request for data at address 0x00000000
Faulting instruction address: 0xc00000000019dc20
cpu 0x0: Vector: 300 (Data Access) at [c000000076406f10]
    pc: c00000000019dc20: .crypto_xcbc_digest_update2+0xf8/0x304
    lr: c00000000019dda0: .crypto_xcbc_digest_update2+0x278/0x304
    sp: c000000076407190
   msr: 8000000000009032
   dar: 0
 dsisr: 40000000
  current = 0xc000000077c412d0
  paca    = 0xc0000000004d4e00
    pid   = 2171, comm = netperf
enter ? for help
0:mon> t
[c000000076407190] c00000000019dd70 .crypto_xcbc_digest_update2+0x248/0x304
[c000000076407270] c00000000019ebb4 .crypto_authenc_hash+0xf8/0x1ac
[c000000076407330] c00000000019edc8 .crypto_authenc_genicv+0x160/0x1cc
[c000000076407440] d000000000830f94 .esp6_output+0x424/0x498 [esp6]
[c000000076407530] d0000000008d670c .xfrm6_output_finish2+0x2a4/0x3f8 [ipv6]
[c0000000764075d0] d0000000008d69b8 .xfrm6_output+0x64/0x78 [ipv6]
[c000000076407650] d0000000008a41e8 .ip6_push_pending_frames+0x56c/0x678 [ipv6]
[c000000076407720] d0000000008bc800 .udp_v6_push_pending_frames+0x368/0x3b4 [ipv6]
[c0000000764077c0] d0000000008bea7c .udpv6_sendmsg+0x73c/0x9cc [ipv6]
[c000000076407970] c0000000003915c0 .inet_sendmsg+0x7c/0xa8
[c000000076407a10] c0000000003232f4 .sock_sendmsg+0x114/0x15c
[c000000076407c10] c0000000003254dc .sys_sendto+0xec/0x13c
[c000000076407d90] c000000000348060 .compat_sys_socketcall+0x148/0x214
[c000000076407e30] c0000000000086a4 syscall_exit+0x0/0x40
--- Exception: c00 (System Call) at 000000000ff22fa4
SP (ff9ff880) is in userspace

This happened while running stress tests between ppc64 and i386.
Stress tests consist of alternately sending streams of tcp and udp packets using
netperf. I pretty much took the tcp_stream_script and udp_stream_script in
netperf and modified them, TCP with TCPIPV6 and UDP with UDPIPV6 to send v6
packets. Have always done this in past to stress kernel.

Always, during udp, crash occurs.
UDP packets are sent in 3 ways/tests in netperf.
So far, always crashes on 3rd one.

1. ./netperf -l 60 -H fc00:0:0:105::35 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m
64 -s 32768 -S 32768

2. ./netperf -l 60 -H fc00:0:0:105::35 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m
1024 -s 32768 -S 32768

3. ./netperf -l 60 -H fc00:0:0:105::35 -i 10,2 -I 99,10 -t UDPIPV6_STREAM -- -m
1472 -s 32768 -S 32768

Using an old version of netperf and probably should update to more recent
version... but this should not matter.

The stack trace indicates following:

(gdb) l *0xc00000000019dc20
0xc00000000019dc20 is in crypto_xcbc_digest_update2 (include/linux/mmzone.h:581).
576     #define SECTION_MAP_MASK        (~(SECTION_MAP_LAST_BIT-1))
577     #define SECTION_NID_SHIFT       2
579     static inline struct page *__section_mem_map_addr(struct mem_section
580     {
581             unsigned long map = section->section_mem_map;
582             map &= SECTION_MAP_MASK;
583             return (struct page *)map;
584     }
(gdb) l*0xc00000000019dda0
0xc00000000019dda0 is in crypto_xcbc_digest_update2 (crypto/xcbc.c:176).
171                                     len -= bs;
172                             }
174                             /* keeping the surplus of blocksize */
175                             if (len) {
176                                     memcpy(ctx->odds, p, len);
177                                     ctx->len = len;
178                             }
179                             ncrypto_kunmap(p, 0);
180                             ncrypto_yield(pdesc->flags);

Not sure why this only seems to occur when sending udpipv6 packets and with "-m
1472" in netperf on ppc64? 

ok, looks like my i386 may have crashed while trying this out with v4.
Comment 1 IBM Bug Proxy 2008-03-31 17:25:22 EDT
------- Comment From latten@us.ibm.com 2008-03-31 17:23 EDT-------
I think I know why xcbc crashes.
esp6_output gives it data such that nbytes=1496
and slen=1440 first time in do loop in crypto_xcbc_digest_update2().

The first 1440 bytes are in first sg enrty.
The last 56 bits are in next sg entry, only thing is we do not
check to see if its length is zero and if so go look it up.

I noticed in hmac.c, we do a scatterwalk_sg_next() which does this for us.

Thus I changed xcbc to use scatterwalk_sg_next() like hmac.c

I think this may be broken upstream too. Checking upstream right now.

Will run an ipsec stress test overnight using aes-xcbc-mac with fix and also
include aes-ctr, since these are both new algorithms.
Comment 2 IBM Bug Proxy 2008-03-31 19:16:45 EDT
Created attachment 299788 [details]
Patch for xcbc.

OK, this is a problem in upstream kernel too.

I have patched an upstream kernel and RHEL5.2 87 kernel with following patch
and will run a stress tests a couple of hours and see what happens.

Red Hat guys, please let me know if patch looks ok.
Will also post upstream once stress test completes successfully.
Comment 3 Herbert Xu 2008-04-01 08:29:04 EDT
Looks good to me.
Comment 4 IBM Bug Proxy 2008-04-01 11:25:14 EDT
------- Comment From latten@us.ibm.com 2008-04-01 11:23 EDT-------
The stress test has run for 15 hours with no problems!
The test ran between an upstream kernel with patch and
a RHEL5.2 kernel (87 from dzickus) with patch sending streams
of tcp and udp packets.

Will also submit this patch upstream.
Comment 8 Don Zickus 2008-04-09 14:44:47 EDT
in kernel-2.6.18-89.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5
Comment 11 IBM Bug Proxy 2008-04-23 17:24:31 EDT
------- Comment From latten@us.ibm.com 2008-04-23 17:20 EDT-------
This verified successfully in the 90 kernel.
Comment 13 errata-xmlrpc 2008-05-21 11:13:48 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.