Created attachment 764534 [details] Log of "BUG: unable to handle kernel NULL pointer dereference" Description of problem: The server system provides an ipsec server for Android based smartphones and tablets. It uses racoon together with xl2tpd and ppp. Clients are able to connect and to use the tunnel as intended, but very, very often after a client has disconnected, the server "crashes". No network connection - neither in nor out - is possible anymore and the serial console of the server (it's a kvm guest) isn't reachable. According to the system protocols (i. e. /var/log/messages) the server is still up in some way. Version-Release number of selected component (if applicable): * kernel-3.9.5-201.fc18.x86_64 (with the previous kernel the problem occured too) * ipsec-tools-0.8.0-5.fc18.x86_64 * xl2tpd-1.3.1-13.fc18.x86_64 * ppp-2.4.5-25.fc18.x86_64 How reproducible: Steps to Reproduce: 1. Set up raccon, xl2tp, ppp and iptables 2. Connect to the ipsec server from an Android 4.x based system 3. Disconnect from the server Actual results: Something goes wrong and the server isn't reachable anymore. The log file "/var/log/messages" shows: BUG: unable to handle kernel NULL pointer dereference at 0000000000000020 ... (Full information attached to this report) Expected results: Everything is fine. Additional info: /etc/racoon/racoon.conf ======================== # Racoon IKE daemon configuration file. # See 'man racoon.conf' for a description of the format and entries. path include "/etc/racoon"; path pre_shared_key "/etc/racoon/psk.txt"; path certificate "/etc/racoon/certs"; path script "/etc/racoon/scripts"; log info; timer { natt_keepalive 45 sec; } remote anonymous { exchange_mode aggressive,main; generate_policy on; nat_traversal on; passive on; proposal_check obey; support_proxy on; ike_frag on; dpd_delay 20; proposal { encryption_algorithm 3des; hash_algorithm sha1; authentication_method pre_shared_key; dh_group modp1024; } proposal { encryption_algorithm aes; hash_algorithm sha1; authentication_method pre_shared_key; dh_group modp1024; } } sainfo anonymous { encryption_algorithm 3des,aes; authentication_algorithm hmac_sha1,hmac_md5; compression_algorithm deflate; pfs_group modp1024; } /etc/racoon/setkey.txt ====================== #!/sbin/setkey -f # Flush the SAD and SPD flush; spdflush; # If the client is behind a NAT racoon generates the wrong IPSec policies # (see https://plus.google.com/101331249670615452206/posts/ERnLaSkz1dS for # further information). To fix this issue it's possible to manually load a # policy that forces the encryption of all the communication with the L2TP # daemon # Note: Port l2tp = 1701 spdadd 0.0.0.0/0 193.46.215.21[l2tp] udp -P in ipsec esp/transport//require; spdadd 193.46.215.21[l2tp] 0.0.0.0/0 udp -P out ipsec esp/transport//require; /etc/xl2tpd/xl2tpd.conf ======================= ; ; This is a minimal sample xl2tpd configuration file for use ; with L2TP over IPsec. ; ; IMPORTANT: always set listen-addr to a specific address, to work around a ; udpfromto bug!!! [global] ; listen-addr = 192.168.1.98 listen-addr = 193.46.215.21 ; we have to disable the builtin access control of the daemon since ; the remote IP address is not known access control = no ; ; requires openswan-2.5.18 or higher - Also does not yet work in combination ; with kernel mode l2tp as present in linux 2.6.23+ ; ipsec saref = yes ; Use refinfo of 22 if using an SAref kernel patch based on openswan 2.6.35 or ; when using any of the SAref kernel patches for kernels up to 2.6.35. ; ipsec refinfo = 30 ; ; Required on CentOS 6.3 and higher and Fedora Linux ; force userspace = yes force userspace = yes ; Debugging (you may enable all options in case of problems) ;debug avp = yes ;debug network = yes ;debug state = yes ;debug tunnel = yes ipsec saref = yes [lns default] local ip = 192.168.150.1 ip range = 192.168.150.128-192.168.150.254 name = VPNserver require authentication = yes refuse pap = yes length bit = yes pppoptfile = /etc/ppp/options.xl2tpd ; Debugging ;ppp debug = yes
un 21 16:46:00 rom kernel: [526198.066239] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020 Jun 21 16:46:00 rom kernel: [526198.066239] IP: [<ffffffff815ed433>] xfrm_output_resume+0x2b3/0x400 Jun 21 16:46:00 rom kernel: [526198.066239] PGD 0 Jun 21 16:46:00 rom kernel: [526198.066239] Oops: 0000 [#1] SMP Jun 21 16:46:00 rom kernel: [526198.066239] Modules linked in: ppp_deflate bsd_comp ppp_async crc_ccitt authenc esp4 xfrm4_mode_transport xt_multiport xt_policy xt_recent deflate zlib_deflate twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic camellia_x86_64 serpent_sse2_x86_64 serpent_generic glue_helper blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common nf_nat_ftp nf_conntrack_ftp xt_state des_generic xcbc xt_REDIRECT ipt_MASQUERADE xt_LOG rmd160 xt_limit sha512_generic l2tp_ppp l2tp_netlink l2tp_core pppox ppp_generic crypto_null iptable_nat nf_nat_ipv4 nf_nat af_key iptable_mangle slhc nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack joydev microcode i2c_piix4 virtio_net virtio_balloon uinput binfmt_misc cirrus drm_kms_helper ttm drm i2c_core virtio_blk Jun 21 16:46:00 rom kernel: [526198.066239] CPU 0 Jun 21 16:46:00 rom kernel: [526198.066239] Pid: 30211, comm: xl2tpd Not tainted 3.9.5-201.fc18.x86_64 #1 Bochs Bochs Jun 21 16:46:00 rom kernel: [526198.066239] RIP: 0010:[<ffffffff815ed433>] [<ffffffff815ed433>] xfrm_output_resume+0x2b3/0x400 Jun 21 16:46:00 rom kernel: [526198.066239] RSP: 0018:ffff8800875df908 EFLAGS: 00010246 Jun 21 16:46:00 rom kernel: [526198.066239] RAX: 0000000000000000 RBX: ffff880037dfd200 RCX: 00000001800d0001 Jun 21 16:46:00 rom kernel: [526198.066239] RDX: 00000001800d0002 RSI: 00000000800d0001 RDI: ffff880037dfd200 Jun 21 16:46:00 rom kernel: [526198.066239] RBP: ffff8800875df948 R08: 0000000000000000 R09: 0000000000000001 Jun 21 16:46:00 rom kernel: [526198.066239] R10: 9c44456d02ce113c R11: 0040000049000045 R12: 0000000000000000 Jun 21 16:46:00 rom kernel: [526198.066239] R13: ffff8800858f2c00 R14: ffffffff81cb9e40 R15: ffff8800858f2c3c Jun 21 16:46:00 rom kernel: [526198.066239] FS: 00007f9fb0cf6740(0000) GS:ffff88008ca00000(0000) knlGS:0000000000000000 Jun 21 16:46:00 rom kernel: [526198.066239] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jun 21 16:46:00 rom kernel: [526198.066239] CR2: 0000000000000020 CR3: 000000008890d000 CR4: 00000000000006f0 Jun 21 16:46:00 rom kernel: [526198.066239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 21 16:46:00 rom kernel: [526198.066239] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jun 21 16:46:00 rom kernel: [526198.066239] Process xl2tpd (pid: 30211, threadinfo ffff8800875de000, task ffff880085b90000) Jun 21 16:46:00 rom kernel: [526198.066239] Stack: Jun 21 16:46:00 rom kernel: [526198.066239] ffff88008879f500 0000000000000000 ffff8800875df998 ffffffff81cb9e40 Jun 21 16:46:00 rom kernel: [526198.066239] ffff880037dfd200 0000000000000000 ffff88008879f500 0000000000000035 Jun 21 16:46:00 rom kernel: [526198.066239] ffff8800875df958 ffffffff815ed593 ffff8800875df988 ffffffff815ed5ee Jun 21 16:46:00 rom kernel: [526198.066239] Call Trace: Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff815ed593>] xfrm_output2+0x13/0x20 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff815ed5ee>] xfrm_output+0x4e/0x110 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff815e0f80>] ? xfrm4_extract_output+0xe0/0xe0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff815e0fa7>] xfrm4_output_finish+0x27/0x40 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff815e100b>] xfrm4_output+0x4b/0x80 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff8158f2ca>] ? __ip_local_out+0xaa/0xb0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff8158f2f9>] ip_local_out+0x29/0x30 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff81590629>] ip_send_skb+0x19/0x50 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff815b69ac>] udp_send_skb+0x2cc/0x3a0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff8158d700>] ? ip_copy_metadata+0x1a0/0x1a0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff815b827d>] udp_sendmsg+0x2fd/0xa10 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff811b2600>] ? __pollwait+0xf0/0xf0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff812a2805>] ? sock_has_perm+0x75/0x90 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff811b2602>] ? pollwake+0x2/0x70 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff815c33a3>] inet_sendmsg+0x63/0xb0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff812a2933>] ? selinux_socket_sendmsg+0x23/0x30 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff8153a160>] sock_sendmsg+0xb0/0xe0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff8130c1a1>] ? vsnprintf+0x461/0x640 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff811b9579>] ? evict+0x109/0x1a0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff8153bb6c>] __sys_sendmsg+0x3ac/0x3c0 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff811b33fd>] ? core_sys_select+0x26d/0x320 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff8101e658>] ? __restore_xstate_sig+0x238/0x540 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff81316e81>] ? list_del+0x11/0x40 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff81043daf>] ? kvm_clock_read+0x1f/0x30 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff81043dc9>] ? kvm_clock_get_cycles+0x9/0x10 Jun 21 16:46:00 rom kernel: [526198.066239] [<ffffffff810b18fc>] ? ktime_get_ts+0x4c/0xf0 Jun 21 16:46:01 rom kernel: [526198.066239] [<ffffffff81043daf>] ? kvm_clock_read+0x1f/0x30 Jun 21 16:46:01 rom kernel: [526198.066239] [<ffffffff8153db39>] sys_sendmsg+0x49/0x90 Jun 21 16:46:01 rom kernel: [526198.066239] [<ffffffff8166a2d9>] system_call_fastpath+0x16/0x1b Jun 21 16:46:01 rom kernel: [526198.066239] Code: 00 48 85 ff 74 0f 3e ff 0f 0f 94 c0 84 c0 74 05 e8 e3 93 b9 ff 48 8b 43 58 48 c7 83 98 00 00 00 00 00 00 00 48 89 df 48 83 e0 fe <48> 8b 40 20 ff 50 60 83 f8 01 41 89 c4 0f 85 0c fe ff ff 48 8b Jun 21 16:46:01 rom kernel: [526198.066239] RIP [<ffffffff815ed433>] xfrm_output_resume+0x2b3/0x400 Jun 21 16:46:01 rom kernel: [526198.066239] RSP <ffff8800875df908> Jun 21 16:46:01 rom kernel: [526198.066239] CR2: 0000000000000020 Jun 21 16:46:01 rom kernel: [526198.608716] ---[ end trace 7fdf6fc33051a5b3 ]---
For testing purposes we have set up the same configuration on an other server system running CentOS 6.4. The configuration files of Racoon, Xl2tpd and PPP have been copied from the Fedora based system. On CentOS the vpn is running fine without any problems. CentOS uses the following versions: - kernel-2.6.32-358.6.2.el6.x86_64 - ipsec-tools-0.8.0-3defpsk.el6.x86_64 - xl2tpd-1.3.1-7.el6.x86_64 - ppp-2.4.5-5.el6.x86_64 BTW: We are disappointed about the way the reported bug is handled here. We know that all developers have little time, but this bug kills the network connectivity. Hence it's an important bug IOHO, but for more than three weeks nothing happens here. Just like the previous report (<https://bugzilla.redhat.com/show_bug.cgi?id=601556>).
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 18 kernel bugs. Fedora 18 has now been rebased to 3.11.4-101.fc18. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 19, and are still experiencing this issue, please change the version to Fedora 19. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. It has been over a month since we asked you to test the 3.11 kernel updates and let us know if your issue has been resolved or is still a problem. When this happened, the bug was set to needinfo. Because the needinfo is still set, we assume either this is no longer a problem, or you cannot provide additional information to help us resolve the issue. As a result we are closing with insufficient data. If this is still a problem, we apologize, feel free to reopen the bug and provide more information so that we can work towards a resolution If you experience different issues, please open a new bug report for those.
The problem has been solved with kernel 3.11.4-101.fc18. The vpn is running fine now. Many thanks.