Bug 545411 - (CVE-2009-4272) CVE-2009-4272 kernel: emergency route cache flushing leads to node deadlock
CVE-2009-4272 kernel: emergency route cache flushing leads to node deadlock
Status: CLOSED ERRATA
Product: Security Response
Classification: Other
Component: vulnerability (Show other bugs)
unspecified
All Linux
high Severity high
: ---
: ---
Assigned To: Red Hat Product Security
impact=important,reported=20091208,pu...
: Security
Depends On: 545662 545663 553538
Blocks:
  Show dependency treegraph
 
Reported: 2009-12-08 09:15 EST by Konstantin Khorenko
Modified: 2015-07-10 12:20 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-07-10 12:20:49 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
initial revision of a patch (526 bytes, patch)
2009-12-09 06:43 EST, Thomas Graf
no flags Details | Diff
additional patch to fix uninitialized rp issue (1.47 KB, patch)
2009-12-15 09:11 EST, Thomas Graf
no flags Details | Diff

  None (edit)
Description Konstantin Khorenko 2009-12-08 09:15:33 EST
Recently Parallels Virtuozzo Containers team has found a kernel bug that leads to a node freeze.
We were able to get the root of the problem: it is linux-2.6-net-allow-for-on-demand-emergency-route-cache-flushing.patch, added in 2.6.18-164.x kernels (i've checked 164.6.1.el5, it still contains the bug).

The calltrace of the deadlocked process:
...
rt_intern_hash(
 spin_lock_bh(rt_hash_lock_addr(hash));
 rt_emergency_hash_rebuild
  rt_secret_rebuild_oneshot
   rt_cache_flush
    rt_run_flush()
             for (i = rt_hash_mask; i >= 0; i--) {
                spin_lock_bh(rt_hash_lock_addr(i));

deadlock, bh is disabled => all the node quickly becomes "freezed".

We believe this is a security issue, cause this is a remote DOS.

Original raw calltrace:

Route hash chain too long!
Adjust your secret_interval!
SysRq HELP : loglevel0-8 vsced_stAte reBoot Crashdump tErm Full debuG kIll thaw-filesystems(J) saK showMem Nice powerOff showPc unRaw Sync showTasks Unmount shoWcpus
SysRq: Show Regs
----------- IPI show regs -----------
Pid: 23107, comm:              dcpumon, oopses: 0
EIP: 0060:[<0261abf6>] CPU: 3, VCPU: 9665:2
EIP is at _spin_lock_bh+0xf/0x18
 EFLAGS: 00000282    Tainted:  P       (2.6.18-028stab066.8-ent #1 028stab066)
EAX: c00bc000 EBX: d9f25950 ECX: 00000088 EDX: c00bc000
ESI: 00000000 EDI: 00000000 EBP: 00001950 DS: 007b ES: 007b
CR0: 80050033 CR2: b7dfa03c CR3: 00799000 CR4: 000006f0
 [<025d0da4>] rt_run_flush+0x74/0x123
 [<025d1bd9>] rt_intern_hash+0x4a4/0x6f2
 [<025d26bf>] __ip_route_output_key+0x898/0x909
 [<025d2743>] ip_route_output_flow+0x13/0x24f
 [<025de3b9>] inet_csk_route_req+0xb8/0x1c7
 [<025edc9a>] tcp_v4_send_synack+0x17/0x169
 [<025f072d>] tcp_v4_conn_request+0x4cc/0x517
 [<0261ab57>] _read_lock_bh+0x8/0x17
 [<025e7b52>] tcp_rcv_state_process+0x5d/0xd66
 [<025eebb1>] tcp_v4_do_rcv+0x2c3/0x323
 [<025f1781>] tcp_v4_rcv+0x95e/0x9b1
 [<025d60e5>] ip_local_deliver+0x1a6/0x26d
 [<025d5efc>] ip_rcv+0x505/0x548

 [<025b8b7e>] netif_receive_skb+0x3b2/0x3f0
 [<025ba6f7>] process_backlog+0x7e/0xd1
 [<025ba8c6>] net_rx_action+0x9c/0x1a7
 [<024275c4>] __do_softirq+0xe0/0x193
 [<024276ad>] do_softirq+0x36/0x3a
 [<024277c8>] local_bh_enable+0x74/0x7f
 [<025baf37>] dev_queue_xmit+0x34f/0x380
 [<025bf6f4>] neigh_resolve_output+0x1a7/0x1d1
 [<025db407>] ip_output+0x299/0x2c2
 [<025daa33>] ip_queue_xmit+0x4cf/0x515
 [<025d26bf>] __ip_route_output_key+0x898/0x909
 [<025ef380>] tcp_v4_send_check+0x77/0xbd
 [<025e9da6>] tcp_transmit_skb+0x687/0x6b5
 [<025ec3e2>] tcp_connect+0x341/0x3c1
 [<025f0040>] tcp_v4_connect+0x708/0x89f
 [<024690a1>] kunmap_high+0x14/0x84
 [<025fad1d>] inet_stream_connect+0x7d/0x208
 [<02475c7a>] get_user_size+0x30/0x57
 [<025afa9e>] sys_connect+0x6d/0x98
 [<0246cfea>] follow_page_pte+0x173/0x195
 [<02475997>] __kmap_atomic_user_page+0xdb/0xf3
 [<024758a8>] __kunmap_atomic_user_page+0x10/0x24
 [<02475c2f>] rw_vm+0x10d/0x128
 [<025afbe6>] sys_socketcall+0x8e/0x1af
 [<02407400>] do_syscall_trace+0xab/0xb1
 =======================
----------- IPI show regs -----------

--
Best regards,

Konstantin Khorenko,
PVC/OpenVZ developer,
Parallels
Comment 6 Thomas Graf 2009-12-09 06:43:34 EST
Created attachment 377148 [details]
initial revision of a patch

This is how a fix could look like, I have not tested it yet because I was not able to reproduce the issue myself yet.

73e42897e8e5619eacb787d2ce69be12f47cfc21
b6280b47a7a42970d098a3059f4ebe7e55e90d8d
Comment 15 Thomas Graf 2009-12-15 09:11:54 EST
Created attachment 378505 [details]
additional patch to fix uninitialized rp issue

This is a follow up patch based on the upstream commits
73e42897e8e5619eacb787d2ce69be12f47cfc21
b6280b47a7a42970d098a3059f4ebe7e55e90d8d

to address the uninitialized rp issues when rt caching is disabled.
Comment 19 errata-xmlrpc 2010-01-19 18:38:34 EST
This issue has been addressed in following products:

  Red Hat Enterprise Linux 5

Via RHSA-2010:0046 https://rhn.redhat.com/errata/RHSA-2010-0046.html
Comment 22 errata-xmlrpc 2010-02-09 10:24:34 EST
This issue has been addressed in following products:

  Red Hat Enterprise Virtualization for RHEL-5

Via RHSA-2010:0095 https://rhn.redhat.com/errata/RHSA-2010-0095.html

Note You need to log in before you can comment on or make changes to this bug.