Bug 617523 (CVE-2010-2791)

Summary:

CVE-2010-2791 httpd: Reverse proxy sends wrong responses after time-outs

Product:

[Other] Security Response

Reporter:

Jeremy Sowden <jeremy>

Component:

vulnerability

Assignee:

Red Hat Product Security <security-response-team>

Status:

CLOSED ERRATA

QA Contact:

Severity:

medium

Docs Contact:

Priority:

medium

Version:

unspecified

CC:

jim.redhat, jorton, kieran, puzza007+redhat, vdanen

Target Milestone:

---

Keywords:

Security

Target Release:

---

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2013-05-08 17:28:55 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

623210, 623211

Bug Blocks:

Attachments:

Description	Flags
Demonstration client and server	none

Description Jeremy Sowden 2010-07-23 10:53:33 UTC

Created attachment 433917 [details]
Demonstration client and server

Description of problem:

  If a request forwarded by a reverse proxy to the back-end server times out and
  the proxy is reusing back-end connexions, it may still receive the response
  and send it to a different client making a later, unrelated request.  This can
  result in the leakage of privileged information.

  The code committed to modules/proxy/mod_proxy_http.c in rev. 660936 [0] and
  tagged in 2.2.9 [1] introduced a bug in the case of a proxy time-out such that
  when a reverse proxy attempts to force a retry, ap_proxy_http_process_response
  returns OK and so the connection to the back-end server remains intact,
  allowing the server to send its response.  If the connexion is subsequently
  reused to process another request, the client will receive the response to the
  earlier request that timed out.

  This bug was then introduced into the RHEL5 httpd package by the
  httpd-2.2.3-proxy229.patch patch (in 2.2.3-16.el5 to judge by the change-log),
  which rebases modules/proxy to 2.2.9.

  The bug was fixed upstream (for Unix and Linux) in rev. 699841 [2], which was
  tagged in 2.2.10 [3], and (for Windows, Netware and OS2) rev. 953616 [4],
  which was tagged in 2.2.16 [5].

  Enabling "DisableReuse" in ProxyPass directives works around the bug.

  0 - http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?r1=657443&r2=660936
  1 - http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.9/CHANGES?view=markup
  2 - http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?r1=691230&r2=699841
  3 - http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.10/CHANGES?view=markup
  4 - http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?r1=942878&r2=953616
  5 - http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.16/CHANGES?view=markup

Version-Release number of selected component (if applicable):

  2.2.3-43.el5

How reproducible:

  Send two requests via a reverse proxy, ensuring that the second request times
  out.  A subsequent request via the reverse proxy which reuses the connexion
  used for the second request will receive the second response.

  I have attached a simple Tcl client and server which I have used to reproduce
  the problem.  The client sends a GET request with an ID in the query-string.
  The server waits a geometrically increasing number of milliseconds before
  sending a plain-text response comprising the ID.  Eventually the proxy times
  out.  In the following request, the client receives the wrong response.

Steps to Reproduce:

  1.  Configure the web-server to enable keep-alives and set up a reverse proxy
      with two second time-out:

	KeepAlive On

	ProxyRequests Off

	<Proxy http://localhost:12345/proxy_test>
	  Order deny,allow
	  Allow from all
	</Proxy>

	ProxyPass        /proxy_test http://localhost:12345/proxy_test timeout=2
	ProxyPassReverse /proxy_test http://localhost:12345/proxy_test

  2.  Start the web-server in debug mode to ensure that only one process is
      running and hence that there is only one connexion to the back-end server
      available:

	sudo /usr/sbin/httpd -X

  3.  Start the server:

	./proxy_server.tcl /proxy_test 12345

  4.  Start the client:

	./proxy_client.tcl /proxy_test localhost 80

Actual results:

  The server outputs the following:

    $ ./proxy_server.tcl /proxy_test 12345
    [2010-07-23 11:07:21.116] [1189] accept_conn: /proxy_test sock6 192.168.122.195 58385
    [2010-07-23 11:07:21.118] [1189] read_req(sock6,192.168.122.195,58385): GET /proxy_test?id=04a700000001 HTTP/1.1
    [2010-07-23 11:07:21.119] [1189] read_req(sock6,192.168.122.195,58385): req_id = 04a700000001
    [2010-07-23 11:07:21.119] [1189] read_req(sock6,192.168.122.195,58385): Host: centos5:12345
    [2010-07-23 11:07:21.119] [1189] read_req(sock6,192.168.122.195,58385): Port: 80
    [2010-07-23 11:07:21.119] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-For: 127.0.0.1
    [2010-07-23 11:07:21.120] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Host: localhost
    [2010-07-23 11:07:21.120] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Server: 192.168.122.195
    [2010-07-23 11:07:21.120] [1189] read_req(sock6,192.168.122.195,58385): Connection: Keep-Alive
    [2010-07-23 11:07:21.120] [1189] read_req(sock6,192.168.122.195,58385): request complete, delay = 500.
    [2010-07-23 11:07:21.622] [1189] write_resp(sock6,192.168.122.195,58385): sending 04a700000001
    [2010-07-23 11:07:21.628] [1189] read_req(sock6,192.168.122.195,58385): GET /proxy_test?id=04a700000002 HTTP/1.1
    [2010-07-23 11:07:21.628] [1189] read_req(sock6,192.168.122.195,58385): req_id = 04a700000002
    [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): Host: centos5:12345
    [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): Port: 80
    [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-For: 127.0.0.1
    [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Host: localhost
    [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Server: 192.168.122.195
    [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): Connection: Keep-Alive
    [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): request complete, delay = 1000.
    [2010-07-23 11:07:22.631] [1189] write_resp(sock6,192.168.122.195,58385): sending 04a700000002
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): GET /proxy_test?id=04a700000003 HTTP/1.1
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): req_id = 04a700000003
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): Host: centos5:12345
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): Port: 80
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-For: 127.0.0.1
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Host: localhost
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Server: 192.168.122.195
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): Connection: Keep-Alive
    [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): request complete, delay = 2000.
    [2010-07-23 11:07:24.641] [1189] write_resp(sock6,192.168.122.195,58385): sending 04a700000003
    [2010-07-23 11:07:24.744] [1189] read_req(sock6,192.168.122.195,58385): GET /proxy_test?id=04a700000004 HTTP/1.1
    [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): req_id = 04a700000004
    [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): Host: centos5:12345
    [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): Port: 80
    [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-For: 127.0.0.1
    [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Host: localhost
    [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Server: 192.168.122.195
    [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): Connection: Keep-Alive
    [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): request complete, delay = 4000.
    [2010-07-23 11:07:28.746] [1189] write_resp(sock6,192.168.122.195,58385): sending 04a700000004

  The third response is delayed sufficiently to cause the reverse proxy to time
  out.

  The client outputs the following:

    $ ./proxy_client.tcl /proxy_test localhost 80
    [2010-07-23 11:07:21.099] [1191] connect: /proxy_test localhost 80
    [2010-07-23 11:07:21.114] [1191] write_req(sock5): sending 04a700000001
    [2010-07-23 11:07:21.623] [1191] read_resp(sock5): header = HTTP/1.1 200 OK
    [2010-07-23 11:07:21.623] [1191] read_resp(sock5): header = Date: Fri, 23 Jul 2010 10:07:21 GMT
    [2010-07-23 11:07:21.623] [1191] read_resp(sock5): header = Last-Modified: Wed, 07 Jul 2010 11:07:21 GMT
    [2010-07-23 11:07:21.624] [1191] read_resp(sock5): header = Content-Length: 14
    [2010-07-23 11:07:21.624] [1191] read_resp(sock5): header = Content-Type: text/plain; charset=UTF-8
    [2010-07-23 11:07:21.624] [1191] read_resp(sock5): header = Keep-Alive: timeout=15, max=100
    [2010-07-23 11:07:21.624] [1191] read_resp(sock5): header = Connection: Keep-Alive
    [2010-07-23 11:07:21.626] [1191] read_resp(sock5): received 04a700000001
    [2010-07-23 11:07:21.626] [1191] write_req(sock5): sending 04a700000002
    [2010-07-23 11:07:22.631] [1191] read_resp(sock5): header = HTTP/1.1 200 OK
    [2010-07-23 11:07:22.633] [1191] read_resp(sock5): header = Date: Fri, 23 Jul 2010 10:07:21 GMT
    [2010-07-23 11:07:22.633] [1191] read_resp(sock5): header = Last-Modified: Wed, 07 Jul 2010 11:07:22 GMT
    [2010-07-23 11:07:22.634] [1191] read_resp(sock5): header = Content-Length: 14
    [2010-07-23 11:07:22.634] [1191] read_resp(sock5): header = Content-Type: text/plain; charset=UTF-8
    [2010-07-23 11:07:22.634] [1191] read_resp(sock5): header = Keep-Alive: timeout=15, max=99
    [2010-07-23 11:07:22.634] [1191] read_resp(sock5): header = Connection: Keep-Alive
    [2010-07-23 11:07:22.634] [1191] read_resp(sock5): received 04a700000002
    [2010-07-23 11:07:22.638] [1191] write_req(sock5): sending 04a700000003
    [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = HTTP/1.1 502 Bad Gateway
    [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = Date: Fri, 23 Jul 2010 10:07:22 GMT
    [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = Keep-Alive: timeout=15, max=98
    [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = Connection: Keep-Alive
    [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = Transfer-Encoding: chunked
    [2010-07-23 11:07:24.640] [1191] read_resp(sock5): header = Content-Type: text/html; charset=iso-8859-1
    [2010-07-23 11:07:24.640] [1191] read_resp(sock5): header = Expires: Fri, 23 Jul 2010 10:07:22 GMT
    [2010-07-23 11:07:24.741] [1191] connect: /proxy_test localhost 80
    [2010-07-23 11:07:24.742] [1191] write_req(sock5): sending 04a700000004
    [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = HTTP/1.1 200 OK
    [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Date: Fri, 23 Jul 2010 10:07:24 GMT
    [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Last-Modified: Wed, 07 Jul 2010 11:07:24 GMT
    [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Content-Length: 14
    [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Content-Type: text/plain; charset=UTF-8
    [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Keep-Alive: timeout=15, max=100
    [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Connection: Keep-Alive
    [2010-07-23 11:07:24.743] [1191] read_resp(sock5): received 04a700000003

  On the third request the client receives a 502 response from the reverse
  proxy.  It reconnects, sends the fourth request, but receives the response
  sent by the back-end server to the previous request.

Expected results:

   All requests should receive the same ID in response that they send to the
   server.

Additional info:

Comment 1 Jeremy Sowden 2010-07-23 12:51:01 UTC

(In reply to comment #0)
>   The code committed to modules/proxy/mod_proxy_http.c in rev. 660936
>   [0] and tagged in 2.2.9 [1] introduced a bug in the case of a proxy
>   time-out such that when a reverse proxy attempts to force a retry,
>   ap_proxy_http_process_response returns OK and so the connection to
>   the back-end server remains intact, allowing the server to send its
>   response.

Sorry, got the revision wrong; it should be 657443:

http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?r1=657440&r2=657443

Comment 2 Jeremy Sowden 2010-07-29 15:53:02 UTC

Raising the severity of this bug report to match the importance
ascribed to the related issue in Apache's bugzilla:

https://issues.apache.org/bugzilla/show_bug.cgi?id=49417

Comment 3 Joe Orton 2010-07-30 09:40:22 UTC

Thanks a lot for the report and diagnosis.  

It doesn't look like any security impact of this was noticed upstream.

Comment 4 Vincent Danen 2010-08-11 14:48:03 UTC

Changing this to a security response bug, and giving it its assigned CVE name: CVE-2010-2791 as per:

http://marc.info/?l=apache-httpd-dev&m=128050296121660&w=2

Comment 6 Tomas Hoger 2010-08-20 15:37:50 UTC

Upstream security page lists this issue as only affecting httpd 2.2.9:

  http://httpd.apache.org/security/vulnerabilities_22.html#2.2.10

While httpd version shipped with Red Hat Enterprise Linux 5 is based on 2.2.3, it was affected by this issue due to a rebase of mod_proxy and mod_cache modules to version used in upstream httpd version 2.2.9.  This rebase was done in RHBA-2009:0185, released as part of Red Hat Enterprise Linux 5.3 update:

  https://rhn.redhat.com/errata/RHBA-2009-0185.html

Comment 7 errata-xmlrpc 2010-08-30 12:30:46 UTC

This issue has been addressed in following products:

  Red Hat Enterprise Linux 5

Via RHSA-2010:0659 https://rhn.redhat.com/errata/RHSA-2010-0659.html