Created attachment 433917 [details] Demonstration client and server Description of problem: If a request forwarded by a reverse proxy to the back-end server times out and the proxy is reusing back-end connexions, it may still receive the response and send it to a different client making a later, unrelated request. This can result in the leakage of privileged information. The code committed to modules/proxy/mod_proxy_http.c in rev. 660936 [0] and tagged in 2.2.9 [1] introduced a bug in the case of a proxy time-out such that when a reverse proxy attempts to force a retry, ap_proxy_http_process_response returns OK and so the connection to the back-end server remains intact, allowing the server to send its response. If the connexion is subsequently reused to process another request, the client will receive the response to the earlier request that timed out. This bug was then introduced into the RHEL5 httpd package by the httpd-2.2.3-proxy229.patch patch (in 2.2.3-16.el5 to judge by the change-log), which rebases modules/proxy to 2.2.9. The bug was fixed upstream (for Unix and Linux) in rev. 699841 [2], which was tagged in 2.2.10 [3], and (for Windows, Netware and OS2) rev. 953616 [4], which was tagged in 2.2.16 [5]. Enabling "DisableReuse" in ProxyPass directives works around the bug. 0 - http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?r1=657443&r2=660936 1 - http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.9/CHANGES?view=markup 2 - http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?r1=691230&r2=699841 3 - http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.10/CHANGES?view=markup 4 - http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?r1=942878&r2=953616 5 - http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.16/CHANGES?view=markup Version-Release number of selected component (if applicable): 2.2.3-43.el5 How reproducible: Send two requests via a reverse proxy, ensuring that the second request times out. A subsequent request via the reverse proxy which reuses the connexion used for the second request will receive the second response. I have attached a simple Tcl client and server which I have used to reproduce the problem. The client sends a GET request with an ID in the query-string. The server waits a geometrically increasing number of milliseconds before sending a plain-text response comprising the ID. Eventually the proxy times out. In the following request, the client receives the wrong response. Steps to Reproduce: 1. Configure the web-server to enable keep-alives and set up a reverse proxy with two second time-out: KeepAlive On ProxyRequests Off <Proxy http://localhost:12345/proxy_test> Order deny,allow Allow from all </Proxy> ProxyPass /proxy_test http://localhost:12345/proxy_test timeout=2 ProxyPassReverse /proxy_test http://localhost:12345/proxy_test 2. Start the web-server in debug mode to ensure that only one process is running and hence that there is only one connexion to the back-end server available: sudo /usr/sbin/httpd -X 3. Start the server: ./proxy_server.tcl /proxy_test 12345 4. Start the client: ./proxy_client.tcl /proxy_test localhost 80 Actual results: The server outputs the following: $ ./proxy_server.tcl /proxy_test 12345 [2010-07-23 11:07:21.116] [1189] accept_conn: /proxy_test sock6 192.168.122.195 58385 [2010-07-23 11:07:21.118] [1189] read_req(sock6,192.168.122.195,58385): GET /proxy_test?id=04a700000001 HTTP/1.1 [2010-07-23 11:07:21.119] [1189] read_req(sock6,192.168.122.195,58385): req_id = 04a700000001 [2010-07-23 11:07:21.119] [1189] read_req(sock6,192.168.122.195,58385): Host: centos5:12345 [2010-07-23 11:07:21.119] [1189] read_req(sock6,192.168.122.195,58385): Port: 80 [2010-07-23 11:07:21.119] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-For: 127.0.0.1 [2010-07-23 11:07:21.120] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Host: localhost [2010-07-23 11:07:21.120] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Server: 192.168.122.195 [2010-07-23 11:07:21.120] [1189] read_req(sock6,192.168.122.195,58385): Connection: Keep-Alive [2010-07-23 11:07:21.120] [1189] read_req(sock6,192.168.122.195,58385): request complete, delay = 500. [2010-07-23 11:07:21.622] [1189] write_resp(sock6,192.168.122.195,58385): sending 04a700000001 [2010-07-23 11:07:21.628] [1189] read_req(sock6,192.168.122.195,58385): GET /proxy_test?id=04a700000002 HTTP/1.1 [2010-07-23 11:07:21.628] [1189] read_req(sock6,192.168.122.195,58385): req_id = 04a700000002 [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): Host: centos5:12345 [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): Port: 80 [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-For: 127.0.0.1 [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Host: localhost [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Server: 192.168.122.195 [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): Connection: Keep-Alive [2010-07-23 11:07:21.629] [1189] read_req(sock6,192.168.122.195,58385): request complete, delay = 1000. [2010-07-23 11:07:22.631] [1189] write_resp(sock6,192.168.122.195,58385): sending 04a700000002 [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): GET /proxy_test?id=04a700000003 HTTP/1.1 [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): req_id = 04a700000003 [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): Host: centos5:12345 [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): Port: 80 [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-For: 127.0.0.1 [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Host: localhost [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Server: 192.168.122.195 [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): Connection: Keep-Alive [2010-07-23 11:07:22.639] [1189] read_req(sock6,192.168.122.195,58385): request complete, delay = 2000. [2010-07-23 11:07:24.641] [1189] write_resp(sock6,192.168.122.195,58385): sending 04a700000003 [2010-07-23 11:07:24.744] [1189] read_req(sock6,192.168.122.195,58385): GET /proxy_test?id=04a700000004 HTTP/1.1 [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): req_id = 04a700000004 [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): Host: centos5:12345 [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): Port: 80 [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-For: 127.0.0.1 [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Host: localhost [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): X-Forwarded-Server: 192.168.122.195 [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): Connection: Keep-Alive [2010-07-23 11:07:24.745] [1189] read_req(sock6,192.168.122.195,58385): request complete, delay = 4000. [2010-07-23 11:07:28.746] [1189] write_resp(sock6,192.168.122.195,58385): sending 04a700000004 The third response is delayed sufficiently to cause the reverse proxy to time out. The client outputs the following: $ ./proxy_client.tcl /proxy_test localhost 80 [2010-07-23 11:07:21.099] [1191] connect: /proxy_test localhost 80 [2010-07-23 11:07:21.114] [1191] write_req(sock5): sending 04a700000001 [2010-07-23 11:07:21.623] [1191] read_resp(sock5): header = HTTP/1.1 200 OK [2010-07-23 11:07:21.623] [1191] read_resp(sock5): header = Date: Fri, 23 Jul 2010 10:07:21 GMT [2010-07-23 11:07:21.623] [1191] read_resp(sock5): header = Last-Modified: Wed, 07 Jul 2010 11:07:21 GMT [2010-07-23 11:07:21.624] [1191] read_resp(sock5): header = Content-Length: 14 [2010-07-23 11:07:21.624] [1191] read_resp(sock5): header = Content-Type: text/plain; charset=UTF-8 [2010-07-23 11:07:21.624] [1191] read_resp(sock5): header = Keep-Alive: timeout=15, max=100 [2010-07-23 11:07:21.624] [1191] read_resp(sock5): header = Connection: Keep-Alive [2010-07-23 11:07:21.626] [1191] read_resp(sock5): received 04a700000001 [2010-07-23 11:07:21.626] [1191] write_req(sock5): sending 04a700000002 [2010-07-23 11:07:22.631] [1191] read_resp(sock5): header = HTTP/1.1 200 OK [2010-07-23 11:07:22.633] [1191] read_resp(sock5): header = Date: Fri, 23 Jul 2010 10:07:21 GMT [2010-07-23 11:07:22.633] [1191] read_resp(sock5): header = Last-Modified: Wed, 07 Jul 2010 11:07:22 GMT [2010-07-23 11:07:22.634] [1191] read_resp(sock5): header = Content-Length: 14 [2010-07-23 11:07:22.634] [1191] read_resp(sock5): header = Content-Type: text/plain; charset=UTF-8 [2010-07-23 11:07:22.634] [1191] read_resp(sock5): header = Keep-Alive: timeout=15, max=99 [2010-07-23 11:07:22.634] [1191] read_resp(sock5): header = Connection: Keep-Alive [2010-07-23 11:07:22.634] [1191] read_resp(sock5): received 04a700000002 [2010-07-23 11:07:22.638] [1191] write_req(sock5): sending 04a700000003 [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = HTTP/1.1 502 Bad Gateway [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = Date: Fri, 23 Jul 2010 10:07:22 GMT [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = Keep-Alive: timeout=15, max=98 [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = Connection: Keep-Alive [2010-07-23 11:07:24.639] [1191] read_resp(sock5): header = Transfer-Encoding: chunked [2010-07-23 11:07:24.640] [1191] read_resp(sock5): header = Content-Type: text/html; charset=iso-8859-1 [2010-07-23 11:07:24.640] [1191] read_resp(sock5): header = Expires: Fri, 23 Jul 2010 10:07:22 GMT [2010-07-23 11:07:24.741] [1191] connect: /proxy_test localhost 80 [2010-07-23 11:07:24.742] [1191] write_req(sock5): sending 04a700000004 [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = HTTP/1.1 200 OK [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Date: Fri, 23 Jul 2010 10:07:24 GMT [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Last-Modified: Wed, 07 Jul 2010 11:07:24 GMT [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Content-Length: 14 [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Content-Type: text/plain; charset=UTF-8 [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Keep-Alive: timeout=15, max=100 [2010-07-23 11:07:24.743] [1191] read_resp(sock5): header = Connection: Keep-Alive [2010-07-23 11:07:24.743] [1191] read_resp(sock5): received 04a700000003 On the third request the client receives a 502 response from the reverse proxy. It reconnects, sends the fourth request, but receives the response sent by the back-end server to the previous request. Expected results: All requests should receive the same ID in response that they send to the server. Additional info:
(In reply to comment #0) > The code committed to modules/proxy/mod_proxy_http.c in rev. 660936 > [0] and tagged in 2.2.9 [1] introduced a bug in the case of a proxy > time-out such that when a reverse proxy attempts to force a retry, > ap_proxy_http_process_response returns OK and so the connection to > the back-end server remains intact, allowing the server to send its > response. Sorry, got the revision wrong; it should be 657443: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy_http.c?r1=657440&r2=657443
Raising the severity of this bug report to match the importance ascribed to the related issue in Apache's bugzilla: https://issues.apache.org/bugzilla/show_bug.cgi?id=49417
Thanks a lot for the report and diagnosis. It doesn't look like any security impact of this was noticed upstream.
Changing this to a security response bug, and giving it its assigned CVE name: CVE-2010-2791 as per: http://marc.info/?l=apache-httpd-dev&m=128050296121660&w=2
Upstream security page lists this issue as only affecting httpd 2.2.9: http://httpd.apache.org/security/vulnerabilities_22.html#2.2.10 While httpd version shipped with Red Hat Enterprise Linux 5 is based on 2.2.3, it was affected by this issue due to a rebase of mod_proxy and mod_cache modules to version used in upstream httpd version 2.2.9. This rebase was done in RHBA-2009:0185, released as part of Red Hat Enterprise Linux 5.3 update: https://rhn.redhat.com/errata/RHBA-2009-0185.html
This issue has been addressed in following products: Red Hat Enterprise Linux 5 Via RHSA-2010:0659 https://rhn.redhat.com/errata/RHSA-2010-0659.html