Bug 1771847

Summary: BalancerMember ping parameter for mod_proxy_http doesn't work
Product: Red Hat Enterprise Linux 8 Reporter: Muneaki Sugaya <msugaya>
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED ERRATA QA Contact: Branislav Náter <bnater>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.1CC: bnater, jorton, luhliari, nbhumkar, pasik
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-04 03:35:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Patch fixing the issue none

Description Muneaki Sugaya 2019-11-13 06:47:02 UTC 
Description of problem:
BalancerMember ping parameter for mod_proxy_http doesn't work.

ping description in "Worker|BalancerMember parameters" says "For HTTP, it causes mod_proxy_http to send a 100-Continue to the backend...".

ref. https://httpd.apache.org/docs/2.4/en/mod/mod_proxy.html#proxypass


Version-Release number of selected component (if applicable):
httpd-2.4.37-16.module+el8.1.0+4134+e6bad0ed.x86_64

How reproducible:
Always

Steps to Reproduce:

[Sample settings]

1. Reverse proxy server 

hostname: rhel8-web1
IP: 192.168.122.47
OS: RHEL8.1
RPM: httpd-2.4.37-16.module+el8.1.0+4134+e6bad0ed.x86_64

Add the following in /etc/httpd/conf/httpd.conf
~~~
ProxyPass /sample/ balancer://cluster/
<Proxy balancer://cluster>
  BalancerMember http://web1:80 ping=5
  BalancerMember http://web2:80 ping=5
</Proxy>
~~~


2. Backend server

hostname: web1
IP: 192.168.122.134

Make /var/www/cgi-bin/test.cgi as follows.
~~~
#!/usr/bin/perl
print "Content-type: text/plain\r\n";
print "\r\n";
read (STDIN, $data, $ENV{'CONTENT_LENGTH'});
~~~

Run tcpdump as follows.
~~~
# tcpdump -i ens3 -nnv port 80
~~~


3. Client

Run curl as follows.
~~~
# curl -X POST --data-binary '@/etc/hosts' http://rhel8-web1/sample/cgi-bin/test.cgi
~~~

Actual results:

The reverse proxy server sends "Expect: 100-continue" and request body at the same time.

tcpdump on web1 shows the following.
~~~
15:22:42.849872 IP (tos 0x0, ttl 64, id 30127, offset 0, flags [DF], proto TCP (6), length 7465)
    192.168.122.138.46952 > 192.168.122.134.80: Flags [P.], cksum 0x937d (incorrect -> 0x8655), seq 1:7414, ack 1, win 229, options [nop,nop,TS val 3899105020 ecr 11338665], length 7413: HTTP, length: 7413
	POST /cgi-bin/test.cgi HTTP/1.1
	Host: web1
	User-Agent: curl/7.29.0
	Accept: */*
	Content-Type: application/x-www-form-urlencoded
	Expect: 100-continue
	X-Forwarded-For: 192.168.122.47
	X-Forwarded-Host: rhel8-web1
	X-Forwarded-Server: rhel8-web1.example.com
	Connection: Keep-Alive
	Content-Length: 7104
	
	127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
	::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
...
~~~


Expected results:

The reverse proxy server should send only "Expect: 100-continue" at first.


Additional info:

httpd-2.4.41 includes some patches.

http://people.apache.org/~ylavic/patches/httpd-2.4.x-forward_100_continue-v4.patch.
~~~
Handle end-to-end 100-continue, according to RFC 7231, such that the client
request body is not read/forwarded (according to its "Expect:" header) until
the backend wants to receive it (with interim 100 continue response), or never
forwarded if the backend provides a (non-interim) response and doesn't need
the client body at all.
...
~~~

==> https://bz.apache.org/bugzilla/show_bug.cgi?id=60330
Comment 1 Muneaki Sugaya 2019-11-19 07:44:24 UTC 
The customer found another issue on httpd-2.4.41 where the httpd discard the request body on fail-over. So would you please take care of it too?

Steps to Reproduce:
It's almost the same as this description section.

1. Reverse proxy server(rhel8-web1/192.168.122.47) 

2. Backend servers
Run the following on web2
~~~
# iptables -A INPUT -s 192.168.122.47 -j DROP
~~~

Run tcpdump as follows on web1(192.168.122.134)
~~~
# tcpdump -i ens3 -nnv port 80
~~~

3. Client
Run curl as follows.
~~~
# echo abcd > /tmp/data
# curl -X POST --data-binary '@/tmp/data' http://rhel8-web1/sample/cgi-bin/test.cgi
~~~


Actual results:
After rhel8-web1 does ping to web2, it fails over the request to web1. Then, however it discards the request body. tcpdump on web1 shows the following.
~~~
16:10:21.413117 IP (tos 0x0, ttl 64, id 39364, offset 0, flags [DF], proto TCP (6), length 346)
    192.168.122.130.60214 > 192.168.122.116.80: Flags [P.], cksum 0x1931 (correct), seq 299:593, ack 311, win 254, options [nop,nop,TS val 3178626292 ecr 14370923], length 294: HTTP, length: 294
        POST /cgi-bin/test.cgi HTTP/1.1
        Host: web1
        User-Agent: curl/7.29.0
        Accept: */*
        Content-Type: application/x-www-form-urlencoded
        Expect: 100-Continue
        X-Forwarded-For: 192.168.122.47
        X-Forwarded-Host: rhel8-web1
        X-Forwarded-Server: rhel8-web1.example.com
        Content-Length: 0
        Connection: Keep-Alive

16:10:21.417739 IP (tos 0x0, ttl 64, id 36850, offset 0, flags [DF], proto TCP (6), length 264)
    192.168.122.116.80 > 192.168.122.130.60214: Flags [P.], cksum 0x7742 (incorrect -> 0xe98a), seq 311:523, ack 593, win 260, options [nop,nop,TS val 14385904 ecr 3178626292], length 212: HTTP, length: 212
        HTTP/1.1 200 OK
        Date: Tue, 19 Nov 2019 07:10:21 GMT
        Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/5.4.16
        Connection: close
        Content-Type: text/plain; charset=UTF-8
        
16:10:21.459081 IP (tos 0x0, ttl 64, id 39365, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.130.60214 > 192.168.122.116.80: Flags [.], cksum 0xb3b6 (correct), ack 523, win 262, options [nop,nop,TS val 3178626339 ecr 14385904], length 0
16:10:21.465621 IP (tos 0x0, ttl 64, id 36851, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.116.80 > 192.168.122.130.60214: Flags [F.], cksum 0x766e (incorrect -> 0xb387), seq 523, ack 593, win 260, options [nop,nop,TS val 14385952 ecr 3178626339], length 0
16:10:21.466304 IP (tos 0x0, ttl 64, id 39366, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.130.60214 > 192.168.122.116.80: Flags [F.], cksum 0xb37d (correct), seq 593, ack 524, win 262, options [nop,nop,TS val 3178626346 ecr 14385952], length 0
16:10:21.466313 IP (tos 0x0, ttl 64, id 36852, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.116.80 > 192.168.122.130.60214: Flags [.], cksum 0x766e (incorrect -> 0xb37f), ack 594, win 260, options [nop,nop,TS val 14385952 ecr 3178626346], length 0
~~~


Expected results:

rhel8-web1 should send the request body, the string "abcd" as follows.
~~~
192.168.122.130.60214 > 192.168.122.116.80: Flags [P.], cksum 0xc81b (correct), seq 593:598, ack 26, win 229, options [nop,nop,TS val 3178768885 ecr 14528490], length 5: HTTP
~~~
Comment 2 Luboš Uhliarik 2019-11-19 12:45:26 UTC 
(In reply to Muneaki Sugaya from comment #1)
> The customer found another issue on httpd-2.4.41 where the httpd discard the
> request body on fail-over. So would you please take care of it too?
> 

Hi Muneaki,

in what version customer found that issue? We don't distribute 2.4.41 in RHEL-8, only 2.4.37.
Comment 6 Luboš Uhliarik 2020-03-04 11:48:20 UTC 
Created attachment 1667461 [details]
Patch fixing the issue

I can confirm, that following patch fixes the issue mentioned in Comment 1.
Comment 19 errata-xmlrpc 2020-11-04 03:35:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: httpd:2.4 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4751