RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1947750 - haproxy doesn't comply with the provided fall/rise check parameters [rhel-7.9.z]
Summary: haproxy doesn't comply with the provided fall/rise check parameters [rhel-7.9.z]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: haproxy
Version: 7.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Ryan O'Hara
QA Contact: Brandon Perkins
URL:
Whiteboard:
Depends On:
Blocks: 1946915
TreeView+ depends on / blocked
 
Reported: 2021-04-09 06:56 UTC by Gregory Thiemonge
Modified: 2021-06-08 22:37 UTC (History)
3 users (show)

Fixed In Version: haproxy-1.5.18-9.el7_9.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-08 22:37:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
haproxy config to reproduce the issue (425 bytes, text/plain)
2021-04-09 06:56 UTC, Gregory Thiemonge 		no flags Details
fake python HTTP server (328 bytes, text/x-python)
2021-04-09 06:56 UTC, Gregory Thiemonge 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:2326 0 None None None 2021-06-08 22:37:21 UTC

Internal Links: 1946915

Description Gregory Thiemonge 2021-04-09 06:56:09 UTC 
Created attachment 1770497 [details]
haproxy config to reproduce the issue

Description of problem:
haproxy marks a server as UP even if it didn't get enough consecutive successful health checks.

An OpenStack Platform customer has an issue when using health checks in haproxy.
haproxy doesn't respect the 'rise' parameter in case of network outages.


Version-Release number of selected component (if applicable):
RHEL 7.7
haproxy-1.5.18-9.el7.x86_64


How reproducible:
100%


Steps to Reproduce:
1. creates a backend server with the following params:
   option httpchk GET /check
   http-check expect rstatus 200
   server server1 127.0.0.1:5000 weight 1 check inter 5s fall 3 rise 3

2. creates a dummy HTTP server that alternatively replies 200 and 404 codes

3. launch haproxy
   ./haproxy -f check.cfg -d 2>&1 | gawk '{ print strftime("[%Y-%m-%d %H:%M:%S]"), $0 }'

4. launch the dummy webserver
   python3 flask_server.py

5. haproxy marks the server as DOWN (because it wasn't running when haproxy started)

[2021-04-08 16:45:07] [WARNING] 097/164507 (2198077) : Server backend1/server1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

6. haproxy sends periodic health checks, servers replies 200 and 404 (but never it replies 3 consecutive 200 or 3 consecutive 404):

 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 852-298-336
127.0.0.1 - - [08/Apr/2021 16:45:12] "GET /check HTTP/1.0" 404 -
127.0.0.1 - - [08/Apr/2021 16:45:17] "GET /check HTTP/1.0" 200 -
127.0.0.1 - - [08/Apr/2021 16:45:22] "GET /check HTTP/1.0" 404 -
127.0.0.1 - - [08/Apr/2021 16:45:27] "GET /check HTTP/1.0" 200 -
127.0.0.1 - - [08/Apr/2021 16:45:32] "GET /check HTTP/1.0" 404 -
127.0.0.1 - - [08/Apr/2021 16:45:37] "GET /check HTTP/1.0" 200 -
127.0.0.1 - - [08/Apr/2021 16:45:42] "GET /check HTTP/1.0" 404 -
127.0.0.1 - - [08/Apr/2021 16:45:47] "GET /check HTTP/1.0" 200 -

7. at 16:45:37, haproxy sets the server1 UP, because it got 3 successful health checks (but not consecutive)

[2021-04-08 16:45:37] [WARNING] 097/164537 (2198077) : Server backend1/server1 is UP, reason: Layer7 check passed, code: 200, info: "HTTP status check returned code <3C>200<3E>", check duration: 4ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.


Actual results:

Server is UP but it didn't match the 'rise' criteria.


Expected results:

Server should stay down


Additional info:
Customer is using Octavia in OSP13z12 and is requesting a hotfix
haproxy conf and dummy webserver are provided in attachment.

haproxy releases from 1.5 to 1.8 are affected, the bug was fixed in haproxy 1.9 with the following commit:
https://github.com/haproxy/haproxy/commit/b119a79fc336f2b6074de1c3113b1682c717985c
Comment 2 Gregory Thiemonge 2021-04-09 06:56:44 UTC 
Created attachment 1770498 [details]
fake python HTTP server
Comment 3 Ryan O'Hara 2021-04-09 16:37:25 UTC 
I've not yet tried this, but the immediate thing that jumps out at my in the config file is:

    http-check expect rstatus 200

Note that rstatus expects a regular expression. If you just want to check that the status is 200, change "rstatus" to "status" and test again.
Comment 4 Ryan O'Hara 2021-04-09 16:38:05 UTC 
See documentation here:

http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-http-check%20expect
Comment 8 Ryan O'Hara 2021-04-13 17:35:49 UTC 
(In reply to Gregory Thiemonge from comment #0)
> 
> haproxy releases from 1.5 to 1.8 are affected, the bug was fixed in haproxy
> 1.9 with the following commit:
> https://github.com/haproxy/haproxy/commit/
> b119a79fc336f2b6074de1c3113b1682c717985c

Please note that haproxy 1.8 in RHEL8 and RHSCL are not affected by this bug. Each of the current releases in those products contains the patch.
Comment 20 errata-xmlrpc 2021-06-08 22:37:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (haproxy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2326
Note You need to log in before you can comment on or make changes to this bug.