Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionGregory Thiemonge
2021-04-09 06:56:09 UTC
Created attachment 1770497[details]
haproxy config to reproduce the issue
Description of problem:
haproxy marks a server as UP even if it didn't get enough consecutive successful health checks.
An OpenStack Platform customer has an issue when using health checks in haproxy.
haproxy doesn't respect the 'rise' parameter in case of network outages.
Version-Release number of selected component (if applicable):
RHEL 7.7
haproxy-1.5.18-9.el7.x86_64
How reproducible:
100%
Steps to Reproduce:
1. creates a backend server with the following params:
option httpchk GET /check
http-check expect rstatus 200
server server1 127.0.0.1:5000 weight 1 check inter 5s fall 3 rise 3
2. creates a dummy HTTP server that alternatively replies 200 and 404 codes
3. launch haproxy
./haproxy -f check.cfg -d 2>&1 | gawk '{ print strftime("[%Y-%m-%d %H:%M:%S]"), $0 }'
4. launch the dummy webserver
python3 flask_server.py
5. haproxy marks the server as DOWN (because it wasn't running when haproxy started)
[2021-04-08 16:45:07] [WARNING] 097/164507 (2198077) : Server backend1/server1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
6. haproxy sends periodic health checks, servers replies 200 and 404 (but never it replies 3 consecutive 200 or 3 consecutive 404):
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 852-298-336
127.0.0.1 - - [08/Apr/2021 16:45:12] "GET /check HTTP/1.0" 404 -
127.0.0.1 - - [08/Apr/2021 16:45:17] "GET /check HTTP/1.0" 200 -
127.0.0.1 - - [08/Apr/2021 16:45:22] "GET /check HTTP/1.0" 404 -
127.0.0.1 - - [08/Apr/2021 16:45:27] "GET /check HTTP/1.0" 200 -
127.0.0.1 - - [08/Apr/2021 16:45:32] "GET /check HTTP/1.0" 404 -
127.0.0.1 - - [08/Apr/2021 16:45:37] "GET /check HTTP/1.0" 200 -
127.0.0.1 - - [08/Apr/2021 16:45:42] "GET /check HTTP/1.0" 404 -
127.0.0.1 - - [08/Apr/2021 16:45:47] "GET /check HTTP/1.0" 200 -
7. at 16:45:37, haproxy sets the server1 UP, because it got 3 successful health checks (but not consecutive)
[2021-04-08 16:45:37] [WARNING] 097/164537 (2198077) : Server backend1/server1 is UP, reason: Layer7 check passed, code: 200, info: "HTTP status check returned code <3C>200<3E>", check duration: 4ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Actual results:
Server is UP but it didn't match the 'rise' criteria.
Expected results:
Server should stay down
Additional info:
Customer is using Octavia in OSP13z12 and is requesting a hotfix
haproxy conf and dummy webserver are provided in attachment.
haproxy releases from 1.5 to 1.8 are affected, the bug was fixed in haproxy 1.9 with the following commit:
https://github.com/haproxy/haproxy/commit/b119a79fc336f2b6074de1c3113b1682c717985c
Comment 2Gregory Thiemonge
2021-04-09 06:56:44 UTC
I've not yet tried this, but the immediate thing that jumps out at my in the config file is:
http-check expect rstatus 200
Note that rstatus expects a regular expression. If you just want to check that the status is 200, change "rstatus" to "status" and test again.
(In reply to Gregory Thiemonge from comment #0)
>
> haproxy releases from 1.5 to 1.8 are affected, the bug was fixed in haproxy
> 1.9 with the following commit:
> https://github.com/haproxy/haproxy/commit/
> b119a79fc336f2b6074de1c3113b1682c717985c
Please note that haproxy 1.8 in RHEL8 and RHSCL are not affected by this bug. Each of the current releases in those products contains the patch.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (haproxy bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2021:2326