Bug 1975790
| Summary: | HAProxy processes consume too much memory in ACTIVE_STANDBY topology | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Gregory Thiemonge <gthiemon> | |
| Component: | openstack-octavia | Assignee: | Gregory Thiemonge <gthiemon> | |
| Status: | CLOSED ERRATA | QA Contact: | Bruna Bonguardo <bbonguar> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 16.1 (Train) | CC: | bperkins, gregraka, ihrachys, lpeer, majopela, scohen | |
| Target Milestone: | z7 | Keywords: | Triaged | |
| Target Release: | 16.1 (Train on RHEL 8.2) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-octavia-5.0.3-1.20210712123304.8c32d2e.el8ost | Doc Type: | Bug Fix | |
| Doc Text: |
Before this update, when a configuration change to a Load-balancing service amphora caused an haproxy reload, the process consumed a lot of memory that could lead to memory allocation errors. The issue was caused by the `lo` interface not being configured in the amphora-haproxy namespace in the amphora. With this update, the namespace issue has been corrected and the problem is resolved.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1976826 (view as bug list) | Environment: | ||
| Last Closed: | 2021-12-09 20:20:10 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1907965, 1976826 | |||
Bug cannot be verified until 16.1 z7 puddle is available. Bug cannot be verified until 16.1 z7 puddle is available. #Tested on version:
(overcloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed
16.1 -p RHOS-16.1-RHEL-8-20210804.n.0
#No members:
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root 5119 0.0 1.0 80712 8996 ? Ss Aug23 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 11668
nobody 11748 0.0 1.2 91468 10444 ? Ss Aug23 2:56 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 11668
root 396610 0.0 0.1 12108 932 pts/0 R+ 16:11 0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 50000 10.0.0.218:22 0.0.0.0:* users:(("haproxy",pid=11748,fd=7))
LISTEN 0 6 10.0.0.184:1025 0.0.0.0:* users:(("haproxy",pid=11748,fd=9))
ESTAB 0 0 10.0.0.184:1025 10.0.0.199:12193 users:(("haproxy",pid=11748,fd=13))
#Created two members:
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.20 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| address | 10.0.0.20 |
| admin_state_up | True |
| created_at | 2021-08-30T20:30:46 |
| id | 08042d29-f496-473f-930c-101efa20615e |
| name | |
| operating_status | NO_MONITOR |
| project_id | 75228e22f16c4c2087a2ab2324427aa8 |
| protocol_port | 80 |
| provisioning_status | PENDING_CREATE |
| subnet_id | 5f5aea73-0bc1-452c-aa17-64616b99d953 |
| updated_at | None |
| weight | 1 |
| monitor_port | None |
| monitor_address | None |
| backup | False |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.30 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| address | 10.0.0.30 |
| admin_state_up | True |
| created_at | 2021-08-30T20:31:00 |
| id | 0e1d39c8-37f9-4e95-9f0f-cf92d410e401 |
| name | |
| operating_status | NO_MONITOR |
| project_id | 75228e22f16c4c2087a2ab2324427aa8 |
| protocol_port | 80 |
| provisioning_status | PENDING_CREATE |
| subnet_id | 5f5aea73-0bc1-452c-aa17-64616b99d953 |
| updated_at | None |
| weight | 1 |
| monitor_port | None |
| monitor_address | None |
| backup | False |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer status show lb1
{
"loadbalancer": {
"id": "61e24d04-0e21-4cc5-9ba4-94469e1e7028",
"name": "lb1",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"listeners": [
{
"id": "4ef152f1-018f-4113-8707-aadc81192702",
"name": "listener1",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"pools": [
{
"id": "10aa023e-509d-4ee8-8d2b-98e3e461bbcc",
"name": "",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE",
"members": [
{
"id": "08042d29-f496-473f-930c-101efa20615e",
"name": "",
"operating_status": "NO_MONITOR",
"provisioning_status": "ACTIVE",
"address": "10.0.0.20",
"protocol_port": 80
},
{
"id": "0e1d39c8-37f9-4e95-9f0f-cf92d410e401",
"name": "",
"operating_status": "NO_MONITOR",
"provisioning_status": "ACTIVE",
"address": "10.0.0.30",
"protocol_port": 80
}
]
#On the amphora:
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root 5119 0.0 1.0 80716 8936 ? Ss Aug23 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397383
nobody 397458 0.0 1.2 91472 10380 ? Ss 16:31 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397383
root 397472 0.0 0.1 12108 1088 pts/0 S+ 16:31 0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 50000 10.0.0.218:22 0.0.0.0:* users:(("haproxy",pid=397458,fd=7))
LISTEN 0 6 10.0.0.184:1025 0.0.0.0:* users:(("haproxy",pid=397458,fd=9))
TIME-WAIT 0 0 10.0.0.184:1025 10.0.0.199:12753
TIME-WAIT 0 0 10.0.0.184:1025 10.0.0.199:12741
#After some time:
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root 5119 0.0 1.0 80716 8936 ? Ss Aug23 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397383
nobody 397458 0.0 1.2 91472 10380 ? Ss 16:31 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397383
root 397530 0.0 0.1 12108 968 pts/0 R+ 16:32 0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 50000 10.0.0.218:22 0.0.0.0:* users:(("haproxy",pid=397458,fd=7))
LISTEN 0 6 10.0.0.184:1025 0.0.0.0:* users:(("haproxy",pid=397458,fd=9))
ESTAB 0 0 10.0.0.184:17513 10.0.0.199:1025 users:(("haproxy",pid=397458,fd=13))
#Adding more members:
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.40 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.50 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.60 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.70 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.80 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
#While adding the members:
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root 5119 0.0 1.0 80716 9080 ? Ss Aug23 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397950
nobody 398030 0.0 1.2 91472 10452 ? Ss 16:37 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397950
root 398036 0.0 0.1 12108 1028 pts/0 R+ 16:37 0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 50000 10.0.0.218:22 0.0.0.0:* users:(("haproxy",pid=398030,fd=7))
LISTEN 0 6 10.0.0.184:1025 0.0.0.0:* users:(("haproxy",pid=398030,fd=9))
TIME-WAIT 0 0 10.0.0.184:1025 10.0.0.199:12917
ESTAB 0 0 10.0.0.184:1025 10.0.0.199:12959 users:(("haproxy",pid=398030,fd=13))
TIME-WAIT 0 0 10.0.0.184:1025 10.0.0.199:12951
#After a while:
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root 5119 0.0 1.0 80716 9080 ? Ss Aug23 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397950
nobody 398030 0.0 1.2 91472 10452 ? Ss 16:37 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397950
root 398096 0.0 0.1 12108 968 pts/0 R+ 16:38 0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 50000 10.0.0.218:22 0.0.0.0:* users:(("haproxy",pid=398030,fd=7))
LISTEN 0 6 10.0.0.184:1025 0.0.0.0:* users:(("haproxy",pid=398030,fd=9))
ESTAB 0 0 10.0.0.184:1025 10.0.0.199:12991 users:(("haproxy",pid=398030,fd=14))
Moving the bug to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3762 |
Description of problem: When using a load balancer in ACTIVE_STANDBY topology, the haproxy instance that runs in the amphora is prone to memory allocation errors, because each haproxy worker consumes a lot of memory, and multiple workers are running at the same time after a configuration update. The visible side effects in the octavia worker are some exceptions thrown when calling the amphora-agent: ERROR oslo_messaging.rpc.server octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error ERROR octavia.amphorae.drivers.haproxy.exceptions [XXX - XXX - - -] Amphora agent returned unexpected result code 500 with response {'message': 'Error reloading haproxy', 'details': 'Redirecting to /bin/systemctl reload haproxy-XXX.service\nJob for haproxy-XXX.service canceled.\n'} Version-Release number of selected component (if applicable): 16.1 How reproducible: 100% Steps to Reproduce: 1. Create a LB in Active standby 2. Create a listener and a pool 3. Create many members Actual results: After each new member creation, there is one more haproxy processes that should be cleaned up, memory consumption can be important depending on the listener configuration Expected results: A configuration change should not reduce the available memory in the amphora Additional info: Detailed report can be found in the upstream story https://storyboard.openstack.org/#!/story/2009005