Bug 1975790 - HAProxy processes consume too much memory in ACTIVE_STANDBY topology
Summary: HAProxy processes consume too much memory in ACTIVE_STANDBY topology
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-octavia
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z7
: 16.1 (Train on RHEL 8.2)
Assignee: Gregory Thiemonge
QA Contact: Bruna Bonguardo
URL:
Whiteboard:
Depends On:
Blocks: 1907965 1976826
TreeView+ depends on / blocked
 
Reported: 2021-06-24 12:32 UTC by Gregory Thiemonge
Modified: 2021-12-09 20:20 UTC (History)
6 users (show)

Fixed In Version: openstack-octavia-5.0.3-1.20210712123304.8c32d2e.el8ost
Doc Type: Bug Fix
Doc Text:
Before this update, when a configuration change to a Load-balancing service amphora caused an haproxy reload, the process consumed a lot of memory that could lead to memory allocation errors. The issue was caused by the `lo` interface not being configured in the amphora-haproxy namespace in the amphora. With this update, the namespace issue has been corrected and the problem is resolved.
Clone Of:
: 1976826 (view as bug list)
Environment:
Last Closed: 2021-12-09 20:20:10 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack Storyboard 2009005 0 None None None 2021-06-24 12:32:22 UTC
OpenStack gerrit 797882 0 None MERGED Enable lo interface in the amphora-haproxy netns 2021-07-01 08:16:35 UTC
OpenStack gerrit 798989 0 None MERGED Enable lo interface in the amphora-haproxy netns 2022-05-04 08:13:51 UTC
Red Hat Issue Tracker OSP-5432 0 None None None 2021-11-18 11:34:44 UTC
Red Hat Product Errata RHBA-2021:3762 0 None None None 2021-12-09 20:20:38 UTC

Description Gregory Thiemonge 2021-06-24 12:32:23 UTC 
Description of problem:

When using a load balancer in ACTIVE_STANDBY topology, the haproxy instance that runs in the amphora is prone to memory allocation errors, because each haproxy worker consumes a lot of memory, and multiple workers are running at the same time after a configuration update.

The visible side effects in the octavia worker are some exceptions thrown when calling the amphora-agent:

ERROR oslo_messaging.rpc.server octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error

ERROR octavia.amphorae.drivers.haproxy.exceptions [XXX - XXX - - -] Amphora agent returned unexpected result code 500 with response {'message': 'Error reloading haproxy', 'details': 'Redirecting to /bin/systemctl reload haproxy-XXX.service\nJob for haproxy-XXX.service canceled.\n'}


Version-Release number of selected component (if applicable):
16.1

How reproducible:
100%

Steps to Reproduce:
1. Create a LB in Active standby
2. Create a listener and a pool
3. Create many members

Actual results:
After each new member creation, there is one more haproxy processes that should be cleaned up, memory consumption can be important depending on the listener configuration


Expected results:
A configuration change should not reduce the available memory in the amphora


Additional info:

Detailed report can be found in the upstream story https://storyboard.openstack.org/#!/story/2009005
Comment 7 Bruna Bonguardo 2021-07-22 10:32:57 UTC 
Bug cannot be verified until 16.1 z7 puddle is available.
Comment 8 Bruna Bonguardo 2021-07-22 10:33:10 UTC 
Bug cannot be verified until 16.1 z7 puddle is available.
Comment 12 Bruna Bonguardo 2021-08-30 20:41:07 UTC 
#Tested on version:
(overcloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed
16.1  -p RHOS-16.1-RHEL-8-20210804.n.0

#No members:
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root        5119  0.0  1.0  80712  8996 ?        Ss   Aug23   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 11668
nobody     11748  0.0  1.2  91468 10444 ?        Ss   Aug23   2:56 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 11668
root      396610  0.0  0.1  12108   932 pts/0    R+   16:11   0:00 grep --color=auto haproxy

[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State            Recv-Q           Send-Q                     Local Address:Port                      Peer Address:Port                                                          
LISTEN           0                50000                         10.0.0.218:22                             0.0.0.0:*                users:(("haproxy",pid=11748,fd=7))           
LISTEN           0                6                             10.0.0.184:1025                           0.0.0.0:*                users:(("haproxy",pid=11748,fd=9))           
ESTAB            0                0                             10.0.0.184:1025                        10.0.0.199:12193            users:(("haproxy",pid=11748,fd=13))  


#Created two members:

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.20 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| address             | 10.0.0.20                            |
| admin_state_up      | True                                 |
| created_at          | 2021-08-30T20:30:46                  |
| id                  | 08042d29-f496-473f-930c-101efa20615e |
| name                |                                      |
| operating_status    | NO_MONITOR                           |
| project_id          | 75228e22f16c4c2087a2ab2324427aa8     |
| protocol_port       | 80                                   |
| provisioning_status | PENDING_CREATE                       |
| subnet_id           | 5f5aea73-0bc1-452c-aa17-64616b99d953 |
| updated_at          | None                                 |
| weight              | 1                                    |
| monitor_port        | None                                 |
| monitor_address     | None                                 |
| backup              | False                                |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.30 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| address             | 10.0.0.30                            |
| admin_state_up      | True                                 |
| created_at          | 2021-08-30T20:31:00                  |
| id                  | 0e1d39c8-37f9-4e95-9f0f-cf92d410e401 |
| name                |                                      |
| operating_status    | NO_MONITOR                           |
| project_id          | 75228e22f16c4c2087a2ab2324427aa8     |
| protocol_port       | 80                                   |
| provisioning_status | PENDING_CREATE                       |
| subnet_id           | 5f5aea73-0bc1-452c-aa17-64616b99d953 |
| updated_at          | None                                 |
| weight              | 1                                    |
| monitor_port        | None                                 |
| monitor_address     | None                                 |
| backup              | False                                |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer status show lb1
{
    "loadbalancer": {
        "id": "61e24d04-0e21-4cc5-9ba4-94469e1e7028",
        "name": "lb1",
        "operating_status": "ONLINE",
        "provisioning_status": "ACTIVE",
        "listeners": [
            {
                "id": "4ef152f1-018f-4113-8707-aadc81192702",
                "name": "listener1",
                "operating_status": "ONLINE",
                "provisioning_status": "ACTIVE",
                "pools": [
                    {
                        "id": "10aa023e-509d-4ee8-8d2b-98e3e461bbcc",
                        "name": "",
                        "provisioning_status": "ACTIVE",
                        "operating_status": "ONLINE",
                        "members": [
                            {
                                "id": "08042d29-f496-473f-930c-101efa20615e",
                                "name": "",
                                "operating_status": "NO_MONITOR",
                                "provisioning_status": "ACTIVE",
                                "address": "10.0.0.20",
                                "protocol_port": 80
                            },
                            {
                                "id": "0e1d39c8-37f9-4e95-9f0f-cf92d410e401",
                                "name": "",
                                "operating_status": "NO_MONITOR",
                                "provisioning_status": "ACTIVE",
                                "address": "10.0.0.30",
                                "protocol_port": 80
                            }
                        ]


#On the amphora:
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root        5119  0.0  1.0  80716  8936 ?        Ss   Aug23   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397383
nobody    397458  0.0  1.2  91472 10380 ?        Ss   16:31   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397383
root      397472  0.0  0.1  12108  1088 pts/0    S+   16:31   0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State              Recv-Q           Send-Q                     Local Address:Port                     Peer Address:Port                                                         
LISTEN             0                50000                         10.0.0.218:22                            0.0.0.0:*               users:(("haproxy",pid=397458,fd=7))          
LISTEN             0                6                             10.0.0.184:1025                          0.0.0.0:*               users:(("haproxy",pid=397458,fd=9))          
TIME-WAIT          0                0                             10.0.0.184:1025                       10.0.0.199:12753                                                        
TIME-WAIT          0                0                             10.0.0.184:1025                       10.0.0.199:12741

#After some time:
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root        5119  0.0  1.0  80716  8936 ?        Ss   Aug23   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397383
nobody    397458  0.0  1.2  91472 10380 ?        Ss   16:31   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397383
root      397530  0.0  0.1  12108   968 pts/0    R+   16:32   0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State              Recv-Q             Send-Q                         Local Address:Port                          Peer Address:Port                                                              
LISTEN             0                  50000                             10.0.0.218:22                                 0.0.0.0:*                 users:(("haproxy",pid=397458,fd=7))             
LISTEN             0                  6                                 10.0.0.184:1025                               0.0.0.0:*                 users:(("haproxy",pid=397458,fd=9))             
ESTAB              0                  0                                 10.0.0.184:17513                           10.0.0.199:1025              users:(("haproxy",pid=397458,fd=13)) 

#Adding more members:
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.40 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.50 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.60 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.70 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc
openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.80 --protocol-port 80 10aa023e-509d-4ee8-8d2b-98e3e461bbcc


#While adding the members:

[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root        5119  0.0  1.0  80716  9080 ?        Ss   Aug23   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397950
nobody    398030  0.0  1.2  91472 10452 ?        Ss   16:37   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397950
root      398036  0.0  0.1  12108  1028 pts/0    R+   16:37   0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State                Recv-Q            Send-Q                         Local Address:Port                         Peer Address:Port                                                              
LISTEN               0                 50000                             10.0.0.218:22                                0.0.0.0:*                 users:(("haproxy",pid=398030,fd=7))             
LISTEN               0                 6                                 10.0.0.184:1025                              0.0.0.0:*                 users:(("haproxy",pid=398030,fd=9))             
TIME-WAIT            0                 0                                 10.0.0.184:1025                           10.0.0.199:12917                                                             
ESTAB                0                 0                                 10.0.0.184:1025                           10.0.0.199:12959             users:(("haproxy",pid=398030,fd=13))            
TIME-WAIT            0                 0                                 10.0.0.184:1025                           10.0.0.199:12951         

#After a while:

[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ps aux | grep haproxy
root        5119  0.0  1.0  80716  9080 ?        Ss   Aug23   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397950
nobody    398030  0.0  1.2  91472 10452 ?        Ss   16:37   0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/61e24d04-0e21-4cc5-9ba4-94469e1e7028/61e24d04-0e21-4cc5-9ba4-94469e1e7028.pid -L vR4codEQYETFRBhVVDa6jcDUjUY -sf 397950
root      398096  0.0  0.1  12108   968 pts/0    R+   16:38   0:00 grep --color=auto haproxy
[root@amphora-7bf6634e-4f56-428d-b1ff-48e45d6099d0 ~]# ss -natp
State                Recv-Q               Send-Q                              Local Address:Port                               Peer Address:Port                                                                   
LISTEN               0                    50000                                  10.0.0.218:22                                      0.0.0.0:*                    users:(("haproxy",pid=398030,fd=7))               
LISTEN               0                    6                                      10.0.0.184:1025                                    0.0.0.0:*                    users:(("haproxy",pid=398030,fd=9))               
ESTAB                0                    0                                      10.0.0.184:1025                                 10.0.0.199:12991                users:(("haproxy",pid=398030,fd=14))              
                                                    

Moving the bug to VERIFIED.
Comment 25 errata-xmlrpc 2021-12-09 20:20:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3762
Note You need to log in before you can comment on or make changes to this bug.