Description of problem: octavia_tempest_plugin.tests.api.v2.test_pool.PoolAPITest.test_pool_delete scenario test fails when ACTIVE_STANDBY topology is enabled Version-Release number of selected component (if applicable): 16.1 How reproducible: 100% Steps to Reproduce: 1. Configure Octavia for ACTIVE_STANDBY 2. Launch Octavia tempest plugin (can use 'PoolAPITest' regex) 3. Actual results: Traceback (most recent call last): File "/home/stack/plugins/octavia/octavia_tempest_plugin/tests/api/v2/test_pool.py", line 732, in test_pool_delete CONF.load_balancer.check_timeout) File "/home/stack/plugins/octavia/octavia_tempest_plugin/tests/waiters.py", line 182, in wait_for_deleted_status_or_not_found raise exceptions.TimeoutException(message) tempest.lib.exceptions.TimeoutException: Request timed out Details: (PoolAPITest:test_pool_delete) show_pool provisioning_status failed to update to DELETED or become not found (404) within the required time 120. Current status of show_pool: PENDING_DELETE Expected results: Test should pass Additional info:
I reproduced it in devstack, using the OSP16.1 Octavia amphora: 1. with tempest $ tox -eall -- "octavia_tempest_plugin.*PoolAPITest.*(test_pool_create_with_listener|test_pool_delete)" 2. or with the CLI: Create a LB and a listener (in ACTIVE_STANDBY) $ openstack loadbalancer create --vip-subnet private-subnet --name lb1 $ openstack loadbalancer listener create --protocol HTTP --protocol-port 80 --name listener1 lb1 Create a first pool with APP_COOKIE session-persistence $ openstack loadbalancer pool create --protocol HTTP --listener listener1 --lb-algorithm ROUND_ROBIN --session-persistence type=APP_COOKIE,cookie_name=my_cookie --name pool1 Create a second pool with session-persistence $ openstack loadbalancer pool create --protocol HTTP --loadbalancer lb1 --lb-algorithm ROUND_ROBIN --session-persistence type=APP_COOKIE,cookie_name=my_cookie --name pool2 Delete the second pool $ openstack loadbalancer pool delete pool2 Haproxy fails while reloading: Jun 09 03:52:05 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal amphora-agent[1193]: 2020-06-09 03:52:05.755 1193 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Found init system: systemd upload_haproxy_config /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:152 Jun 09 03:52:05 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal systemd[1]: Reloading. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal systemd[1]: Reloading. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal systemd[1]: Reloading HAProxy Load Balancer. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal haproxy[5271]: Configuration file is valid Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: [WARNING] 160/035043 (4915) : Reexecuting Master process Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: [WARNING] 160/035206 (4915) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 2500040, limit is 2097152. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: [WARNING] 160/035206 (4915) : [/usr/sbin/haproxy.main()] FD limit (2097152) too low for maxconn=1000000/maxsock=2500040. Please raise 'ulimit-n' to 2500040 or more to avoid any trouble. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: [ALERT] 160/035206 (4915) : [/usr/sbin/haproxy.main()] Cannot fork. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: [WARNING] 160/035206 (4915) : Reexecuting Master process in waitpid mode Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: [WARNING] 160/035206 (4915) : Reexecuting Master process Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: Usage : haproxy [-f <cfgfile|cfgdir>]* [ -vdVD ] [ -n <maxconn> ] [ -N <maxpconn> ] Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: [ -p <pidfile> ] [ -m <max megs> ] [ -C <dir> ] [-- <cfgfile>*] Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -v displays version ; -vv shows known build options. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -d enters debug mode ; -db only disables background mode. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -dM[<byte>] poisons memory with <byte> (defaults to 0x50) Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -V enters verbose mode (disables quiet mode) Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -D goes daemon ; -C changes to <dir> before loading files. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -W master-worker mode. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -Ws master-worker mode with systemd notify support. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -q quiet mode : don't display messages Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -c check mode : only check config files and exit Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -n sets the maximum total # of connections (2000) Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -m limits the usable amount of memory (in MB) Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -N sets the default, per-proxy maximum # of connections (2000) Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -L set local peer name (default to hostname) Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -p writes pids of all children to this file Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -de disables epoll() usage even when available Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -dp disables poll() usage even when available Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -dS disables splice usage (broken on old kernels) Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -dG disables getaddrinfo() usage Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -dR disables SO_REUSEPORT usage Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -dr ignores server address resolution failures Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -dV disables SSL verify on servers side Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -sf/-st [pid ]* finishes/terminates old pids. Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: -x <unix_socket> get listening sockets from a unix socket Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: HA-Proxy version 1.8.23 2019/11/25 Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[4915]: Copyright 2000-2019 Willy Tarreau <willy> Jun 09 03:52:06 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal systemd[1]: haproxy-cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.service: Main process exited, code=exited, status=1/FAILURE Jun 09 03:52:08 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal Keepalived_vrrp[4697]: Script `check_script` now returning 1 Jun 09 03:52:13 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal Keepalived_vrrp[4697]: VRRP_Script(check_script) failed (exited with status 1) Jun 09 03:52:13 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal Keepalived_vrrp[4697]: (cbbef8ef65504a559c2f19f1a63bdbdd) Entering FAULT STATE
It looks like reloading haproxy when session-persistence is enabled makes it consume twice the amount of memory: haproxy configuration: global daemon user nobody log /run/rsyslog/octavia/log local0 log /run/rsyslog/octavia/log local1 notice stats socket /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.sock mode 0666 level user maxconn 1000000 defaults log global retries 3 option redispatch option splice-request option splice-response option http-keep-alive peers cbbef8ef65504a559c2f19f1a63bdbdd_peers peer G4SXZTP8batgIQ4iAoBGkWJMeeg 10.0.0.28:1025 peer eTOcOhwbysoktiQ1CxhjxbKu2BU 10.0.0.54:1025 frontend 73f33f7d-b8b7-4c79-bb92-de2e97ba08f6 maxconn 1000000 bind 10.0.0.11:80 mode http default_backend a69bee16-7b23-44b9-bd63-602a108cf477:73f33f7d-b8b7-4c79-bb92-de2e97ba08f6 timeout client 50000 log-format ec4fc6e9845d44ada460fab5c985adf9\ cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd\ %f\ %ci\ %cp\ %t\ %{+Q}r\ %ST\ %B\ %U\ %[ssl_c_verify]\ %{+Q}[ssl_c_s_dn]\ %b\ %s\ %Tt\ %tsc backend a69bee16-7b23-44b9-bd63-602a108cf477:73f33f7d-b8b7-4c79-bb92-de2e97ba08f6 mode http http-reuse safe balance roundrobin stick-table type string len 64 size 10k peers cbbef8ef65504a559c2f19f1a63bdbdd_peers stick store-response res.cook(my_cookie) stick match req.cook(my_cookie) fullconn 1000000 option allbackups timeout connect 5000 timeout server 50000 Before: [root@amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1 cloud-user]# ps axu | grep haproxy root 5760 0.2 1.0 80712 8744 ? Ss 03:58 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.pid -L G4SXZTP8batgIQ4iAoBGkWJMeeg -sf 6130 nobody 6199 0.0 18.9 295568 158812 ? Ss 04:01 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.pid -L G4SXZTP8batgIQ4iAoBGkWJMeeg -sf 6130 root 6223 0.0 0.1 12108 1060 pts/1 S+ 04:02 0:00 grep --color=auto haproxy Reload conf: [root@amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1 cloud-user]# systemctl reload haproxy-cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.service Jun 09 04:03:40 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal systemd[1]: Reloading HAProxy Load Balancer. Jun 09 04:03:40 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal haproxy[6277]: Configuration file is valid Jun 09 04:03:40 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [WARNING] 160/040152 (5760) : Reexecuting Master process Jun 09 04:03:40 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [WARNING] 160/040340 (5760) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 2500040, limit is 2097152. Jun 09 04:03:40 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [WARNING] 160/040340 (5760) : [/usr/sbin/haproxy.main()] FD limit (2097152) too low for maxconn=1000000/maxsock=2500040. Please raise 'ulimit-n' to 2500040 or more to avoid any trouble. Jun 09 04:03:40 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal systemd[1]: Reloaded HAProxy Load Balancer. After: [root@amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1 cloud-user]# ps axu | grep haproxy root 5760 0.1 1.0 80712 8584 ? Ss 03:58 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.pid -L G4SXZTP8batgIQ4iAoBGkWJMeeg -sf 6199 nobody 6199 0.0 18.9 295568 158812 ? Ss 04:01 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.pid -L G4SXZTP8batgIQ4iAoBGkWJMeeg -sf 6130 nobody 6279 0.0 18.9 295568 158808 ? Ss 04:03 0:00 /usr/sbin/haproxy -Ws -f /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd/cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.pid -L G4SXZTP8batgIQ4iAoBGkWJMeeg -sf 6199 root 6281 0.0 0.1 12108 1112 pts/1 S+ 04:03 0:00 grep --color=auto haproxy [root@amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1 cloud-user]# systemctl reload haproxy-cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.service That command is stucked^ Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal systemd[1]: Reloading HAProxy Load Balancer. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal haproxy[6335]: Configuration file is valid Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [WARNING] 160/040340 (5760) : Reexecuting Master process Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [WARNING] 160/040503 (5760) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 2500040, limit is 2097152. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [WARNING] 160/040503 (5760) : [/usr/sbin/haproxy.main()] FD limit (2097152) too low for maxconn=1000000/maxsock=2500040. Please raise 'ulimit-n' to 2500040 or more to avoid any trouble. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [ALERT] 160/040503 (5760) : [/usr/sbin/haproxy.main()] Cannot fork. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [WARNING] 160/040503 (5760) : Reexecuting Master process in waitpid mode Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [WARNING] 160/040503 (5760) : Reexecuting Master process Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: Usage : haproxy [-f <cfgfile|cfgdir>]* [ -vdVD ] [ -n <maxconn> ] [ -N <maxpconn> ] Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: [ -p <pidfile> ] [ -m <max megs> ] [ -C <dir> ] [-- <cfgfile>*] Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -v displays version ; -vv shows known build options. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -d enters debug mode ; -db only disables background mode. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -dM[<byte>] poisons memory with <byte> (defaults to 0x50) Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -V enters verbose mode (disables quiet mode) Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -D goes daemon ; -C changes to <dir> before loading files. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -W master-worker mode. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -Ws master-worker mode with systemd notify support. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -q quiet mode : don't display messages Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -c check mode : only check config files and exit Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -n sets the maximum total # of connections (2000) Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -m limits the usable amount of memory (in MB) Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -N sets the default, per-proxy maximum # of connections (2000) Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -L set local peer name (default to hostname) Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -p writes pids of all children to this file Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -de disables epoll() usage even when available Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -dp disables poll() usage even when available Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -dS disables splice usage (broken on old kernels) Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -dG disables getaddrinfo() usage Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -dR disables SO_REUSEPORT usage Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -dr ignores server address resolution failures Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -dV disables SSL verify on servers side Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -sf/-st [pid ]* finishes/terminates old pids. Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: -x <unix_socket> get listening sockets from a unix socket Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: HA-Proxy version 1.8.23 2019/11/25 Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal ip[5760]: Copyright 2000-2019 Willy Tarreau <willy> Jun 09 04:05:03 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal systemd[1]: haproxy-cbbef8ef-6550-4a55-9c2f-19f1a63bdbdd.service: Main process exited, code=exited, status=1/FAILURE Jun 09 04:05:08 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal Keepalived_vrrp[4697]: Script `check_script` now returning 1 Jun 09 04:05:13 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal Keepalived_vrrp[4697]: VRRP_Script(check_script) failed (exited with status 1) Jun 09 04:05:13 amphora-de6c35a7-40b9-4c8d-8722-c663a0c15de1.novalocal Keepalived_vrrp[4697]: (cbbef8ef65504a559c2f19f1a63bdbdd) Entering FAULT STATE
Update to my last comment: Reloading haproxy when session-persistence is enabled makes it consume twice the amount of memory for 2 minutes: Reloading haproxy: Jun 09 04:32:48 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal systemd[1]: Reloading HAProxy Load Balancer. Jun 09 04:32:48 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal haproxy[7638]: Configuration file is valid Jun 09 04:32:48 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal ip[7213]: [WARNING] 160/043210 (7213) : Reexecuting Master process Jun 09 04:32:48 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal ip[7213]: [WARNING] 160/043248 (7213) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 2500040, limit is 2534. Jun 09 04:32:48 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal ip[7213]: [WARNING] 160/043248 (7213) : [/usr/sbin/haproxy.main()] FD limit (2534) too low for maxconn=1000000/maxsock=2500040. Please raise 'ulimit-n' to 2500040 or more to avoid any trouble. Jun 09 04:32:48 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal systemd[1]: Reloaded HAProxy Load Balancer. Jun 09 04:34:59 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal ip[7213]: [WARNING] 160/043248 (7213) : Former worker 7605 exited with code 0 Former worker was cleaned 2min after launching the new worker. Same here: Jun 09 04:37:00 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal systemd[1]: Reloading HAProxy Load Balancer. Jun 09 04:37:00 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal haproxy[7797]: Configuration file is valid Jun 09 04:37:00 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal ip[7213]: [WARNING] 160/043248 (7213) : Reexecuting Master process Jun 09 04:37:00 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal ip[7213]: [WARNING] 160/043700 (7213) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 2500040, limit is 2534. Jun 09 04:37:00 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal ip[7213]: [WARNING] 160/043700 (7213) : [/usr/sbin/haproxy.main()] FD limit (2534) too low for maxconn=1000000/maxsock=2500040. Please raise 'ulimit-n' to 2500040 or more to avoid any trouble. Jun 09 04:37:00 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal systemd[1]: Reloaded HAProxy Load Balancer. Jun 09 04:39:09 amphora-f66f2cca-4962-4608-8620-acebace3ef21.novalocal ip[7213]: [WARNING] 160/043700 (7213) : Former worker 7640 exited with code 0
*** Bug 1919267 has been marked as a duplicate of this bug. ***
Waiting for this patch [1] to be merged Upstream - so we can verify this bug in our Downstream CI. [1] https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/787177
Created attachment 1780313 [details] a log of an OSP16.1 instance of the bug
@mgarciac This bug is occuring in OSP16.1 https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/8/ the relevant tempest log has been uploaded as a log of an OSP16.1 instance of the bug
Verified that this bug no longer occrs in 16.1 https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/11/ 09/05/2021
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.6 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2097