Description of problem: Since 1.8.x version (RHEL8), haproxy consumes at least 160MB at init time using the default configuration provided by octavia. When a LB is updated, haproxy configuration is updated and a signal is sent to haproxy to reload the configuration. When haproxy reloads its configuration, it creates a new worker, does some allocations (up to 160MB), then destroys the previous worker. So during a short time, memory consumption is increased, and if 2 processes reload a the same time, it may fail with a "Cannot fork" error: Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal systemd[1]: Reloading. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal systemd[1]: Reloading. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal systemd[1]: Reloading HAProxy Load Balancer. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[4203]: Configuration file is valid Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal ip[3908]: [WARNING] 098/120329 (3908) : Reexecuting Master process Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[3908]: Proxy f1075bcd-09df-496f-b5e1-ad9d1f7946e2 started. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal ip[3908]: [WARNING] 098/120343 (3908) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 2500032, limit is 2097152. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[3908]: Proxy f1075bcd-09df-496f-b5e1-ad9d1f7946e2 started. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[3908]: Proxy 4bd6bad2-fc01-4aba-8bea-523a218c49dc started. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[3908]: Proxy 4bd6bad2-fc01-4aba-8bea-523a218c49dc started. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[4108]: Stopping frontend f1075bcd-09df-496f-b5e1-ad9d1f7946e2 in 0 ms. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal ip[3908]: [WARNING] 098/120343 (3908) : [/usr/sbin/haproxy.main()] FD limit (2097152) too low for maxconn=1000000/maxsock=2500032. Please raise 'ulimit-n' to 2500032 or more to avoid any trouble. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal ip[3908]: [ALERT] 098/120343 (3908) : [/usr/sbin/haproxy.main()] Cannot fork. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal ip[3908]: [WARNING] 098/120343 (3908) : Reexecuting Master process in waitpid mode Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal ip[3908]: [WARNING] 098/120343 (3908) : Reexecuting Master process Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[4108]: Stopping frontend f1075bcd-09df-496f-b5e1-ad9d1f7946e2 in 0 ms. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[4108]: Stopping backend 4bd6bad2-fc01-4aba-8bea-523a218c49dc in 0 ms. Apr 09 12:03:43 amphora-24dd8f5a-4e09-4b8b-a794-58ea4607363d.novalocal haproxy[4108]: Stopping backend 4bd6bad2-fc01-4aba-8bea-523a218c49dc in 0 ms. Version-Release number of selected component (if applicable): OSP15 How reproducible: 100% Steps to Reproduce: 1. create 2 listeners that use the same pool 2. add a new member in the pool 3. both listeners are restarted, which causes memory allocation issues Actual results: haproxy crashes, amphora doesn't work anymore Expected results: Additional info:
This issue can be worked around by limiting the per-listener connection limits. By default this setting is -1, which is "unlimited connections". To work around this issue, set the individual listener connection limit to a setting appropriate for the application. The lower the per-listener connection limits, the more listeners a load balancer can handle. For example, a load balancer with twenty listeners can support each listener using 100,000 for the connection limit. Via the API this setting is "connection_limit" on the listener. Via the CLI this setting is "--connection-limit" on the listener. This is an interim work around until a patch can be made available for HAProxy 1.8 based amphora.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811