Bug 2125612
| Summary: | [Octavia] Spam of "nf_conntrack: table full, dropping packet" messages during performance tests | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Gregory Thiemonge <gthiemon> |
| Component: | openstack-octavia | Assignee: | Gregory Thiemonge <gthiemon> |
| Status: | CLOSED ERRATA | QA Contact: | Omer Schwartz <oschwart> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 17.0 (Wallaby) | CC: | astupnik, bbonguar, cmuresan, gthiemon, ifrangs, joflynn, jraju, lpeer, majopela, njohnston, oschwart, rcernin, scohen |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | 17.1 | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-octavia-8.0.2-1.20221208181214.b0379d6.el9ost | Doc Type: | Bug Fix |
| Doc Text: |
Before this update, users might have experienced the following warning message in the amphora log file of the Load-balancing service (octavia) when the load balancer was loaded with multiple concurrent sessions: `nf_conntrack: table full, dropping packet`. This error occurred if the amphora dropped Transport Control Protocol (TCP) flows and caused latency on user traffic. With this update, connection tracking (conntrack) is disabled for TCP flows in the Load-balancing service that uses amphora, and new TCP flows are not dropped. Conntrack is only required for User Datagram Protocol (UDP) flows.
|
Story Points: | --- |
| Clone Of: | 2123226 | Environment: | |
| Last Closed: | 2023-08-16 01:12:09 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2122016, 2123225, 2123226 | ||
| Bug Blocks: | |||
|
Description
Gregory Thiemonge
2022-09-09 13:01:05 UTC
I ran the following verification steps on a SINGLE topology Octavia LB: (overcloud) [stack@undercloud-0 ~]$ cat core_puddle_version RHOS-17.1-RHEL-9-20230131.n.2 (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --vip-subnet external_subnet --name lb1 --wait /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | availability_zone | None | | created_at | 2023-02-21T09:56:42 | | description | | | flavor_id | None | | id | 6c234d54-008d-4966-b2c4-f1bfd8a3d605 | | listeners | | | name | lb1 | | operating_status | ONLINE | | pools | | | project_id | 946cd27e13f14b7395cac4de6dc82abe | | provider | amphora | | provisioning_status | ACTIVE | | updated_at | 2023-02-21T09:57:51 | | vip_address | 10.0.0.159 | | vip_network_id | c0c8a991-388f-447c-9a9a-59d3d0a9290a | | vip_port_id | 4279ca68-48f5-4117-bea9-3b59458576a7 | | vip_qos_policy_id | None | | vip_subnet_id | c8e98308-413b-4a36-898d-7588327f02af | | tags | | +---------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol HTTP --protocol-port 80 --name listener1 lb1 /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +-----------------------------+--------------------------------------+ | Field | Value | +-----------------------------+--------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2023-02-21T09:59:17 | | default_pool_id | None | | default_tls_container_ref | None | | description | | | id | eefd33d3-14a3-4477-b09b-0f15f82dc76b | | insert_headers | None | | l7policies | | | loadbalancers | 6c234d54-008d-4966-b2c4-f1bfd8a3d605 | | name | listener1 | | operating_status | OFFLINE | | project_id | 946cd27e13f14b7395cac4de6dc82abe | | protocol | HTTP | | protocol_port | 80 | | provisioning_status | PENDING_CREATE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | None | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | | tls_ciphers | None | | tls_versions | None | | alpn_protocols | None | | tags | | +-----------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer pool create --protocol HTTP --listener listener1 --lb-algorithm ROUND_ROBIN --name pool1 /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2023-02-21T09:59:22 | | description | | | healthmonitor_id | | | id | 65276094-29b4-4832-b53c-307296d0f8e3 | | lb_algorithm | ROUND_ROBIN | | listeners | eefd33d3-14a3-4477-b09b-0f15f82dc76b | | loadbalancers | 6c234d54-008d-4966-b2c4-f1bfd8a3d605 | | members | | | name | pool1 | | operating_status | OFFLINE | | project_id | 946cd27e13f14b7395cac4de6dc82abe | | protocol | HTTP | | provisioning_status | PENDING_CREATE | | session_persistence | None | | updated_at | None | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | | tls_ciphers | None | | tls_versions | None | | tags | | | alpn_protocols | None | +----------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ for i in {1..500}; do curl 10.0.0.159; done In the same time of cURLing the LB, I ssh the amphora and made sure the conntrack table did not contain any entries: (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list --loadbalancer lb1 +--------------------------------------+--------------------------------------+-----------+------------+---------------+------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+------------+---------------+------------+ | 03deb08b-6062-42fd-b623-43fdbfc3dd78 | 6c234d54-008d-4966-b2c4-f1bfd8a3d605 | ALLOCATED | STANDALONE | 172.24.0.56 | 10.0.0.159 | +--------------------------------------+--------------------------------------+-----------+------------+---------------+------------+ [stack@undercloud-0 ~]$ eval $(ssh-agent) Agent pid 898688 [stack@undercloud-0 ~]$ sudo -E ssh-add /etc/octavia/ssh/octavia_id_rsa Identity added: /etc/octavia/ssh/octavia_id_rsa (root.local) [stack@undercloud-0 ~]$ ssh -A -t tripleo-admin ssh cloud-user.0.56 Warning: Permanently added 'controller-0.ctlplane' (ED25519) to the list of known hosts. [cloud-user@amphora-03deb08b-6062-42fd-b623-43fdbfc3dd78 ~]$ sudo ip netns exec amphora-haproxy cat /proc/net/nf_conntrack [cloud-user@amphora-03deb08b-6062-42fd-b623-43fdbfc3dd78 ~]$ sudo ip netns exec amphora-haproxy cat /proc/net/nf_conntrack [cloud-user@amphora-03deb08b-6062-42fd-b623-43fdbfc3dd78 ~]$ sudo ip netns exec amphora-haproxy cat /proc/net/nf_conntrack [cloud-user@amphora-03deb08b-6062-42fd-b623-43fdbfc3dd78 ~]$ sudo ip netns exec amphora-haproxy cat /proc/net/nf_conntrack [cloud-user@amphora-03deb08b-6062-42fd-b623-43fdbfc3dd78 ~]$ sudo ip netns exec amphora-haproxy cat /proc/net/nf_conntrack [cloud-user@amphora-03deb08b-6062-42fd-b623-43fdbfc3dd78 ~]$ sudo ip netns exec amphora-haproxy cat /proc/net/nf_conntrack [cloud-user@amphora-03deb08b-6062-42fd-b623-43fdbfc3dd78 ~]$ sudo ip netns exec amphora-haproxy cat /proc/net/nf_conntrack [cloud-user@amphora-03deb08b-6062-42fd-b623-43fdbfc3dd78 ~]$ sudo ip netns exec amphora-haproxy cat /proc/net/nf_conntrack Looks good to me. I am moving the BZ status to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577 |