Description of problem: haproxy sends a FIN flag after sending the body of huge pages Version-Release number of selected component (if applicable): 17.0 How reproducible: 100% Steps to Reproduce: 1. Deploy OSP17.0 with Octavia 2. Create a LB with a backend member that generates huge buffers (like 64k) 3. Use a benchmark tool that sends multiple requests in the same flow (h2load) Actual results: Only the first request passes haproxy closes the connection after the first reply, the client cannot send the next request. Expected results: Additional info:
$ ssh-keygen -f id_rsa_me $ openstack keypair create --public-key id_rsa_me.pub default $ curl -O https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20220425.0.x86_64.qcow2 $ openstack image create --disk-format qcow2 --container-format bare --public --file CentOS-Stream-GenericCloud-9-20220425.0.x86_64.qcow2 CentOS-Stream-GenericCloud-9-20220425.0.x86_64 $ openstack security group create ssh+http $ openstack security group rule create --ingress --protocol tcp --dst-port 22 ssh+http $ openstack security group rule create --ingress --protocol tcp --dst-port 8000 ssh+http $ openstack flavor create --disk 10 --vcpus 2 --ram 2048 --public m1.small $ openstack server create --image CentOS-Stream-GenericCloud-9-20220425.0.x86_64 --key-name default --security-group ssh+http --flavor m1.small --network private server1 $ openstack floating ip create public [..] | floating_ip_address | 10.0.0.245 | [..] ($ $ openstack router add subnet router1 private-subnet) $ openstack server add floating ip server1 10.0.0.245 $ ssh -i id_rsa_me cloud-user.0.245 [cloud-user@server1 ~]$ sudo dnf install -y epel-release [cloud-user@server1 ~]$ sudo dnf install -y git make gcc gcc-g++ autoconf automake libtool c-ares-devel openssl-devel libev-devel zlib-devel libev-devel [cloud-user@server1 ~]$ git clone https://github.com/nghttp2/nghttp2/ [cloud-user@server1 ~]$ cd nghttp2/ [cloud-user@server1 nghttp2]$ git submodule update --init [cloud-user@server1 nghttp2]$ autoreconf -i [cloud-user@server1 nghttp2]$ automake [cloud-user@server1 nghttp2]$ autoconf [cloud-user@server1 nghttp2]$ ./configure --enable-app [cloud-user@server1 nghttp2]$ make -j2 [cloud-user@server1 nghttp2]$ cd [cloud-user@server1 ~]$ git clone https://github.com/gthiemonge/httpterm [cloud-user@server1 ~]$ cd httpterm [cloud-user@server1 httpterm]$ make -j2 [cloud-user@server1 httpterm]$ sudo ./httpterm -L 0.0.0.0:8000 & [cloud-user@server1 httpterm]$ cd $ openstack loadbalancer create --vip-subnet-id private-subnet --name lb1 $ openstack loadbalancer listener create --protocol HTTP --protocol-port 80 --name listener1 lb1 $ openstack loadbalancer pool create --protocol HTTP --listener listener1 --lb-algorithm ROUND_ROBIN --name pool1 $ openstack loadbalancer member create --address 10.0.0.245 --protocol-port 8000 --subnet private-subnet --name member1 pool1 [cloud-user@server1 ~]$ curl http://<vip_address>/? [cloud-user@server1 ~]$ sudo tcpdump -nn -i eth0 port 80 [cloud-user@server1 ~]$ ./nghttp2/src/h2load -n2 -t1 -c1 --h1 http://10.0.64.41/?s=64k starting benchmark... spawning thread #0: 1 total client(s). 2 total requests Application protocol: http/1.1 progress: 50% done finished in 7.90ms, 126.65 req/s, 7.93MB/s requests: 2 total, 2 started, 1 done, 1 succeeded, 1 failed, 1 errored, 0 timeout status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 64.13KB (65673) total, 106B (106) headers (space savings 0.00%), 64.00KB (65536) data min max mean sd +/- sd time for request: 6.16ms 6.16ms 6.16ms 0us 100.00% time for connect: 1.35ms 1.35ms 1.35ms 0us 100.00% time to 1st byte: 5.30ms 5.30ms 5.30ms 0us 100.00% req/s : 130.31 130.31 130.31 0.00 100.00% Only one request passed successfully tcpdump output: 09:38:57.045052 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [S], seq 3819001560, win 64492, options [mss 1402,sackOK,TS val 2762648593 ecr 0,nop,wscale 7], length 0 09:38:57.046029 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [S.], seq 4050438552, ack 3819001561, win 31970, options [mss 1402,sackOK,TS val 3391116417 ecr 2762648593,nop,wscale 4], length 0 09:38:57.046052 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [.], ack 1, win 504, options [nop,nop,TS val 2762648594 ecr 3391116417], length 0 09:38:57.046232 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [P.], seq 1:82, ack 1, win 504, options [nop,nop,TS val 2762648594 ecr 3391116417], length 81: HTTP: GET /?s=64k HTTP/1.1 09:38:57.049839 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [P.], seq 1:6951, ack 82, win 1994, options [nop,nop,TS val 3391116421 ecr 2762648594], length 6950: HTTP: HTTP/1.1 200 OK 09:38:57.049854 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [.], ack 6951, win 477, options [nop,nop,TS val 2762648598 ecr 3391116421], length 0 09:38:57.050076 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [P.], seq 6951:13901, ack 82, win 1994, options [nop,nop,TS val 3391116421 ecr 2762648594], length 6950: HTTP 09:38:57.050083 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [.], ack 13901, win 477, options [nop,nop,TS val 2762648598 ecr 3391116421], length 0 09:38:57.051034 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [P.], seq 13901:27801, ack 82, win 1994, options [nop,nop,TS val 3391116422 ecr 2762648598], length 13900: HTTP 09:38:57.051044 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [.], ack 27801, win 445, options [nop,nop,TS val 2762648599 ecr 3391116422], length 0 09:38:57.051088 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [.], seq 27801:29191, ack 82, win 1994, options [nop,nop,TS val 3391116422 ecr 2762648598], length 1390: HTTP 09:38:57.051095 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [.], ack 29191, win 435, options [nop,nop,TS val 2762648599 ecr 3391116422], length 0 09:38:57.051166 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [P.], seq 29191:38093, ack 82, win 1994, options [nop,nop,TS val 3391116422 ecr 2762648598], length 8902: HTTP 09:38:57.051173 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [.], ack 38093, win 377, options [nop,nop,TS val 2762648599 ecr 3391116422], length 0 09:38:57.051630 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [P.], seq 38093:53383, ack 82, win 1994, options [nop,nop,TS val 3391116422 ecr 2762648599], length 15290: HTTP 09:38:57.051637 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [.], ack 53383, win 445, options [nop,nop,TS val 2762648599 ecr 3391116422], length 0 09:38:57.052192 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [P.], seq 53383:65674, ack 82, win 1994, options [nop,nop,TS val 3391116423 ecr 2762648599], length 12291: HTTP 09:38:57.052198 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [.], ack 65674, win 445, options [nop,nop,TS val 2762648600 ecr 3391116423], length 0 09:38:57.052247 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [F.], seq 65674, ack 82, win 1994, options [nop,nop,TS val 3391116423 ecr 2762648599], length 0 09:38:57.052420 IP 10.0.64.48.34064 > 10.0.64.41.80: Flags [F.], seq 82, ack 65675, win 502, options [nop,nop,TS val 2762648600 ecr 3391116423], length 0 09:38:57.052812 IP 10.0.64.41.80 > 10.0.64.48.34064: Flags [.], ack 83, win 1994, options [nop,nop,TS val 3391116424 ecr 2762648600], length 0 The flow is established, the request sent, haproxy replies with the payload but it also sends a tcp packet with a FIN flag (at 09:38:57.052247), which means that it closes the connection. It breaks the keep-alive behavior of HTTP/1.1, the client expects that the connection stays open (note: this is an HTTP/1.1-only bug, it doesn't affect HTTPS, encrypted HTTP2 or unencrypted HTTP2) When requesting small payloads (like <15KB), it works correctly. This is a bug in haproxy, it was fixed in https://github.com/haproxy/haproxy/commit/140f1a585248a7da1fefbb5f20f260dfafb32a9a This commit was backported first in 2.4.11 (RHEL 9.0 uses 2.4.7) RHEL 9.1 should have 2.4.17, the problem should be fixed in OSP 17.1
Update: $ openstack loadbalancer member create --address 10.0.0.245 --protocol-port 8000 --subnet private-subnet --name member1 pool1 The address of the member should be the address of server1 on private-subnet (the FIP should work too, but I think it should not be used) In 17.1, haproxy 2.4.17 fixes the issue, tcpdump shows: 09:09:52.332696 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [S], seq 3583115516, win 64492, options [mss 1402,sackOK,TS val 2384885361 ecr 0,nop,wscale 7], length 0 09:09:52.335275 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [S.], seq 1454529294, ack 3583115517, win 43090, options [mss 1402,sackOK,TS val 2330963479 ecr 2384885361,nop,wscale 11], length 0 09:09:52.335369 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 1, win 504, options [nop,nop,TS val 2384885364 ecr 2330963479], length 0 09:09:52.336271 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [P.], seq 1:82, ack 1, win 504, options [nop,nop,TS val 2384885365 ecr 2330963479], length 81: HTTP: GET /?s=64k HTTP/1.1 09:09:52.341162 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 1:6951, ack 82, win 22, options [nop,nop,TS val 2330963485 ecr 2384885365], length 6950: HTTP: HTTP/1.1 200 OK 09:09:52.341224 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 6951, win 479, options [nop,nop,TS val 2384885370 ecr 2330963485], length 0 09:09:52.341258 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 6951:13901, ack 82, win 22, options [nop,nop,TS val 2330963485 ecr 2384885365], length 6950: HTTP 09:09:52.341282 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 13901, win 446, options [nop,nop,TS val 2384885370 ecr 2330963485], length 0 09:09:52.341849 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 13901:27801, ack 82, win 22, options [nop,nop,TS val 2330963486 ecr 2384885370], length 13900: HTTP 09:09:52.341858 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 27801, win 423, options [nop,nop,TS val 2384885371 ecr 2330963486], length 0 09:09:52.341905 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [.], seq 27801:29191, ack 82, win 22, options [nop,nop,TS val 2330963486 ecr 2384885370], length 1390: HTTP 09:09:52.341911 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 29191, win 417, options [nop,nop,TS val 2384885371 ecr 2330963486], length 0 09:09:52.342007 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 29191:41701, ack 82, win 22, options [nop,nop,TS val 2330963486 ecr 2384885370], length 12510: HTTP 09:09:52.342016 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 41701, win 363, options [nop,nop,TS val 2384885371 ecr 2330963486], length 0 09:09:52.342769 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 41701:65674, ack 82, win 22, options [nop,nop,TS val 2330963486 ecr 2384885371], length 23973: HTTP 09:09:52.342779 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 65674, win 411, options [nop,nop,TS val 2384885371 ecr 2330963486], length 0 09:09:52.342985 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [P.], seq 82:163, ack 65674, win 502, options [nop,nop,TS val 2384885372 ecr 2330963486], length 81: HTTP: GET /?s=64k HTTP/1.1 09:09:52.345821 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 65674:79574, ack 163, win 22, options [nop,nop,TS val 2330963489 ecr 2384885372], length 13900: HTTP: HTTP/1.1 200 OK 09:09:52.345833 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 79574, win 453, options [nop,nop,TS val 2384885375 ecr 2330963489], length 0 09:09:52.345913 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 79574:96254, ack 163, win 22, options [nop,nop,TS val 2330963490 ecr 2384885372], length 16680: HTTP 09:09:52.345921 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 96254, win 384, options [nop,nop,TS val 2384885375 ecr 2330963490], length 0 09:09:52.346173 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 96254:121274, ack 163, win 22, options [nop,nop,TS val 2330963490 ecr 2384885372], length 25020: HTTP 09:09:52.346182 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 121274, win 893, options [nop,nop,TS val 2384885375 ecr 2330963490], length 0 09:09:52.346364 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [P.], seq 121274:131347, ack 163, win 22, options [nop,nop,TS val 2330963490 ecr 2384885375], length 10073: HTTP 09:09:52.346373 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 131347, win 1050, options [nop,nop,TS val 2384885375 ecr 2330963490], length 0 09:09:52.346620 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [F.], seq 163, ack 131347, win 1050, options [nop,nop,TS val 2384885375 ecr 2330963490], length 0 09:09:52.347162 IP 10.0.64.29.80 > 10.0.64.15.60436: Flags [F.], seq 131347, ack 164, win 22, options [nop,nop,TS val 2330963491 ecr 2384885375], length 0 09:09:52.347174 IP 10.0.64.15.60436 > 10.0.64.29.80: Flags [.], ack 131348, win 1050, options [nop,nop,TS val 2384885376 ecr 2330963491], length 0
After I ran the verification steps provided in comment 1 and 2, everything worked and it looks good to me. I am moving the status of this BZ to verified. More details about the steps: [stack@undercloud-0 ~]$ cat core_puddle_version RHOS-17.1-RHEL-9-20230216.n.1 (overcloud) [stack@undercloud-0 ~]$ ssh-keygen -f id_rsa_me (overcloud) [stack@undercloud-0 ~]$ openstack keypair create --public-key id_rsa_me.pub default (overcloud) [stack@undercloud-0 ~]$ curl -O https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20220425.0.x86_64.qcow2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 781M 100 781M 0 0 132M 0 0:00:05 0:00:05 --:--:-- 141M (overcloud) [stack@undercloud-0 ~]$ openstack image create --disk-format qcow2 --container-format bare --public --file CentOS-Stream-GenericCloud-9-20220425.0.x86_64.qcow2 CentOS-Stream-GenericCloud-9-20220425.0.x86_64 +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | container_format | bare | | created_at | 2023-03-07T11:29:12Z | | disk_format | qcow2 | | file | /v2/images/b9bfb772-2933-4664-ae1c-cd0027feed41/file | | id | b9bfb772-2933-4664-ae1c-cd0027feed41 | | min_disk | 0 | | min_ram | 0 | | name | CentOS-Stream-GenericCloud-9-20220425.0.x86_64 | | owner | c12c21f13ff7428a988219a758590296 | | properties | os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/CentOS-Stream-GenericCloud-9-20220425.0.x86_64', owner_specified.openstack.sha256='' | | protected | False | | schema | /v2/schemas/image | | status | queued | | tags | | | updated_at | 2023-03-07T11:29:12Z | | visibility | public | +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack security group create ssh+http +-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | created_at | 2023-03-07T11:29:36Z | | description | ssh+http | | id | 8af20a6f-7315-41ab-99bf-ee8bc7b91191 | | name | ssh+http | | project_id | c12c21f13ff7428a988219a758590296 | | revision_number | 1 | | rules | created_at='2023-03-07T11:29:36Z', direction='egress', ethertype='IPv4', id='8f29d066-b8d2-47cf-ba71-2e0090068280', standard_attr_id='503', updated_at='2023-03-07T11:29:36Z' | | | created_at='2023-03-07T11:29:36Z', direction='egress', ethertype='IPv6', id='fe7fcff8-4271-498b-a836-3adc92faa0ba', standard_attr_id='506', updated_at='2023-03-07T11:29:36Z' | | stateful | True | | tags | [] | | updated_at | 2023-03-07T11:29:36Z | +-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack security group rule create --ingress --protocol tcp --dst-port 22 ssh+http +-------------------------+--------------------------------------+ | Field | Value | +-------------------------+--------------------------------------+ | created_at | 2023-03-07T11:29:40Z | | description | | | direction | ingress | | ether_type | IPv4 | | id | 96c1882d-ade7-454d-a1e2-639f3e34ee51 | | name | None | | port_range_max | 22 | | port_range_min | 22 | | project_id | c12c21f13ff7428a988219a758590296 | | protocol | tcp | | remote_address_group_id | None | | remote_group_id | None | | remote_ip_prefix | 0.0.0.0/0 | | revision_number | 0 | | security_group_id | 8af20a6f-7315-41ab-99bf-ee8bc7b91191 | | tags | [] | | updated_at | 2023-03-07T11:29:40Z | +-------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack security group rule create --ingress --protocol tcp --dst-port 8000 ssh+http +-------------------------+--------------------------------------+ | Field | Value | +-------------------------+--------------------------------------+ | created_at | 2023-03-07T11:29:44Z | | description | | | direction | ingress | | ether_type | IPv4 | | id | 42fe1f22-ae7d-49be-a314-7587daaef869 | | name | None | | port_range_max | 8000 | | port_range_min | 8000 | | project_id | c12c21f13ff7428a988219a758590296 | | protocol | tcp | | remote_address_group_id | None | | remote_group_id | None | | remote_ip_prefix | 0.0.0.0/0 | | revision_number | 0 | | security_group_id | 8af20a6f-7315-41ab-99bf-ee8bc7b91191 | | tags | [] | | updated_at | 2023-03-07T11:29:44Z | +-------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack flavor create --disk 10 --vcpus 2 --ram 2048 --public m1.small /usr/lib/python3.9/site-packages/openstack/config/cloud_region.py:452: UserWarning: You have a configured API_VERSION with 'latest' in it. In the context of openstacksdk this doesn't make any sense. warnings.warn( +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | description | None | | disk | 10 | | id | 54665664-cb39-4d96-8e04-8100e0c1f5a4 | | name | m1.small | | os-flavor-access:is_public | True | | properties | | | ram | 2048 | | rxtx_factor | 1.0 | | swap | | | vcpus | 2 | +----------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack server create --image CentOS-Stream-GenericCloud-9-20220425.0.x86_64 --key-name default --security-group ssh+http --flavor m1.small --network private server1 +-------------------------------------+---------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+---------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hostname | server1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-pclbi5wr | | OS-EXT-SRV-ATTR:root_device_name | None | | OS-EXT-SRV-ATTR:user_data | None | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | 7xie8oPUgXHk | | config_drive | | | created | 2023-03-07T11:29:58Z | | description | None | | flavor | disk='10', ephemeral='0', , original_name='m1.small', ram='2048', swap='0', vcpus='2' | | hostId | | | host_status | | | id | 6d09fe02-cbe3-4ed0-81ad-3694ce0b1f9b | | image | CentOS-Stream-GenericCloud-9-20220425.0.x86_64 (b9bfb772-2933-4664-ae1c-cd0027feed41) | | key_name | default | | locked | False | | locked_reason | None | | name | server1 | | progress | 0 | | project_id | c12c21f13ff7428a988219a758590296 | | properties | | | security_groups | name='8af20a6f-7315-41ab-99bf-ee8bc7b91191' | | server_groups | [] | | status | BUILD | | tags | | | trusted_image_certificates | None | | updated | 2023-03-07T11:29:58Z | | user_id | 1628c765b1f24f76ba718c7f98cadfcc | | volumes_attached | | +-------------------------------------+---------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack floating ip create public +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | created_at | 2023-03-07T11:30:04Z | | description | | | dns_domain | | | dns_name | | | fixed_ip_address | None | | floating_ip_address | 10.0.0.191 | | floating_network_id | 28310332-9a40-4d85-8842-baa997e3d1d8 | | id | 1d54e813-e341-4f27-8a46-b3a6fb62d169 | | name | 10.0.0.191 | | port_details | None | | port_id | None | | project_id | c12c21f13ff7428a988219a758590296 | | qos_policy_id | None | | revision_number | 0 | | router_id | None | | status | DOWN | | subnet_id | None | | tags | [] | | updated_at | 2023-03-07T11:30:04Z | +---------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack router add subnet router1 private-subnet (overcloud) [stack@undercloud-0 ~]$ openstack server add floating ip server1 10.0.0.191 (overcloud) [stack@undercloud-0 ~]$ ssh -i id_rsa_me cloud-user.0.191 Warning: Permanently added '10.0.0.191' (ED25519) to the list of known hosts. [cloud-user@server1 ~]$ sudo dnf install -y epel-release ... ... Installed: epel-next-release-9-2.el9.noarch epel-release-9-2.el9.noarch Complete! [cloud-user@server1 ~]$ sudo dnf install -y git make gcc gcc-g++ autoconf automake libtool c-ares-devel openssl-devel libev-devel zlib-devel libev-devel ... ... Complete! [cloud-user@server1 ~]$ git clone https://github.com/nghttp2/nghttp2/ Cloning into 'nghttp2'... remote: Enumerating objects: 44856, done. remote: Counting objects: 100% (49/49), done. remote: Compressing objects: 100% (35/35), done. remote: Total 44856 (delta 12), reused 43 (delta 11), pack-reused 44807 Receiving objects: 100% (44856/44856), 35.19 MiB | 23.02 MiB/s, done. Resolving deltas: 100% (33435/33435), done. [cloud-user@server1 ~]$ cd nghttp2 [cloud-user@server1 nghttp2]$ git submodule update --init Submodule 'third-party/mruby' (https://github.com/mruby/mruby) registered for path 'third-party/mruby' Submodule 'third-party/neverbleed' (https://github.com/tatsuhiro-t/neverbleed.git) registered for path 'third-party/neverbleed' Cloning into '/home/cloud-user/nghttp2/third-party/mruby'... Cloning into '/home/cloud-user/nghttp2/third-party/neverbleed'... Submodule path 'third-party/mruby': checked out '87260e7bb1a9edfb2ce9b41549c4142129061ca5' Submodule path 'third-party/neverbleed': checked out 'f31bf05f792b0c41de8f774c68d278df2067aaf8' [cloud-user@server1 nghttp2]$ autoreconf -i libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' configure.ac:32: installing './compile' configure.ac:35: installing './config.guess' configure.ac:35: installing './config.sub' configure.ac:41: installing './install-sh' configure.ac:41: installing './missing' Makefile.am: installing './INSTALL' examples/Makefile.am: installing './depcomp' parallel-tests: installing './test-driver' [cloud-user@server1 nghttp2]$ automake [cloud-user@server1 nghttp2]$ autoconf [cloud-user@server1 nghttp2]$ ./configure --enable-app ... ... [cloud-user@server1 nghttp2]$ make -j2 ... [cloud-user@server1 nghttp2]$ cd [cloud-user@server1 ~]$ git clone https://github.com/gthiemonge/httpterm Cloning into 'httpterm'... remote: Enumerating objects: 1501, done. remote: Counting objects: 100% (1501/1501), done. remote: Compressing objects: 100% (536/536), done. remote: Total 1501 (delta 948), reused 1494 (delta 941), pack-reused 0 Receiving objects: 100% (1501/1501), 649.87 KiB | 6.63 MiB/s, done. Resolving deltas: 100% (948/948), done. [cloud-user@server1 ~]$ cd httpterm/ [cloud-user@server1 httpterm]$ make -j2 gcc -Wall -I. -O2 -DENABLE_POLL -DENABLE_EPOLL -DENABLE_SPLICE -DENABLE_ACCEPT4 -g -c -o httpterm.o httpterm.c gcc -g -o httpterm httpterm.o [cloud-user@server1 httpterm]$ sudo ./httpterm -L 0.0.0.0:8000 & [1] 25736 [cloud-user@server1 httpterm]$ cd Meanwhile, on the undercloud: (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --vip-subnet-id private-subnet --name lb1 wa/usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | availability_zone | None | | created_at | 2023-03-07T11:43:11 | | description | | | flavor_id | None | | id | 8e19e89b-2c8f-4845-adde-e4cce342735f | | listeners | | | name | lb1 | | operating_status | OFFLINE | | pools | | | project_id | c12c21f13ff7428a988219a758590296 | | provider | amphora | | provisioning_status | PENDING_CREATE | | updated_at | None | | vip_address | 10.0.64.19 | | vip_network_id | b3cd6c88-0e46-4e87-a082-a7bb8f3b845d | | vip_port_id | 704ada2d-8654-4d98-ae52-c0ef781bcd12 | | vip_qos_policy_id | None | | vip_subnet_id | af703376-4246-4a68-a05c-4f8f7fa6fb2c | | tags | | +---------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol HTTP --protocol-port 80 --name listener1 lb1 /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +-----------------------------+--------------------------------------+ | Field | Value | +-----------------------------+--------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2023-03-07T11:44:52 | | default_pool_id | None | | default_tls_container_ref | None | | description | | | id | 666ef01a-9e44-43fa-9db7-ec09f9c8fdf6 | | insert_headers | None | | l7policies | | | loadbalancers | 8e19e89b-2c8f-4845-adde-e4cce342735f | | name | listener1 | | operating_status | OFFLINE | | project_id | c12c21f13ff7428a988219a758590296 | | protocol | HTTP | | protocol_port | 80 | | provisioning_status | PENDING_CREATE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | None | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | | tls_ciphers | None | | tls_versions | None | | alpn_protocols | None | | tags | | +-----------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer pool create --protocol HTTP --listener listener1 --lb-algorithm ROUND_ROBIN --name pool1 /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2023-03-07T11:44:57 | | description | | | healthmonitor_id | | | id | c35441ad-6bb1-4ffe-8ecb-e4026c36a451 | | lb_algorithm | ROUND_ROBIN | | listeners | 666ef01a-9e44-43fa-9db7-ec09f9c8fdf6 | | loadbalancers | 8e19e89b-2c8f-4845-adde-e4cce342735f | | members | | | name | pool1 | | operating_status | OFFLINE | | project_id | c12c21f13ff7428a988219a758590296 | | protocol | HTTP | | provisioning_status | PENDING_CREATE | | session_persistence | None | | updated_at | None | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | | tls_ciphers | None | | tls_versions | None | | tags | | | alpn_protocols | None | +----------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer member create --address 10.0.64.15 --protocol-port 8000 --subnet private-subnet --name member1 pool1 /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | address | 10.0.64.15 | | admin_state_up | True | | created_at | 2023-03-07T11:45:45 | | id | 3fa02ae4-0859-46a6-9732-3f4e45b8d0e8 | | name | member1 | | operating_status | NO_MONITOR | | project_id | c12c21f13ff7428a988219a758590296 | | protocol_port | 8000 | | provisioning_status | PENDING_CREATE | | subnet_id | af703376-4246-4a68-a05c-4f8f7fa6fb2c | | updated_at | None | | weight | 1 | | monitor_port | None | | monitor_address | None | | backup | False | | tags | | +---------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer show lb1 -f value -c vip_address /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( 10.0.64.19 Back on the server [cloud-user@server1 ~]$ sudo tcpdump -nn -i eth0 port 80 And when I ran: [cloud-user@server1 ~]$ ./nghttp2/src/h2load -n2 -t1 -c1 --h1 http://10.0.64.19/?s=64k starting benchmark... spawning thread #0: 1 total client(s). 2 total requests Application protocol: http/1.1 progress: 50% done progress: 100% done finished in 11.52ms, 173.58 req/s, 10.87MB/s requests: 2 total, 2 started, 2 done, 2 succeeded, 0 failed, 0 errored, 0 timeout status codes: 2 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 128.27KB (131346) total, 212B (212) headers (space savings 0.00%), 128.00KB (131072) data min max mean sd +/- sd time for request: 2.04ms 5.76ms 3.90ms 2.62ms 100.00% time for connect: 3.23ms 3.23ms 3.23ms 0us 100.00% time to 1st byte: 7.72ms 7.72ms 7.72ms 0us 100.00% req/s : 178.46 178.46 178.46 0.00 100.00% I got on tcpdump the following: 06:49:14.699483 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [S], seq 989823094, win 64492, options [mss 1402,sackOK,TS val 4010852715 ecr 0,nop,wscale 7], length 0 06:49:14.702431 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [S.], seq 2748759552, ack 989823095, win 43090, options [mss 1402,sackOK,TS val 2430283288 ecr 4010852715,nop,wscale 11], length 0 06:49:14.702483 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 1, win 504, options [nop,nop,TS val 4010852718 ecr 2430283288], length 0 06:49:14.702765 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [P.], seq 1:82, ack 1, win 504, options [nop,nop,TS val 4010852718 ecr 2430283288], length 81: HTTP: GET /?s=64k HTTP/1.1 06:49:14.706984 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 1:6951, ack 82, win 22, options [nop,nop,TS val 2430283294 ecr 4010852718], length 6950: HTTP: HTTP/1.1 200 OK 06:49:14.707039 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 6951, win 471, options [nop,nop,TS val 4010852723 ecr 2430283294], length 0 06:49:14.707044 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 6951:13901, ack 82, win 22, options [nop,nop,TS val 2430283294 ecr 4010852718], length 6950: HTTP 06:49:14.707049 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 13901, win 429, options [nop,nop,TS val 4010852723 ecr 2430283294], length 0 06:49:14.707790 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 13901:27801, ack 82, win 22, options [nop,nop,TS val 2430283294 ecr 4010852723], length 13900: HTTP 06:49:14.707795 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 27801, win 445, options [nop,nop,TS val 4010852723 ecr 2430283294], length 0 06:49:14.707832 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [.], seq 27801:29191, ack 82, win 22, options [nop,nop,TS val 2430283294 ecr 4010852723], length 1390: HTTP 06:49:14.707836 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 29191, win 435, options [nop,nop,TS val 4010852724 ecr 2430283294], length 0 06:49:14.707866 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 29191:41701, ack 82, win 22, options [nop,nop,TS val 2430283295 ecr 4010852723], length 12510: HTTP 06:49:14.707881 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 41701, win 697, options [nop,nop,TS val 4010852724 ecr 2430283295], length 0 06:49:14.708345 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 41701:65674, ack 82, win 22, options [nop,nop,TS val 2430283295 ecr 4010852724], length 23973: HTTP 06:49:14.708351 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 65674, win 1030, options [nop,nop,TS val 4010852724 ecr 2430283295], length 0 06:49:14.708471 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [P.], seq 82:163, ack 65674, win 1030, options [nop,nop,TS val 4010852724 ecr 2430283295], length 81: HTTP: GET /?s=64k HTTP/1.1 06:49:14.709866 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 65674:79574, ack 163, win 22, options [nop,nop,TS val 2430283296 ecr 4010852724], length 13900: HTTP: HTTP/1.1 200 OK 06:49:14.709875 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 79574, win 1075, options [nop,nop,TS val 4010852726 ecr 2430283296], length 0 06:49:14.709937 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 79574:96254, ack 163, win 22, options [nop,nop,TS val 2430283297 ecr 4010852724], length 16680: HTTP 06:49:14.709941 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 96254, win 1003, options [nop,nop,TS val 4010852726 ecr 2430283297], length 0 06:49:14.709974 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 96254:108682, ack 163, win 22, options [nop,nop,TS val 2430283297 ecr 4010852724], length 12428: HTTP 06:49:14.709980 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 108682, win 951, options [nop,nop,TS val 4010852726 ecr 2430283297], length 0 06:49:14.710393 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [P.], seq 108682:131347, ack 163, win 22, options [nop,nop,TS val 2430283297 ecr 4010852726], length 22665: HTTP 06:49:14.710399 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 131347, win 1041, options [nop,nop,TS val 4010852726 ecr 2430283297], length 0 06:49:14.710590 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [F.], seq 163, ack 131347, win 1075, options [nop,nop,TS val 4010852726 ecr 2430283297], length 0 06:49:14.711149 IP 10.0.64.19.80 > 10.0.64.15.53576: Flags [F.], seq 131347, ack 164, win 22, options [nop,nop,TS val 2430283298 ecr 4010852726], length 0 06:49:14.711157 IP 10.0.64.15.53576 > 10.0.64.19.80: Flags [.], ack 131348, win 1075, options [nop,nop,TS val 4010852727 ecr 2430283298], length 0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577