Bug 2078377
| Summary: | UDP listener not reachable with Active-Standby Loadbalancers | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Gregory Thiemonge <gthiemon> |
| Component: | openstack-octavia | Assignee: | Gregory Thiemonge <gthiemon> |
| Status: | CLOSED ERRATA | QA Contact: | Omer Schwartz <oschwart> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 16.1 (Train) | CC: | astupnik, chrisbro, enothen, jelynch, lpeer, majopela, oschwart, scohen |
| Target Milestone: | z9 | Keywords: | Triaged |
| Target Release: | 16.1 (Train on RHEL 8.2) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-octavia-5.0.3-1.20220427074533.8c32d2e.el8ost | Doc Type: | Bug Fix |
| Doc Text: |
Before this update, the Virtual IP (VIP) address of UDP-only load balancers in active-standby mode was not reachable. With this update, the issue is fixed.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-12-07 20:26:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Gregory Thiemonge
2022-04-25 06:58:33 UTC
A workaround exists: if the user creates a TCP/HTTP listener, the UDP listener is "activated" Reproducer and workaround: [stack@undercloud-0 ~]$ sudo dnf install python3-octavia-tests-tempest-golang.x86_64 [stack@undercloud-0 ~]$ sudo iptables -I INPUT -p udp --dport 8080 -j ACCEPT [stack@undercloud-0 ~]$ /usr/libexec/octavia-tempest-plugin-tests-httpd (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --vip-subnet private-subnet --name lb1 (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol-port 1234 --protocol UDP --name listener1 lb1 (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer pool create --lb-algorithm LEAST_CONNECTIONS --listener listener1 --protocol UDP --name pool1 (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer member create --subnet-id external_subnet --address 10.0.0.77 --protocol-port 8080 pool1 (overcloud) [stack@undercloud-0 ~]$ fip_address=$(openstack floating ip create -f value -c floating_ip_address nova) (overcloud) [stack@undercloud-0 ~]$ openstack floating ip set --port $(openstack loadbalancer show -f value -c vip_port_id lb1) $fip_address (overcloud) [stack@undercloud-0 ~]$ nc -u $fip_address 1234 <PRESS ENTER> No data from the UDP server workaround: (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol-port 1234 --protocol HTTP --name listener2 lb1 (overcloud) [stack@undercloud-0 ~]$ nc -u $fip_address 1234 <PRESS ENTER> 1 <PRESS ENTER> 1 We receive data from the UDP server Hotfix verification process:
On one terminal I opened, I ran:
[stack@undercloud-0 ~]$ cat /etc/rhosp-release
Red Hat OpenStack Platform release 16.1.8 (Train)
[stack@undercloud-0 ~]$ sudo dnf install python3-octavia-tests-tempest-golang.x86_64 -y
.
.
.
Installed:
python3-octavia-tests-tempest-golang-1.3.0-1.20210528004836.0ae7f10.el8ost.x86_64
Complete!
[stack@undercloud-0 ~]$ sudo iptables -I INPUT -p udp --dport 8080 -j ACCEPT
(overcloud) [stack@undercloud-0 ~]$ openstack server create --image cirros-0.3.4-x86_64 --security-group sec_group --flavor 1gb --network int_net udp_verification_member2 --key-name test-key1
(overcloud) [stack@undercloud-0 ~]$ openstack floating ip create nova
(overcloud) [stack@undercloud-0 ~]$ openstack server add floating ip udp_verification_member2 10.0.0.162
(overcloud) [stack@undercloud-0 ~]$ scp /usr/libexec/octavia-tempest-plugin-tests-httpd cirros.0.162:/dev/shm/octavia-tempest-plugin-tests-httpd
(overcloud) [stack@undercloud-0 ~]$ ssh cirros.0.162
$ /dev/shm/octavia-tempest-plugin-tests-httpd
(no output should be seen)
# I opened another host's instance on another terminal
# I changed the Octavia topology to ACTIVE_STANDBY
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --vip-subnet int_sub --name lb1
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| created_at | 2022-04-27T11:53:08 |
| description | |
| flavor_id | None |
| id | 2e21e82d-9ed6-4f80-8084-a13a91022f6f |
| listeners | |
| name | lb1 |
| operating_status | OFFLINE |
| pools | |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| provider | amphora |
| provisioning_status | PENDING_CREATE |
| updated_at | None |
| vip_address | 192.168.1.10 |
| vip_network_id | 19a8f0b9-41ea-42f9-9690-907c3f90829e |
| vip_port_id | ad7b6b27-e465-4504-bef0-54ea49f4954d |
| vip_qos_policy_id | None |
| vip_subnet_id | e5e0dc35-fc23-47fb-a285-a79766e3dc3b |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol-port 1234 --protocol UDP --name listener1 lb1
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2022-04-27T11:55:57 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | 8730ed90-9b62-447e-8a1e-f04baa235e28 |
| insert_headers | None |
| l7policies | |
| loadbalancers | 2e21e82d-9ed6-4f80-8084-a13a91022f6f |
| name | listener1 |
| operating_status | OFFLINE |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| protocol | UDP |
| protocol_port | 1234 |
| provisioning_status | PENDING_CREATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | None |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
+-----------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer pool create --lb-algorithm LEAST_CONNECTIONS --listener listener1 --protocol UDP --name pool1
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| admin_state_up | True |
| created_at | 2022-04-27T11:56:02 |
| description | |
| healthmonitor_id | |
| id | a2680bff-6e7c-45d2-b18b-6c5180d76d08 |
| lb_algorithm | LEAST_CONNECTIONS |
| listeners | 8730ed90-9b62-447e-8a1e-f04baa235e28 |
| loadbalancers | 2e21e82d-9ed6-4f80-8084-a13a91022f6f |
| members | |
| name | pool1 |
| operating_status | OFFLINE |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| protocol | UDP |
| provisioning_status | PENDING_CREATE |
| session_persistence | None |
| updated_at | None |
| tls_container_ref | None |
| ca_tls_container_ref | None |
| crl_container_ref | None |
| tls_enabled | False |
+----------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer member create --subnet-id int_sub --address 192.168.1.75 --protocol-port 8080 pool1
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| address | 192.168.1.75 |
| admin_state_up | True |
| created_at | 2022-04-27T11:56:27 |
| id | 2722ed20-076a-4228-8c77-683ebbcf04e8 |
| name | |
| operating_status | NO_MONITOR |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| protocol_port | 8080 |
| provisioning_status | PENDING_CREATE |
| subnet_id | e5e0dc35-fc23-47fb-a285-a79766e3dc3b |
| updated_at | None |
| weight | 1 |
| monitor_port | None |
| monitor_address | None |
| backup | False |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 40cad6c1-ec76-4564-81a1-008379c88b7b | 10.0.0.189 | None | None | 389308e5-31f6-4a00-bb8a-e543939245ab | ec3ebf1fd3144c80bfb12a884fdc7621 |
| aa1e2710-ae31-4555-bdf2-f1f24819aabf | 10.0.0.224 | None | None | 389308e5-31f6-4a00-bb8a-e543939245ab | ec3ebf1fd3144c80bfb12a884fdc7621 |
| f7c8f9ae-a2b5-43e9-8e12-521fabc695cd | 10.0.0.162 | 192.168.1.75 | 2ced98fd-9793-4d50-82f8-a45c205035cc | 389308e5-31f6-4a00-bb8a-e543939245ab | ec3ebf1fd3144c80bfb12a884fdc7621 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack floating ip set --port $(openstack loadbalancer show -f value -c vip_port_id lb1) 10.0.0.189
(overcloud) [stack@undercloud-0 ~]$ nc -u 10.0.0.189 1234
<no response>
^C
#
# Creating an http listener - workaround
#
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol-port 1234 --protocol HTTP --name listener2 lb1
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2022-04-27T11:57:15 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | 50fd14a5-9b76-46fa-9131-b5cdfdbe8083 |
| insert_headers | None |
| l7policies | |
| loadbalancers | 2e21e82d-9ed6-4f80-8084-a13a91022f6f |
| name | listener2 |
| operating_status | OFFLINE |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| protocol | HTTP |
| protocol_port | 1234 |
| provisioning_status | PENDING_CREATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | None |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
+-----------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ nc -u 10.0.0.189 1234
111
1
1^C
#
# Applying hotfix
#
(overcloud) [stack@undercloud-0 ~]$ octavia_version=5.0.3-1.20220427074533.8c32d2e
(overcloud) [stack@undercloud-0 ~]$ rpm -qa | grep octavia-amphora-image
octavia-amphora-image-x86_64-16.1-20220315.1.el8ost.noarch
(overcloud) [stack@undercloud-0 ~]$ sudo dnf install libguestfs-tools -y
.
.
.
Complete!
(overcloud) [stack@undercloud-0 ~]$ rpm -qa | grep octavia-amphora-image
octavia-amphora-image-x86_64-16.1-20220315.1.el8ost.noarch
(overcloud) [stack@undercloud-0 ~]$ amphora_image_version=16.1-20220315.1
(overcloud) [stack@undercloud-0 ~]$ rpm -ql octavia-amphora-image-x86_64-${amphora_image_version}.el8ost.noarch | grep qcow2
/usr/share/openstack-octavia-amphora-images/octavia-amphora-16.1-20220315.1.x86_64.qcow2
(overcloud) [stack@undercloud-0 ~]$ cp /usr/share/openstack-octavia-amphora-images/octavia-amphora-${amphora_image_version}.x86_64.qcow2 .
(overcloud) [stack@undercloud-0 ~]$ virt-customize -a octavia-amphora-${amphora_image_version}.x86_64.qcow2 \
> --copy-in openstack-octavia-amphora-agent-${octavia_version}.el8ost.noarch.rpm:/tmp \
> --copy-in openstack-octavia-common-${octavia_version}.el8ost.noarch.rpm:/tmp \
> --copy-in python3-octavia-${octavia_version}.el8ost.noarch.rpm:/tmp \
> --install /tmp/openstack-octavia-amphora-agent-${octavia_version}.el8ost.noarch.rpm,/tmp/openstack-octavia-common-${octavia_version}.el8ost.noarch.rpm,/tmp/python3-octavia-${octavia_version}.el8ost.noarch.rpm \
> --delete /tmp/openstack-octavia-amphora-agent-${octavia_version}.el8ost.noarch.rpm \
> --delete /tmp/openstack-octavia-common-${octavia_version}.el8ost.noarch.rpm \
> --delete /tmp/python3-octavia-${octavia_version}.el8ost.noarch.rpm \
> --selinux-relabel
[ 0.0] Examining the guest ...
[ 14.4] Setting a random seed
[ 14.4] Setting the machine ID in /etc/machine-id
[ 14.4] Copying: openstack-octavia-amphora-agent-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm to /tmp
[ 14.5] Copying: openstack-octavia-common-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm to /tmp
[ 14.5] Copying: python3-octavia-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm to /tmp
[ 14.5] Installing packages: /tmp/openstack-octavia-amphora-agent-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm /tmp/openstack-octavia-common-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm /tmp/python3-octavia-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm
[ 23.8] Deleting: /tmp/openstack-octavia-amphora-agent-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm
[ 23.8] Deleting: /tmp/openstack-octavia-common-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm
[ 23.8] Deleting: /tmp/python3-octavia-5.0.3-1.20220427074533.8c32d2e.el8ost.noarch.rpm
[ 23.8] SELinux relabelling
[ 57.3] Finishing off
(overcloud) [stack@undercloud-0 ~]$ openstack image create --disk-format qcow2 \
> --container-format bare \
> --project service \
> --tag amphora-image \
> --private \
> --file ./octavia-amphora-${amphora_image_version}.x86_64.qcow2 \
> octavia-amphora-${amphora_image_version}.x86_64.bz2078377
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | 602bec78b5dddbabb24148ecb7e5f9b8 |
| container_format | bare |
| created_at | 2022-04-27T12:05:30Z |
| disk_format | qcow2 |
| file | /v2/images/66678368-ae02-4cd2-9607-4c84eea53315/file |
| id | 66678368-ae02-4cd2-9607-4c84eea53315 |
| min_disk | 0 |
| min_ram | 0 |
| name | octavia-amphora-16.1-20220315.1.x86_64.bz2078377 |
| owner | 6476e6fad3a648a9847154211060da92 |
| properties | direct_url='swift+config://ref1/glance/66678368-ae02-4cd2-9607-4c84eea53315', os_hash_algo='sha512', os_hash_value='e6493a4d337c90baf91ad80439aa506875dbd3b20fed611ef8a6655b1e5ed63be4a1c7c1116492d778b174a6d78ced525e5c858e6e1af7fdbd743b4006944467', os_hidden='False', stores='default_backend' |
| protected | False |
| schema | /v2/schemas/image |
| size | 966131712 |
| status | active |
| tags | amphora-image |
| updated_at | 2022-04-27T12:05:41Z |
| virtual_size | None |
| visibility | private |
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack image unset --tag amphora-image octavia-amphora-${amphora_image_version}.x86_64
#
# Testing hotfix
#
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer delete --cascade lb1
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --vip-subnet int_sub --name lb1
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| created_at | 2022-04-27T12:07:49 |
| description | |
| flavor_id | None |
| id | 0d3a0310-8464-4abd-90aa-e74122f287ca |
| listeners | |
| name | lb1 |
| operating_status | OFFLINE |
| pools | |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| provider | amphora |
| provisioning_status | PENDING_CREATE |
| updated_at | None |
| vip_address | 192.168.1.122 |
| vip_network_id | 19a8f0b9-41ea-42f9-9690-907c3f90829e |
| vip_port_id | 66c15306-e764-4c88-8f00-c5651b17e894 |
| vip_qos_policy_id | None |
| vip_subnet_id | e5e0dc35-fc23-47fb-a285-a79766e3dc3b |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol-port 1234 --protocol UDP --name listener1 lb1
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2022-04-27T12:09:53 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | 15414706-5b31-419d-bd24-1da8341be43b |
| insert_headers | None |
| l7policies | |
| loadbalancers | 0d3a0310-8464-4abd-90aa-e74122f287ca |
| name | listener1 |
| operating_status | OFFLINE |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| protocol | UDP |
| protocol_port | 1234 |
| provisioning_status | PENDING_CREATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | None |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
+-----------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer pool create --lb-algorithm LEAST_CONNECTIONS --listener listener1 --protocol UDP --name pool1
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| admin_state_up | True |
| created_at | 2022-04-27T12:09:59 |
| description | |
| healthmonitor_id | |
| id | 929c52be-9259-4744-a08f-275261b22eb9 |
| lb_algorithm | LEAST_CONNECTIONS |
| listeners | 15414706-5b31-419d-bd24-1da8341be43b |
| loadbalancers | 0d3a0310-8464-4abd-90aa-e74122f287ca |
| members | |
| name | pool1 |
| operating_status | OFFLINE |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| protocol | UDP |
| provisioning_status | PENDING_CREATE |
| session_persistence | None |
| updated_at | None |
| tls_container_ref | None |
| ca_tls_container_ref | None |
| crl_container_ref | None |
| tls_enabled | False |
+----------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer member create --subnet-id int_sub --address 192.168.1.75 --protocol-port 8080 pool1
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| address | 192.168.1.75 |
| admin_state_up | True |
| created_at | 2022-04-27T12:10:06 |
| id | 5f4a88aa-bdf4-440a-aca5-0f5e966f1a13 |
| name | |
| operating_status | NO_MONITOR |
| project_id | ec3ebf1fd3144c80bfb12a884fdc7621 |
| protocol_port | 8080 |
| provisioning_status | PENDING_CREATE |
| subnet_id | e5e0dc35-fc23-47fb-a285-a79766e3dc3b |
| updated_at | None |
| weight | 1 |
| monitor_port | None |
| monitor_address | None |
| backup | False |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack floating ip set --port $(openstack loadbalancer show -f value -c vip_port_id lb1) 10.0.0.189
(overcloud) [stack@undercloud-0 ~]$ nc -u 10.0.0.189 1234
1
1
1^C
# hotfix verified
Verification process appears at comment #7 (https://bugzilla.redhat.com/show_bug.cgi?id=2078377#c7) I am moving the BZ status to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8795 |