Description of problem: In an OCP 4.13 on OSP 17.1 deployment, when creating a LB type UDP service in OCP with ovn-octavia provider and monitors enabled, the corresponding LB is created in Openstack but it ends up in DEGRADED operating_status (as it's loadbalancer pool) and the traffic from the LB is sent to all the members instead of only to the ONLINE ones. $ openstack loadbalancer show d2922cf3-e9a7-4230-821a-7dcab7190192 /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +---------------------+--------------------------------------------------------------------------------------------+ | Field | Value | +---------------------+--------------------------------------------------------------------------------------------+ | admin_state_up | True | | availability_zone | None | | created_at | 2023-07-28T07:57:12 | | description | Kubernetes external service udp-lb-etplocal-ns/udp-lb-etplocal-svc from cluster kubernetes | | flavor_id | None | | id | d2922cf3-e9a7-4230-821a-7dcab7190192 | | listeners | 82a581a9-7b42-4962-a685-c897db5c6b9b | | name | kube_service_kubernetes_udp-lb-etplocal-ns_udp-lb-etplocal-svc | | operating_status | DEGRADED | | pools | bf4dc1b0-d7bb-4319-bc2f-74c3c6597f04 | | project_id | 3674d08c0f4546b495677e0bbf046bd8 | | provider | ovn | | provisioning_status | ACTIVE | | updated_at | 2023-07-28T07:57:35 | | vip_address | 10.196.1.148 | | vip_network_id | ca0fed66-e69c-492e-99fd-d50aea240e6f | | vip_port_id | 11778e21-f2dc-4c78-a05b-2409753f92ff | | vip_qos_policy_id | None | | vip_subnet_id | 11f3cf7b-1bb5-4098-bc77-8717ef0727fe | | tags | kube_service_kubernetes_udp-lb-etplocal-ns_udp-lb-etplocal-svc | +---------------------+--------------------------------------------------------------------------------------------+ $ openstack loadbalancer pool show bf4dc1b0-d7bb-4319-bc2f-74c3c6597f04 /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +----------------------+-----------------------------------------------------------------------+ | Field | Value | +----------------------+-----------------------------------------------------------------------+ | admin_state_up | True | | created_at | 2023-07-28T07:57:13 | | description | | | healthmonitor_id | 9dda0ee8-7db1-41d6-888c-2b173bc28a1c | | id | bf4dc1b0-d7bb-4319-bc2f-74c3c6597f04 | | lb_algorithm | SOURCE_IP_PORT | | listeners | 82a581a9-7b42-4962-a685-c897db5c6b9b | | loadbalancers | d2922cf3-e9a7-4230-821a-7dcab7190192 | | members | 13ffa095-c8df-4e68-8f75-5fba2af97bc7 | | | 2771ee5c-9eab-4c33-8110-4823e00fad55 | | | 400768e1-9df8-42e4-a6d1-c8aaa8815e99 | | | 8eb654ab-8334-4d1c-9413-1e07f5d9c0f7 | | | ae708565-761d-46d9-9514-8b3954e488cc | | | da714d19-fef0-43a4-9d0c-69e1ce040b90 | | name | pool_0_kube_service_kubernetes_udp-lb-etplocal-ns_udp-lb-etplocal-svc | | operating_status | DEGRADED | | project_id | 3674d08c0f4546b495677e0bbf046bd8 | | protocol | UDP | | provisioning_status | ACTIVE | | session_persistence | None | | updated_at | 2023-07-28T07:57:35 | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | | tls_ciphers | None | | tls_versions | None | | tags | | | alpn_protocols | None | +----------------------+-----------------------------------------------------------------------+ $ openstack loadbalancer healthmonitor show 9dda0ee8-7db1-41d6-888c-2b173bc28a1c /usr/lib/python3.9/site-packages/osc_lib/utils/__init__.py:448: DeprecationWarning: The usage of formatter functions is now discouraged. Consider using cliff.columns.FormattableColumn instead. See reviews linked with bug 1687955 for more detail. warnings.warn( +---------------------+-----------------------------------------------------------------------------+ | Field | Value | +---------------------+-----------------------------------------------------------------------------+ | project_id | 3674d08c0f4546b495677e0bbf046bd8 | | name | monitor_8082_kube_service_kubernetes_udp-lb-etplocal-ns_udp-lb-etplocal-svc | | admin_state_up | True | | pools | bf4dc1b0-d7bb-4319-bc2f-74c3c6597f04 | | created_at | 2023-07-28T07:57:13 | | provisioning_status | ACTIVE | | updated_at | 2023-07-28T07:57:15 | | delay | 5 | | expected_codes | None | | max_retries | 2 | | http_method | None | | timeout | 5 | | max_retries_down | 3 | | url_path | None | | type | UDP-CONNECT | | id | 9dda0ee8-7db1-41d6-888c-2b173bc28a1c | | operating_status | ONLINE | | http_version | None | | domain_name | None | | tags | | +---------------------+-----------------------------------------------------------------------------+ $ openstack loadbalancer member list bf4dc1b0-d7bb-4319-bc2f-74c3c6597f04 +--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+ | 13ffa095-c8df-4e68-8f75-5fba2af97bc7 | ostest-c7zn4-master-1 | 3674d08c0f4546b495677e0bbf046bd8 | ACTIVE | 10.196.0.143 | 32664 | ERROR | 1 | | 2771ee5c-9eab-4c33-8110-4823e00fad55 | ostest-c7zn4-worker-0-hpjlx | 3674d08c0f4546b495677e0bbf046bd8 | ACTIVE | 10.196.3.225 | 32664 | ERROR | 1 | | 400768e1-9df8-42e4-a6d1-c8aaa8815e99 | ostest-c7zn4-worker-0-5mldk | 3674d08c0f4546b495677e0bbf046bd8 | ACTIVE | 10.196.0.93 | 32664 | ONLINE | 1 | | 8eb654ab-8334-4d1c-9413-1e07f5d9c0f7 | ostest-c7zn4-master-0 | 3674d08c0f4546b495677e0bbf046bd8 | ACTIVE | 10.196.0.71 | 32664 | ERROR | 1 | | ae708565-761d-46d9-9514-8b3954e488cc | ostest-c7zn4-worker-0-zqcrk | 3674d08c0f4546b495677e0bbf046bd8 | ACTIVE | 10.196.3.246 | 32664 | ONLINE | 1 | | da714d19-fef0-43a4-9d0c-69e1ce040b90 | ostest-c7zn4-master-2 | 3674d08c0f4546b495677e0bbf046bd8 | ACTIVE | 10.196.2.186 | 32664 | ERROR | 1 | +--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+ This behavior makes the test "[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should apply lb-method on UDP OVN LoadBalancer when an UDP svc with monitors and ETP:Local is created on Openshift" [1] fail in OCP. Version-Release number of selected component (if applicable): OSP 17.1.0 (RHOS-17.1-RHEL-9-20230719.n.1) OCP 4.13.0-0.nightly-2023-07-27-013427 How reproducible: always Steps to Reproduce: 1. Deploy OCP on OSP 17.1 2. Create a project, deployment and a UDP LB type svc (it has monitors and ETP:local in svc definition) cat <<EOF | oc apply -f - --- apiVersion: project.openshift.io/v1 kind: Project metadata: name: udp-lb-etplocal-ns labels: kubernetes.io/metadata.name: udp-lb-etplocal-ns --- apiVersion: apps/v1 kind: Deployment metadata: name: udp-lb-etplocal-dep namespace: udp-lb-etplocal-ns labels: app: udp-lb-etplocal-dep spec: replicas: 2 selector: matchLabels: app: udp-lb-etplocal-dep template: metadata: labels: app: udp-lb-etplocal-dep spec: containers: - name: udp-test securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true seccompProfile: type: RuntimeDefault image: k8s.gcr.io/e2e-test-images/agnhost:2.43 args: - netexec - --udp-port=8081 ports: - containerPort: 8081 protocol: UDP --- apiVersion: v1 kind: Service metadata: name: udp-lb-etplocal-svc namespace: udp-lb-etplocal-ns labels: app: udp-lb-etplocal-dep annotations: loadbalancer.openstack.org/enable-health-monitor: "true" loadbalancer.openstack.org/health-monitor-delay: "5" loadbalancer.openstack.org/health-monitor-max-retries: "2" loadbalancer.openstack.org/health-monitor-timeout: "5" spec: ports: - port: 8082 targetPort: 8081 protocol: UDP selector: app: udp-lb-etplocal-dep type: LoadBalancer externalTrafficPolicy: Local EOF 3. Install nc $ sudo yum install nmap-ncat 4. Test the svc $ for i in {1..100}; do cat <(echo hostname) <(sleep 1) | nc -w 1 -u <servicd FIP> 8082; echo; done > /tmp/result.txt && cat /tmp/result.txt | sort | uniq -c Actual results: traffic from the LB is sent to all the pool members and some requests are lost Expected results: traffic from the LB only sent to the ONLINE pool members Additional info: $ metalsmith list +--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+ | UUID | Node Name | Allocation UUID | Hostname | State | IP Addresses | +--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+ | 4313377f-b172-4385-adb0-76933ca27c50 | compute-0 | 0a0dfabc-ccfc-454f-96b6-da5c8957d63b | compute-0 | ACTIVE | ctlplane=192.168.24.37 | | cc85ca1c-c309-4bad-bfed-62983c881fc6 | controller-0 | 7f9988db-6bb7-4098-b3d5-9ed567121118 | controller-0 | ACTIVE | ctlplane=192.168.24.20 | | 6b993b79-7403-45ef-a889-753c31b49d13 | controller-1 | da2308f8-598e-49bc-8359-a6e03b565072 | controller-1 | ACTIVE | ctlplane=192.168.24.49 | | 8d82aadf-2c0b-4bb7-8991-2d335d83c139 | controller-2 | 5f8dfc5c-d833-45d2-8ce1-018ca82b178b | controller-2 | ACTIVE | ctlplane=192.168.24.40 | +--------------------------------------+--------------+--------------------------------------+--------------+--------+------------------------+ $ oc -n openshift-config get cm cloud-provider-config -o yaml apiVersion: v1 data: [...] config: | [Global] secret-name = openstack-credentials secret-namespace = kube-system region = regionOne ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem [LoadBalancer] lb-provider = ovn lb-method = SOURCE_IP_PORT floating-network-id = xx subnet-id = xx create-monitor = False monitor-delay = 10s monitor-timeout = 10s monitor-max-retries = 1 kind: ConfigMap [...] [1] https://github.com/openshift/openstack-test/blob/b2f8871fe72c24285c4cc22dd4491ea0e609c492/test/extended/openstack/loadbalancers.go#L245C4-L245C4
This issue affects LB type services (with External Traffic Policy set to local) [1] and LB type ingress controllers or routes when using octavia-ovn provider in OCP 4.13 and OCP 4.14, which are the supported versions for OSP 17.1. [1] https://docs.openshift.com/container-platform/4.13/networking/load-balancing-openstack.html#nw-osp-loadbalancer-etp-local_load-balancing-openstack
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:5138