Bug 2100902 - With OVN provider bulk member update returns 500
Summary: With OVN provider bulk member update returns 500
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z9
: 16.1 (Train on RHEL 8.2)
Assignee: Fernando Royo
QA Contact: Fiorella Yanac
URL:
Whiteboard:
Depends On: 2100135
Blocks: 2082496
TreeView+ depends on / blocked
 
Reported: 2022-06-24 15:51 UTC by Fernando Royo
Modified: 2022-12-07 20:27 UTC (History)
12 users (show)

Fixed In Version: python-networking-ovn-7.3.1-1.20220808133333.4e24f4c.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of: 2100135
Environment:
Last Closed: 2022-12-07 20:27:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 847255 0 None MERGED Revert "Fix member_batch_update function to follow api v2" 2022-06-24 15:55:29 UTC
Red Hat Issue Tracker OSP-15998 0 None None None 2022-06-24 15:57:24 UTC
Red Hat Product Errata RHBA-2022:8795 0 None None None 2022-12-07 20:27:52 UTC

Comment 5 rlobillo 2022-11-14 16:01:47 UTC
Verified on RHOS-16.1-RHEL-8-20221108.n.1 using 4.12.0-rc.0


Creating OCP loadbalancer with OVN provider enabled and then update the number of workers so a change in the pool members is triggered by CCM does not trigger an exception on octavia_api container in controllers anymore. The pool remains operative and the functionality is correctly provided.


1. Enable OVN provider on OCP:

	$ oc patch cm/cloud-provider-config -n openshift-config --patch-file cloud-provider-config.patch.yaml

and wait until change is applied.


2. Create a namespace with some pods and a loadbalancer service:

	$ oc apply -f default-manifests.yaml


3. Check that the loadbalancer is created in openstack and it is functioning:

	$ openstack loadbalancer list
	+--------------------------------------+------------------------------------------------------------------+----------------------------------+--------------+---------------------+----------+| id                                   | name                                                             | project_id                       | vip_address  | provisioning_status | provider |
	+--------------------------------------+------------------------------------------------------------------+----------------------------------+--------------+---------------------+----------+| 128be30e-b7ef-4fc5-86ec-96cebb3c7719 | kube_service_kubernetes_udp-lb-default-ovn-ns_udp-lb-default-svc | 8482c60b04a64342b3b8c575b085ba31 | 10.196.2.185 | ACTIVE              | ovn      |+--------------------------------------+------------------------------------------------------------------+----------------------------------+--------------+---------------------+----------+
	$ openstack loadbalancer pool list --loadbalancer kube_service_kubernetes_udp-lb-default-ovn-ns_udp-lb-default-svc          
	+--------------------------------------+-------------------------------------------------------------------------+----------------------------------+---------------------+----------+----------------+----------------+
	| id                                   | name                                                                    | project_id                       | provisioning_status | protocol | lb_algorithm   | admin_state_up |
	+--------------------------------------+-------------------------------------------------------------------------+----------------------------------+---------------------+----------+----------------+----------------+
	| dd8afd57-5bd7-4e25-a871-87b6c19c0f6b | pool_0_kube_service_kubernetes_udp-lb-default-ovn-ns_udp-lb-default-svc | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | UDP      | SOURCE_IP_PORT | True           |
	+--------------------------------------+-------------------------------------------------------------------------+----------------------------------+---------------------+----------+----------------+----------------+
	$ openstack loadbalancer member list pool_0_kube_service_kubernetes_udp-lb-default-ovn-ns_udp-lb-default-svc                
	+--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+
	| id                                   | name                        | project_id                       | provisioning_status | address      | protocol_port | operating_status | weight |
	+--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+
	| 28cf3a39-05bb-4bcf-bd88-384d18aa77d9 | ostest-j6kjf-master-0       | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.1.58  |         31670 | ONLINE           |      1 |
	| 3341fb36-14a3-42ac-814c-cace8a702cf7 | ostest-j6kjf-worker-0-9dht7 | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.0.13  |         31670 | ONLINE           |      1 |
	| 74539bd4-9dfd-4b8e-85cb-10f23c915b3a | ostest-j6kjf-master-2       | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.3.100 |         31670 | ONLINE           |      1 |
	| 97f17184-6040-4ee6-967e-ad6fb151f4f0 | ostest-j6kjf-master-1       | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.0.173 |         31670 | ONLINE           |      1 |
	| ccf76f68-24b3-4dbb-90a8-ca8b4ff49d7d | ostest-j6kjf-worker-0-t7ll2 | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.2.157 |         31670 | ONLINE           |      1 |
	+--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+
	(shiftstack) [stack@undercloud-0 [OVN][UDP]default_configuration]$ 

	$ cat <(echo hostname) <(sleep 1) | nc -w 1 -u 10.46.43.102 8082
	udp-lb-default-dep-7b8784986b-bzx4z
	$ cat <(echo hostname) <(sleep 1) | nc -w 1 -u 10.46.43.102 8082
	udp-lb-default-dep-7b8784986b-2zp9z
	$ oc get endpoints/udp-lb-default-svc -n udp-lb-default-ovn-ns  -o json | jq .subsets[].addresses[].targetRef.name
	"udp-lb-default-dep-7b8784986b-bzx4z"
	"udp-lb-default-dep-7b8784986b-2zp9z"


4. With the given members, spawn a new worker just to trigger the batch member update on octavia:

	$ oc scale machineset/ostest-j6kjf-worker-0 -n openshift-machine-api --replicas=3
	machineset.machine.openshift.io/ostest-j6kjf-worker-0 scaled


	[root@controller-0 ~]# podman exec -it octavia_api tail -f /var/log/octavia/octavia.log
	[...]
	2022-11-14 15:58:30.346 24 DEBUG octavia.db.repositories [req-9016a6bb-51df-4db0-aa73-2b3874d5b26e - 8482c60b04a64342b3b8c575b085ba31 - default default] Checking quota for project: 8482c60b0
	4a64342b3b8c575b085ba31 object: <class 'octavia.common.data_models.Member'> check_quota_met /usr/lib/python3.6/site-packages/octavia/db/repositories.py:373                                  
	2022-11-14 15:58:30.385 24 INFO octavia.api.v2.controllers.member [req-9016a6bb-51df-4db0-aa73-2b3874d5b26e - 8482c60b04a64342b3b8c575b085ba31 - default default] Sending Pool dd8afd57-5bd7-4
	e25-a871-87b6c19c0f6b batch member update to provider ovn                                                                                                                                    
	2022-11-14 15:58:30.392 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=7e1570d0-72fc-4dcd-a352-663f3fa164d1, col_v
	alues=(('external_ids', {'pool_dd8afd57-5bd7-4e25-a871-87b6c19c0f6b': 'member_3341fb36-14a3-42ac-814c-cace8a702cf7_10.196.0.13:31670_fd0ff91b-f649-41c1-b9a5-b674658755bc,member_28cf3a39-05bb-4bcf-bd88-384d18aa77d9_10.196.1.58:31670_fd0ff91b-f649-41c1-b9a5-b674658755bc,member_97f17184-6040-4ee6-967e-ad6fb151f4f0_10.196.0.173:31670_fd0ff91b-f649-41c1-b9a5-b674658755bc,member_74539bd4-9dfd-4b8e-85cb-10f23c915b3a_10.196.3.100:31670_fd0ff91b-f649-41c1-b9a5-b674658755bc,member_ccf76f68-24b3-4dbb-90a8-ca8b4ff49d7d_10.196.2.157:31670_fd0ff91b-f649-41c1-b9a5-b674658755bc,m
	ember_a17c864a-f772-4a8b-94ff-66e46ead2ca0_10.196.2.165:31670_fd0ff91b-f649-41c1-b9a5-b674658755bc'}),)) do_commit /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:852022-11-14 15:58:30.393 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbClearCommand(table=Load_Balancer, record=7e1570d0-72fc-4dcd-a352-663f3fa164d1, column=vips) do_commit /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:85   

	$ openstack loadbalancer member list pool_0_kube_service_kubernetes_udp-lb-default-ovn-ns_udp-lb-default-svc                
	+--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+
	| id                                   | name                        | project_id                       | provisioning_status | address      | protocol_port | operating_status | weight |
	+--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+
	| 28cf3a39-05bb-4bcf-bd88-384d18aa77d9 | ostest-j6kjf-master-0       | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.1.58  |         31670 | ONLINE           |      1 |
	| 3341fb36-14a3-42ac-814c-cace8a702cf7 | ostest-j6kjf-worker-0-9dht7 | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.0.13  |         31670 | ONLINE           |      1 |
	| 74539bd4-9dfd-4b8e-85cb-10f23c915b3a | ostest-j6kjf-master-2       | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.3.100 |         31670 | ONLINE           |      1 |
	| 97f17184-6040-4ee6-967e-ad6fb151f4f0 | ostest-j6kjf-master-1       | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.0.173 |         31670 | ONLINE           |      1 |
	| ccf76f68-24b3-4dbb-90a8-ca8b4ff49d7d | ostest-j6kjf-worker-0-t7ll2 | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.2.157 |         31670 | ONLINE           |      1 |
	| a17c864a-f772-4a8b-94ff-66e46ead2ca0 | ostest-j6kjf-worker-0-t2gqq | 8482c60b04a64342b3b8c575b085ba31 | ACTIVE              | 10.196.2.165 |         31670 | NO_MONITOR       |      1 |
	+--------------------------------------+-----------------------------+----------------------------------+---------------------+--------------+---------------+------------------+--------+

	$ cat <(echo hostname) <(sleep 1) | nc -w 1 -u 10.46.43.102 8082
	udp-lb-default-dep-7b8784986b-bzx4z
	$ cat <(echo hostname) <(sleep 1) | nc -w 1 -u 10.46.43.102 8082
	udp-lb-default-dep-7b8784986b-2zp9z

Comment 12 errata-xmlrpc 2022-12-07 20:27:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795


Note You need to log in before you can comment on or make changes to this bug.