Description of problem: We are seeing `rule:delete_security_group_rule is disallowed by policy` in Kuryr controller after upgrading to Octavia Train (OSP13z13). Version-Release number of selected component (if applicable): 3.11.306 (and 3.11.272) How reproducible: Upgrade OSP13 to z13 on two OpenShift clusters using Kuryr with versions 3.11.272 and 3.11.306. I'll add more details in the BZ after creation.
In our environments, there are two different versions of the Kuryr controller: 3.11.272 (which doesn't have support for Octavia Train OSP13z13) 3.11.306 (which does have support for Octavia Train OSP13z13) These two environments have recently seen their Octavia upgraded as part of the OpenStack OSP13z13 upgrade. which has led us to see the following errors: ================================================================================= 3.11.272 Kuryr controller (which doesn't know how to handle Octavia Train): 2021-01-20 07:13:54.353 1 ERROR kuryr_kubernetes.controller.drivers.lbaasv2 [-] Failed when creating security group rule to enable routes for listener prodnamespace/dart-master:TCP:5001.: NotFound: Security group 43daaf4a-193b-430d-a5bc-477db91a4028 does not exist Neutron server returns request_ids: ['req-5eb0c37f-3e11-4f9e-b73e-b29e12117b70'] AND ----------------------------------------------------------------------------------- 3.11.306 Kuryr controller (which does know how to handle Octavia Train): kuryr-controller-59f678d96c-zxc4j.logs:2021-01-19 04:25:29.955 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'kind': u'Endpoints', u'subsets': [{u'addresses': [{u'ip': u'XXX.XXX.7.47', u'targetRef': {u'kind': u'Pod', u'resourceVersion': u'74682408', u'namespace': u'testnamespace', u'name': u'redis-ha-server-0', u'uid': u'9a27ba5f-5546-11eb-aeac-fa163ee18f34'}, u'nodeName': u'app-node-4.openshift-test1-dcb1.csda.gov.au'}], u'ports': [{u'protocol': u'TCP', u'name': u'exporter', u'port': 9121}, {u'protocol': u'TCP', u'name': u'server', u'port': 6379}, {u'protocol': u'TCP', u'name': u'sentinel', u'port': 26379}]}], u'apiVersion': u'v1', u'metadata': {u'name': u'redis-ha-announce-0', u'labels': {u'release': u'redis-ha', u'app': u'redis-ha', u'heritage': u'Helm', u'chart': u'redis-ha-4.4.4', u'app.kubernetes.io/managed-by': u'Helm'}, u'namespace': u'testnamespace', u'resourceVersion': u'74682410', u'creationTimestamp': u'2020-07-10T01:43:13Z', u'annotations': {u'openstack.org/kuryr-lbaas-spec': u'{"versioned_object.data": {"ip": "XXX.XXX.185.165", "lb_ip": null, "ports": [{"versioned_object.data": {"name": "server", "port": 6379, "protocol": "TCP", "targetPort": "redis"}, "versioned_object.name": "LBaaSPortSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}, {"versioned_object.data": {"name": "sentinel", "port": 26379, "protocol": "TCP", "targetPort": "sentinel"}, "versioned_object.name": "LBaaSPortSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}, {"versioned_object.data": {"name": "exporter", "port": 9121, "protocol": "TCP", "targetPort": "exporter-port"}, "versioned_object.name": "LBaaSPortSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}], "project_id": "89cf74d67cef4c62ab1d1432996afaf1", "security_groups_ids": ["2cb04b99-ffcc-46f9-8c52-f598973e56b7", "1107252a-5a37-41a5-a5b6-467fc1b21c5b"], "subnet_id": "3dd8da27-8f31-46a6-bce1-67c04eca0a66", "type": "ClusterIP"}, "versioned_object.name": "LBaaSServiceSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}', u'openstack.org/kuryr-lbaas-state': u'{"versioned_object.data": {"listeners": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "c6428296-8345-4bcb-9d64-0d28244a0ff0", "loadbalancer_id": "4211d51c-6d3f-4925-9c06-a77070140bdb", "name": "testnamespace/redis-ha-announce-0:TCP:6379", "port": 6379, "project_id": "89cf74d67cef4c62ab1d1432996afaf1", "protocol": "TCP"}, "versioned_object.name": "LBaaSListener", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}, {"versioned_object.changes": ["id"], "versioned_object.data": {"id": "f3ba3229-d81e-43b8-b7d7-53509ec38819", "loadbalancer_id": "4211d51c-6d3f-4925-9c06-a77070140bdb", "name": "testnamespace/redis-ha-announce-0:TCP:26379", "port": 26379, "project_id": "89cf74d67cef4c62ab1d1432996afaf1", "protocol": "TCP"}, "versioned_object.name": "LBaaSListener", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "loadbalancer": {"versioned_object.data": {"id": "4211d51c-6d3f-4925-9c06-a77070140bdb", "ip": "XXX.XXX.185.165", "name": "testnamespace/redis-ha-announce-0", "port_id": "9b993cce-b2b3-4d2a-85c4-f52e70cc586b", "project_id": "89cf74d67cef4c62ab1d1432996afaf1", "provider": "octavia", "security_groups": ["2cb04b99-ffcc-46f9-8c52-f598973e56b7", "1107252a-5a37-41a5-a5b6-467fc1b21c5b"], "subnet_id": "3dd8da27-8f31-46a6-bce1-67c04eca0a66"}, "versioned_object.name": "LBaaSLoadBalancer", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.3"}, "members": [{"versioned_object.data": {"id": "d297498a-ee70-44a3-9b67-3108c5be4765", "ip": "XXX.XXX.7.9", "name": "testnamespace/redis-ha-server-0:6379", "pool_id": "aae59e24-5a5c-4204-87e9-78abc3b4743b", "port": 6379, "project_id": "89cf74d67cef4c62ab1d1432996afaf1", "subnet_id": "3dd8da27-8f31-46a6-bce1-67c04eca0a66"}, "versioned_object.name": "LBaaSMember", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}, {"versioned_object.data": {"id": "5952dd1f-47f7-48a9-9b27-cc2244693e7a", "ip": "XXX.XXX.7.9", "name": "testnamespace/redis-ha-server-0:26379", "pool_id": "9e993033-eed8-4c99-9f98-c6b281c6726c", "port": 26379, "project_id": "89cf74d67cef4c62ab1d1432996afaf1", "subnet_id": "3dd8da27-8f31-46a6-bce1-67c04eca0a66"}, "versioned_object.name": "LBaaSMember", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "pools": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "aae59e24-5a5c-4204-87e9-78abc3b4743b", "listener_id": "c6428296-8345-4bcb-9d64-0d28244a0ff0", "loadbalancer_id": "4211d51c-6d3f-4925-9c06-a77070140bdb", "name": "testnamespace/redis-ha-announce-0:TCP:6379", "project_id": "89cf74d67cef4c62ab1d1432996afaf1", "protocol": "TCP"}, "versioned_object.name": "LBaaSPool", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}, {"versioned_object.changes": ["id"], "versioned_object.data": {"id": "9e993033-eed8-4c99-9f98-c6b281c6726c", "listener_id": "f3ba3229-d81e-43b8-b7d7-53509ec38819", "loadbalancer_id": "4211d51c-6d3f-4925-9c06-a77070140bdb", "name": "testnamespace/redis-ha-announce-0:TCP:26379", "project_id": "89cf74d67cef4c62ab1d1432996afaf1", "protocol": "TCP"}, "versioned_object.name": "LBaaSPool", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}], "service_pub_ip_info": null}, "versioned_object.name": "LBaaSState", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}'}, u'selfLink': u'/api/v1/namespaces/testnamespace/endpoints/redis-ha-announce-0', u'uid': u'b7ba0623-c24e-11ea-8e7d-fa163ecbe187'}}, u'type': u'ADDED'}: Forbidden: rule:delete_security_group_rule is disallowed by policy kuryr-controller-59f678d96c-zxc4j.logs:2021-01-19 04:25:29.955 1 ERROR kuryr_kubernetes.handlers.logging Forbidden: rule:delete_security_group_rule is disallowed by policy =================================================================================
Introduction In my lab, I am using Kuryr controller 3.11.306, and I have created a test application: [openshift@master-0 octaviatrainissue]$ oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE echo-09 ClusterIP XXX.XXX.136.204 <none> 88/TCP 35d echo-10 ClusterIP XXX.XXX.144.5 <none> 88/TCP 1d Service echo-09 was created when Octavia was OSP13z12 (or OSP13z11, I'm not sure) and echo-10 was created with the new Octavia OSP13z13: echo-09: $ oc get svc echo-09 -o yaml apiVersion: v1 kind: Service metadata: annotations: openstack.org/kuryr-lbaas-spec: '{"versioned_object.data": {"ip": "XXX.XXX..136.204", "lb_ip": null, "ports": [{"versioned_object.data": {"name": "http", "port": 80, "protocol": "TCP", "targetPort": "8080"}, "versioned_object.name": "LBaaSPortSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}], "project_id": "f7b96553d2fd4e26a05beb87c85c67c9", "security_groups_ids": ["adfc0f11-e15d-469e-af16-0c62826074df", "889474b8-4f3c-4d87-a18e-ab90934ca3de"], "subnet_id": "e7a1a65f-28c1-4f47-bcbd-bff7b35a3d45", "type": "ClusterIP"}, "versioned_object.name": "LBaaSServiceSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}' creationTimestamp: "2020-12-16T05:22:48Z" labels: k8s-app: momoecho-09 name: echo-09 namespace: momo resourceVersion: "18507" selfLink: /api/v1/namespaces/momo/services/echo-09 uid: bc3cbf29-3f5e-11eb-8692-fa163eee27a8 spec: clusterIP: XXX.XXX..136.204 ports: - name: http port: 80 protocol: TCP targetPort: 8080 selector: k8s-app: momoecho-09 sessionAffinity: None type: ClusterIP status: loadBalancer: {} echo-10: $ oc get svc echo-10 -o yaml apiVersion: v1 kind: Service metadata: annotations: openstack.org/kuryr-lbaas-spec: '{"versioned_object.data": {"ip": "XXX.XXX..144.5", "lb_ip": null, "ports": [{"versioned_object.data": {"name": "http", "port": 80, "protocol": "TCP", "targetPort": "8080"}, "versioned_object.name": "LBaaSPortSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}], "project_id": "f7b96553d2fd4e26a05beb87c85c67c9", "security_groups_ids": ["adfc0f11-e15d-469e-af16-0c62826074df", "889474b8-4f3c-4d87-a18e-ab90934ca3de"], "subnet_id": "e7a1a65f-28c1-4f47-bcbd-bff7b35a3d45", "type": "ClusterIP"}, "versioned_object.name": "LBaaSServiceSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}' creationTimestamp: "2021-01-20T02:55:00Z" labels: k8s-app: momoecho-10 name: echo-10 namespace: momo resourceVersion: "8432118" selfLink: /api/v1/namespaces/momo/services/echo-10 uid: e2fa45d8-5aca-11eb-b6de-fa163e90af27 spec: clusterIP: XXX.XXX..144.5 ports: - name: http port: 80 protocol: TCP targetPort: 8080 selector: k8s-app: momoecho-10 sessionAffinity: None type: ClusterIP status: loadBalancer: {} Tests ===== 1. Change source port for both services from 80 to 88. 2. Change target port for both services from 8080 to 8080 (obviously the ports don't exist on the pods) What appears in the logs: Logs are showing: ========================================================================================= [openshift@master-0 ~]$ oc logs -f kuryr-controller-5676f699f5-2c98h -n kuryr |grep echo 2021-01-21 04:15:10.604 1 WARNING kuryr_kubernetes.controller.drivers.lbaasv2 [-] Cannot find SG rule for 8e82fc5e-0e40-43f8-a230-b23dddc2967d (momo/echo-10:TCP:80) listener. 2021-01-21 04:30:00.769 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'kind': u'Endpoints', u'subsets': [{u'addresses': [{u'ip': u'XXX.XXX..3.224', u'targetRef': {u'kind': u'Pod', u'resourceVersion': u'8168688', u'namespace': u'momo', u'name': u'echo-09-56fcff9f5d-z9bjt', u'uid': u'd123c963-553d-11eb-b6de-fa163e90af27'}, u'nodeName': u'app-node-1.openshift-dev2-dca1.csda.gov.au'}], u'ports': [{u'protocol': u'TCP', u'name': u'http', u'port': 8088}]}], u'apiVersion': u'v1', u'metadata': {u'name': u'echo-09', u'labels': {u'k8s-app': u'momoecho-09'}, u'namespace': u'momo', u'resourceVersion': u'8434389', u'creationTimestamp': u'2020-12-16T05:22:48Z', u'annotations': {u'openstack.org/kuryr-lbaas-spec': u'{"versioned_object.data": {"ip": "XXX.XXX..136.204", "lb_ip": null, "ports": [{"versioned_object.data": {"name": "http", "port": 88, "protocol": "TCP", "targetPort": "8088"}, "versioned_object.name": "LBaaSPortSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}], "project_id": "f7b96553d2fd4e26a05beb87c85c67c9", "security_groups_ids": ["adfc0f11-e15d-469e-af16-0c62826074df", "889474b8-4f3c-4d87-a18e-ab90934ca3de"], "subnet_id": "e7a1a65f-28c1-4f47-bcbd-bff7b35a3d45", "type": "ClusterIP"}, "versioned_object.name": "LBaaSServiceSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}', u'openstack.org/kuryr-lbaas-state': u'{"versioned_object.data": {"listeners": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "855e0bbf-e6a7-4d84-8aca-fa399e8ab808", "loadbalancer_id": "76f00d36-f938-419d-8c20-090010af471c", "name": "momo/echo-09:TCP:80", "port": 80, "project_id": "f7b96553d2fd4e26a05beb87c85c67c9", "protocol": "TCP"}, "versioned_object.name": "LBaaSListener", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "loadbalancer": {"versioned_object.data": {"id": "76f00d36-f938-419d-8c20-090010af471c", "ip": "XXX.XXX..136.204", "name": "momo/echo-09", "port_id": "ca015a97-99a7-4919-9188-86473ce25daf", "project_id": "f7b96553d2fd4e26a05beb87c85c67c9", "provider": "amphora", "security_groups": ["adfc0f11-e15d-469e-af16-0c62826074df", "889474b8-4f3c-4d87-a18e-ab90934ca3de"], "subnet_id": "e7a1a65f-28c1-4f47-bcbd-bff7b35a3d45"}, "versioned_object.name": "LBaaSLoadBalancer", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.3"}, "members": [{"versioned_object.data": {"id": "891c92bc-e557-4258-b69c-0e40cbad407c", "ip": "XXX.XXX..3.224", "name": "momo/echo-09-56fcff9f5d-z9bjt:8080", "pool_id": "ee3c8afa-e06c-463b-b444-1a9f2f0f90e6", "port": 8080, "project_id": "f7b96553d2fd4e26a05beb87c85c67c9", "subnet_id": "e7a1a65f-28c1-4f47-bcbd-bff7b35a3d45"}, "versioned_object.name": "LBaaSMember", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "pools": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "ee3c8afa-e06c-463b-b444-1a9f2f0f90e6", "listener_id": "855e0bbf-e6a7-4d84-8aca-fa399e8ab808", "loadbalancer_id": "76f00d36-f938-419d-8c20-090010af471c", "name": "momo/echo-09:TCP:80", "project_id": "f7b96553d2fd4e26a05beb87c85c67c9", "protocol": "TCP"}, "versioned_object.name": "LBaaSPool", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}], "service_pub_ip_info": null}, "versioned_object.name": "LBaaSState", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}'}, u'selfLink': u'/api/v1/namespaces/momo/endpoints/echo-09', u'uid': u'bc3e498c-3f5e-11eb-bf6c-fa163e90af27'}}, u'type': u'MODIFIED'}: ResourceNotReady: Resource not ready: LBaaSLoadBalancer(id=76f00d36-f938-419d-8c20-090010af471c,ip=XXX.XXX..136.204,name='momo/echo-09',port_id=ca015a97-99a7-4919-9188-86473ce25daf,project_id='f7b96553d2fd4e26a05beb87c85c67c9',provider='amphora',security_groups=[adfc0f11-e15d-469e-af16-0c62826074df,889474b8-4f3c-4d87-a18e-ab90934ca3de],subnet_id=e7a1a65f-28c1-4f47-bcbd-bff7b35a3d45) 2021-01-21 04:30:00.769 1 ERROR kuryr_kubernetes.handlers.logging ResourceNotReady: Resource not ready: LBaaSLoadBalancer(id=76f00d36-f938-419d-8c20-090010af471c,ip=XXX.XXX..136.204,name='momo/echo-09',port_id=ca015a97-99a7-4919-9188-86473ce25daf,project_id='f7b96553d2fd4e26a05beb87c85c67c9',provider='amphora',security_groups=[adfc0f11-e15d-469e-af16-0c62826074df,889474b8-4f3c-4d87-a18e-ab90934ca3de],subnet_id=e7a1a65f-28c1-4f47-bcbd-bff7b35a3d45) ========================================================================================= I will sanitise and attach these logs. This is after the kuryr controller crashed and restarted. I haven't been able to replicate it where the controller goes into CrashLoopBackOff and won't start untill I delete the OCP service it is complaining about (which we found in our production environment).
Created attachment 1749299 [details] Kuryr controller logs
Notice the status of the loadbalancer (echo-09 existed before the octavia upgrade and echo-10 is new): $ openstack loadbalancer list |egrep 'echo-09|echo-10' | 76f00d36-f938-419d-8c20-090010af471c | momo/echo-09 | f7b96553d2fd4e26a05beb87c85c67c9 | XXX.XXX.136.204 | PENDING_UPDATE | amphora | | 912c24e0-c41a-4b11-9851-5f643bad61f2 | momo/echo-10 | f7b96553d2fd4e26a05beb87c85c67c9 | XXX.XXX.144.5 | ACTIVE | amphora | The kuryr controller has restarted 18 times since logging this BZ hours agi: [openshift@master-0 ~]$ oc get pods -n kuryr NAME READY STATUS RESTARTS AGE kuryr-cni-ds-69lvb 2/2 Running 0 6h kuryr-cni-ds-7mfp6 2/2 Running 0 6h kuryr-cni-ds-8zgkf 2/2 Running 0 6h kuryr-cni-ds-fgzwc 2/2 Running 0 6h kuryr-cni-ds-frqcj 2/2 Running 0 6h kuryr-cni-ds-lncts 2/2 Running 0 6h kuryr-cni-ds-m4k7q 2/2 Running 0 6h kuryr-cni-ds-rzdlx 2/2 Running 0 6h kuryr-cni-ds-sds7d 2/2 Running 0 6h kuryr-cni-ds-vw7m7 2/2 Running 0 6h kuryr-cni-ds-wfg77 2/2 Running 0 6h kuryr-controller-5676f699f5-2c98h 1/1 Running 18 3h
I have tested this for another old echo-08 and got the same result, the service is stuck in PENDING_UPDATE. As advised by Luis, I am upgrading the Kuryr components to 3.11.346 to work from a version that has more fixes. I am also checking if the Octavia Amphora VM image has been updated to correspond with the OSP13z13 update.
To confirm, services before Octavia upgrade have their loadbalancers go into PENDING_UPDATE and get stuck there (e.g., echo-09) while services created after Octavia upgrade do go into PENDING_UPDATE but for a few seconds. I have confirmed this twice.
Confirming that a failover of loadbalancers fixes this problem.
Back to ASSIGNED, this needs one more change.
Failed with v3.11.400 OSP13z13 Removed amphora tag from amphora image $ openstack image unset --tag amphora-image Created a service , the LB moved to error state. Severalloadbalancers were created until added the tag back again and the a new LB one moved to active state but the ones in error state were not deleted Controller log: e-11eb-b778-fa163eff3f77'}}, u'type': u'ADDED'}: TypeError: 'LBaaSLoadBalancer' object has no attribute '__getitem__' 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last): 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__ 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 75, in __call__ 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging self.on_present(obj) 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 188, in on_present 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_members(endpoints, lbaas_state, lbaas_spec): 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 275, in _sync_lbaas_members 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_pools(endpoints, lbaas_state, lbaas_spec): 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 462, in _sync_lbaas_pools 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_listeners(endpoints, lbaas_state, lbaas_spec): 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 522, in _sync_lbaas_listeners 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging if self._add_new_listeners(endpoints, lbaas_spec, lbaas_state): 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 551, in _add_new_listeners 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging service_type=lbaas_spec.type) 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 656, in ensure_listener 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging self._find_listener, _LB_STS_POLL_SLOW_INTERVAL) 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 990, in _ensure_provisioned 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging self._wait_for_provisioning(loadbalancer, remaining, interval) 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 1027, in _wait_for_provisioning 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging loadbalancer['id']) 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging TypeError: 'LBaaSLoadBalancer' object has no attribute '__getitem__' 2021-03-16 08:44:26.540 1 ERROR kuryr_kubernetes.handlers.logging
Verified with version:v3.11.405 1. Unset tags from the amphora imgae 2. openstack image unset --tag amphora-image <amphora-image-id> 3. Created deployment: apiVersion: apps/v1 kind: Deployment metadata: name: demo labels: app: demo spec: replicas: 3 selector: matchLabels: app: demo template: metadata: labels: app: demo spec: containers: - name: demo image: quay.io/kuryr/demo ports: - containerPort: 8080 4. Created a service: apiVersion: v1 kind: Service metadata: name: demo labels: app: demo spec: selector: app: demo ports: - port: 80 protocol: TCP targetPort: 8080 5. Watched the loadbalancer created,moving to an error state and being deleted 6. Set the tag for the amphora image openstack image set --tag amphora-image <amphora-image-id> 7. Watched the loadbalancer is ACTIVE 8. Checked connectivity to the service
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.420 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1147