Bug 1893681 - [Kuryr] Bulk port creation exception not completely formatted
Summary: [Kuryr] Bulk port creation exception not completely formatted
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.6
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.6.z
Assignee: rdobosz
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On: 1890472
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-02 10:57 UTC by OpenShift BugZilla Robot
Modified: 2021-03-30 17:03 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-30 17:03:12 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 390 0 None open [release-4.6] Bug 1893681: Bulk port creation exception not completely formatted 2021-02-15 07:45:18 UTC
Red Hat Product Errata RHBA-2021:0952 0 None None None 2021-03-30 17:03:25 UTC

Description OpenShift BugZilla Robot 2020-11-02 10:57:52 UTC
+++ This bug was initially created as a clone of Bug #1890472 +++

Description of problem:

The openstack.exceptions.SDKException exception that is raised for errors during port bulk creation is not completely
formatted as it's missing the creation request response text.

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait
    listener.cb(fileno)
  File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool
    num_ports=num_ports)
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs
    ports = list(os_net.create_ports(bulk_port_rq))
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports
    response.text)
openstack.exceptions.SDKException: Error when bulk creating ports: %s
                                                                    ^
                                                                    |

Version-Release number of selected component (if applicable):
OCP 4.6.0-0.nightly-2020-10-20-101225
OSP13 2020-10-06.2

How reproducible: always


Steps to Reproduce:
1. Make a subnet runs out of available ports so no new ports can be created
   $ oc new-project test
   $ oc new-app kuryr/demo

Scale up until the max number of ports are reached (for a /23 subnet it will be ~508 ports)
   $ oc scale deployment/demo --replicas 50
   $ oc scale deployment/demo --replicas 100
    ...
   $ oc scale deployment/demo --replicas 510

2. Check Kuryr controller's logs


Actual results: exception printed without the creation request response text.

Expected results: exception correctly printed

Comment 3 Jon Uriarte 2021-03-24 16:44:07 UTC
Verified on OCP 4.6.0-0.nightly-2021-03-21-131139 on top of OSP 13.0.14 (2021-01-20.1) using OVS and amphora provider.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-03-21-131139   True        False         5m54s   Cluster version is 4.6.0-0.nightly-2021-03-21-131139


# Install OCP and run below script to overload the port creation until the exception is generated with the proper information:

$ oc get pods -n openshift-kuryr
NAME                               READY   STATUS    RESTARTS   AGE
kuryr-cni-bgs6j                     1/1     Running   0          31m
kuryr-cni-cb2kh                     1/1     Running   0          31m
kuryr-cni-nd8lg                     1/1     Running   0          19m
kuryr-cni-pgh6k                     1/1     Running   0          31m
kuryr-cni-r2knc                     1/1     Running   0          18m
kuryr-cni-t2pwn                     1/1     Running   0          18m
kuryr-controller-55db7899fb-69tbb   1/1     Running   0          31m

$ oc new-project test
$ for i in `seq 1 504`; do oc run --image kuryr/demo demo-$i; sleep 4; done


# Check kuryr-controller logs exception:

2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif [-] Error creating bulk ports: {'ports': [{'project_id': '8d21e1348d954cc385780591ca1a8e63', 'network_id': '5204d3c1-afeb-4d8f-910e-05c57bea0205', 'fixed_ips': [{'subnet_id': '4667e81c-3705-4076-beb6-ddbf49f23616'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('46a1c03a-be0a-4468-86ef-d86f144dfa13',)}, {'project_id': '8d21e1348d954cc385780591ca1a8e63', 'network_id': '5204d3c1-afeb-4d8f-910e-05c57bea0205', 'fixed_ips': [{'subnet_id': '4667e81c-3705-4076-beb6-ddbf49f23616'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('46a1c03a-be0a-4468-86ef-d86f144dfa13',)}, {'project_id': '8d21e1348d954cc385780591ca1a8e63', 'network_id': '5204d3c1-afeb-4d8f-910e-05c57bea0205', 'fixed_ips': [{'subnet_id': '4667e81c-3705-4076-beb6-ddbf49f23616'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('46a1c03a-be0a-4468-86ef-d86f144dfa13',)}]}: openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network 5204d3c1-afeb-4d8f-910e-05c57bea0205.", "type": "IpAddressGenerationFailure", "detail": ""}}
2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif Traceback (most recent call last):
2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs
2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif     ports = list(os_net.create_ports(bulk_port_rq))                             
2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports
2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif     response.text)
2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network 5204d3c1-afeb-4d8f-910e-05c57bea0205.", "type": "IpAddressGenerationFailure", "detail": ""}}
2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait
    listener.cb(fileno)
  File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool
    num_ports=num_ports)
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs                                                                                                                   
    ports = list(os_net.create_ports(bulk_port_rq))
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports
    response.text)
openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network 5204d3c1-afeb-4d8f-910e-05c57bea0205.", "type": "IpAddressGenerationFailure", "detail": ""}}      
Removing descriptor: 1013

Comment 5 errata-xmlrpc 2021-03-30 17:03:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.23 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0952


Note You need to log in before you can comment on or make changes to this bug.