Bug 1890472 - [Kuryr] Bulk port creation exception not completely formatted
Summary: [Kuryr] Bulk port creation exception not completely formatted
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.6
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.7.0
Assignee: rdobosz
QA Contact: GenadiC
URL:
Whiteboard:
Depends On:
Blocks: 1893681
TreeView+ depends on / blocked
 
Reported: 2020-10-22 10:35 UTC by Jon Uriarte
Modified: 2021-02-24 15:28 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:27:41 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 389 0 None closed Bug 1890472: Bulk port creation exception not completely formatted 2021-01-08 09:03:14 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:28:13 UTC

Description Jon Uriarte 2020-10-22 10:35:20 UTC
Description of problem:

The openstack.exceptions.SDKException exception that is raised for errors during port bulk creation is not completely
formatted as it's missing the creation request response text.

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait
    listener.cb(fileno)
  File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool
    num_ports=num_ports)
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs
    ports = list(os_net.create_ports(bulk_port_rq))
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports
    response.text)
openstack.exceptions.SDKException: Error when bulk creating ports: %s
                                                                    ^
                                                                    |

Version-Release number of selected component (if applicable):
OCP 4.6.0-0.nightly-2020-10-20-101225
OSP13 2020-10-06.2

How reproducible: always


Steps to Reproduce:
1. Make a subnet runs out of available ports so no new ports can be created
   $ oc new-project test
   $ oc new-app kuryr/demo

Scale up until the max number of ports are reached (for a /23 subnet it will be ~508 ports)
   $ oc scale deployment/demo --replicas 50
   $ oc scale deployment/demo --replicas 100
    ...
   $ oc scale deployment/demo --replicas 510

2. Check Kuryr controller's logs


Actual results: exception printed without the creation request response text.

Expected results: exception correctly printed

Comment 3 rlobillo 2020-11-11 16:42:19 UTC
Verified on OCP4.7.0-0.nightly-2020-11-11-055454 over OSP13 with Amphoras (2020-11-04.1)

# Install OCP and run below script to overload the port creation until the exception is generated with the proper information:

$ oc get pods -n openshift-kuryr
NAME                               READY   STATUS    RESTARTS   AGE
kuryr-cni-2cqdh                    1/1     Running   2          5h36m
kuryr-cni-7cmd8                    1/1     Running   0          5h53m
kuryr-cni-9z62s                    1/1     Running   0          5h53m
kuryr-cni-gtghw                    1/1     Running   1          5h36m
kuryr-cni-jq2wz                    1/1     Running   2          5h36m
kuryr-cni-klx6v                    1/1     Running   0          5h53m
kuryr-controller-bc69d85cc-wg6mp   1/1     Running   1          5h53m
(shiftstack) [stack@undercloud-0 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-11-11-055454   True        False         5h20m   Cluster version is 4.7.0-0.nightly-2020-11-11-055454


$ for i in `seq 1 504`; do oc run --image kuryr/demo demo-$i; sleep 4; done


# Check kuryr-controller logs exception:

2020-11-11 16:30:05.126 1 WARNING kuryr_kubernetes.controller.handlers.kuryrport [-] Ignoring event due to pod redhat-marketplace-dzjvk not being scheduled yet.
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif [-] Error creating bulk ports: {'ports': [{'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}, {'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}, {'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}]}: openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}}
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif Traceback (most recent call last):
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif     ports = list(os_net.create_ports(bulk_port_rq))
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif     response.text)
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}}
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif 
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait
    listener.cb(fileno)
  File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool
    num_ports=num_ports)
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs
    ports = list(os_net.create_ports(bulk_port_rq))
  File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports
    response.text)
openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}}
Removing descriptor: 43

Comment 6 errata-xmlrpc 2021-02-24 15:27:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.