+++ This bug was initially created as a clone of Bug #1890472 +++ Description of problem: The openstack.exceptions.SDKException exception that is raised for errors during port bulk creation is not completely formatted as it's missing the creation request response text. Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait listener.cb(fileno) File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main result = function(*args, **kwargs) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool num_ports=num_ports) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs ports = list(os_net.create_ports(bulk_port_rq)) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports response.text) openstack.exceptions.SDKException: Error when bulk creating ports: %s ^ | Version-Release number of selected component (if applicable): OCP 4.6.0-0.nightly-2020-10-20-101225 OSP13 2020-10-06.2 How reproducible: always Steps to Reproduce: 1. Make a subnet runs out of available ports so no new ports can be created $ oc new-project test $ oc new-app kuryr/demo Scale up until the max number of ports are reached (for a /23 subnet it will be ~508 ports) $ oc scale deployment/demo --replicas 50 $ oc scale deployment/demo --replicas 100 ... $ oc scale deployment/demo --replicas 510 2. Check Kuryr controller's logs Actual results: exception printed without the creation request response text. Expected results: exception correctly printed
Verified on OCP 4.6.0-0.nightly-2021-03-21-131139 on top of OSP 13.0.14 (2021-01-20.1) using OVS and amphora provider. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2021-03-21-131139 True False 5m54s Cluster version is 4.6.0-0.nightly-2021-03-21-131139 # Install OCP and run below script to overload the port creation until the exception is generated with the proper information: $ oc get pods -n openshift-kuryr NAME READY STATUS RESTARTS AGE kuryr-cni-bgs6j 1/1 Running 0 31m kuryr-cni-cb2kh 1/1 Running 0 31m kuryr-cni-nd8lg 1/1 Running 0 19m kuryr-cni-pgh6k 1/1 Running 0 31m kuryr-cni-r2knc 1/1 Running 0 18m kuryr-cni-t2pwn 1/1 Running 0 18m kuryr-controller-55db7899fb-69tbb 1/1 Running 0 31m $ oc new-project test $ for i in `seq 1 504`; do oc run --image kuryr/demo demo-$i; sleep 4; done # Check kuryr-controller logs exception: 2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif [-] Error creating bulk ports: {'ports': [{'project_id': '8d21e1348d954cc385780591ca1a8e63', 'network_id': '5204d3c1-afeb-4d8f-910e-05c57bea0205', 'fixed_ips': [{'subnet_id': '4667e81c-3705-4076-beb6-ddbf49f23616'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('46a1c03a-be0a-4468-86ef-d86f144dfa13',)}, {'project_id': '8d21e1348d954cc385780591ca1a8e63', 'network_id': '5204d3c1-afeb-4d8f-910e-05c57bea0205', 'fixed_ips': [{'subnet_id': '4667e81c-3705-4076-beb6-ddbf49f23616'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('46a1c03a-be0a-4468-86ef-d86f144dfa13',)}, {'project_id': '8d21e1348d954cc385780591ca1a8e63', 'network_id': '5204d3c1-afeb-4d8f-910e-05c57bea0205', 'fixed_ips': [{'subnet_id': '4667e81c-3705-4076-beb6-ddbf49f23616'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('46a1c03a-be0a-4468-86ef-d86f144dfa13',)}]}: openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network 5204d3c1-afeb-4d8f-910e-05c57bea0205.", "type": "IpAddressGenerationFailure", "detail": ""}} 2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif Traceback (most recent call last): 2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs 2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif ports = list(os_net.create_ports(bulk_port_rq)) 2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports 2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif response.text) 2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network 5204d3c1-afeb-4d8f-910e-05c57bea0205.", "type": "IpAddressGenerationFailure", "detail": ""}} 2021-03-24 16:36:35.842 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait listener.cb(fileno) File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main result = function(*args, **kwargs) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool num_ports=num_ports) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs ports = list(os_net.create_ports(bulk_port_rq)) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports response.text) openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network 5204d3c1-afeb-4d8f-910e-05c57bea0205.", "type": "IpAddressGenerationFailure", "detail": ""}} Removing descriptor: 1013
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.23 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0952