Description of problem: The openstack.exceptions.SDKException exception that is raised for errors during port bulk creation is not completely formatted as it's missing the creation request response text. Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait listener.cb(fileno) File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main result = function(*args, **kwargs) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool num_ports=num_ports) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs ports = list(os_net.create_ports(bulk_port_rq)) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports response.text) openstack.exceptions.SDKException: Error when bulk creating ports: %s ^ | Version-Release number of selected component (if applicable): OCP 4.6.0-0.nightly-2020-10-20-101225 OSP13 2020-10-06.2 How reproducible: always Steps to Reproduce: 1. Make a subnet runs out of available ports so no new ports can be created $ oc new-project test $ oc new-app kuryr/demo Scale up until the max number of ports are reached (for a /23 subnet it will be ~508 ports) $ oc scale deployment/demo --replicas 50 $ oc scale deployment/demo --replicas 100 ... $ oc scale deployment/demo --replicas 510 2. Check Kuryr controller's logs Actual results: exception printed without the creation request response text. Expected results: exception correctly printed
Verified on OCP4.7.0-0.nightly-2020-11-11-055454 over OSP13 with Amphoras (2020-11-04.1) # Install OCP and run below script to overload the port creation until the exception is generated with the proper information: $ oc get pods -n openshift-kuryr NAME READY STATUS RESTARTS AGE kuryr-cni-2cqdh 1/1 Running 2 5h36m kuryr-cni-7cmd8 1/1 Running 0 5h53m kuryr-cni-9z62s 1/1 Running 0 5h53m kuryr-cni-gtghw 1/1 Running 1 5h36m kuryr-cni-jq2wz 1/1 Running 2 5h36m kuryr-cni-klx6v 1/1 Running 0 5h53m kuryr-controller-bc69d85cc-wg6mp 1/1 Running 1 5h53m (shiftstack) [stack@undercloud-0 ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-11-11-055454 True False 5h20m Cluster version is 4.7.0-0.nightly-2020-11-11-055454 $ for i in `seq 1 504`; do oc run --image kuryr/demo demo-$i; sleep 4; done # Check kuryr-controller logs exception: 2020-11-11 16:30:05.126 1 WARNING kuryr_kubernetes.controller.handlers.kuryrport [-] Ignoring event due to pod redhat-marketplace-dzjvk not being scheduled yet. 2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif [-] Error creating bulk ports: {'ports': [{'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}, {'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}, {'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}]}: openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}} 2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif Traceback (most recent call last): 2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs 2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif ports = list(os_net.create_ports(bulk_port_rq)) 2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports 2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif response.text) 2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}} 2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait listener.cb(fileno) File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main result = function(*args, **kwargs) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool num_ports=num_ports) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs ports = list(os_net.create_ports(bulk_port_rq)) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports response.text) openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}} Removing descriptor: 43
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633