Bug 1890472
| Summary: | [Kuryr] Bulk port creation exception not completely formatted | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jon Uriarte <juriarte> |
| Component: | Networking | Assignee: | rdobosz |
| Networking sub component: | kuryr | QA Contact: | GenadiC <gcheresh> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | low | ||
| Priority: | low | CC: | rdobosz, rlobillo |
| Version: | 4.6 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-24 15:27:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1893681 | ||
Verified on OCP4.7.0-0.nightly-2020-11-11-055454 over OSP13 with Amphoras (2020-11-04.1)
# Install OCP and run below script to overload the port creation until the exception is generated with the proper information:
$ oc get pods -n openshift-kuryr
NAME READY STATUS RESTARTS AGE
kuryr-cni-2cqdh 1/1 Running 2 5h36m
kuryr-cni-7cmd8 1/1 Running 0 5h53m
kuryr-cni-9z62s 1/1 Running 0 5h53m
kuryr-cni-gtghw 1/1 Running 1 5h36m
kuryr-cni-jq2wz 1/1 Running 2 5h36m
kuryr-cni-klx6v 1/1 Running 0 5h53m
kuryr-controller-bc69d85cc-wg6mp 1/1 Running 1 5h53m
(shiftstack) [stack@undercloud-0 ~]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.7.0-0.nightly-2020-11-11-055454 True False 5h20m Cluster version is 4.7.0-0.nightly-2020-11-11-055454
$ for i in `seq 1 504`; do oc run --image kuryr/demo demo-$i; sleep 4; done
# Check kuryr-controller logs exception:
2020-11-11 16:30:05.126 1 WARNING kuryr_kubernetes.controller.handlers.kuryrport [-] Ignoring event due to pod redhat-marketplace-dzjvk not being scheduled yet.
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif [-] Error creating bulk ports: {'ports': [{'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}, {'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}, {'project_id': '351d7660f0ca482a9b9e8b4e4021570f', 'network_id': 'ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6', 'fixed_ips': [{'subnet_id': '1684f29a-5b4c-422f-87b8-65a3cfeb594d'}], 'device_owner': 'compute:kuryr', 'admin_state_up': True, 'security_groups': ('cb4eb73a-750f-4fa9-ad47-12da3688e7c4',)}]}: openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}}
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif Traceback (most recent call last):
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif ports = list(os_net.create_ports(bulk_port_rq))
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif response.text)
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}}
2020-11-11 16:38:47.500 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait
listener.cb(fileno)
File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
result = function(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool
num_ports=num_ports)
File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs
ports = list(os_net.create_ports(bulk_port_rq))
File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports
response.text)
openstack.exceptions.SDKException: Error when bulk creating ports: {"NeutronError": {"message": "No more IP addresses available on network ddda0fc7-0ab0-48a7-aafa-d707fc4a70e6.", "type": "IpAddressGenerationFailure", "detail": ""}}
Removing descriptor: 43
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633 |
Description of problem: The openstack.exceptions.SDKException exception that is raised for errors during port bulk creation is not completely formatted as it's missing the creation request response text. Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 111, in wait listener.cb(fileno) File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main result = function(*args, **kwargs) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 259, in _populate_pool num_ports=num_ports) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 85, in request_vifs ports = list(os_net.create_ports(bulk_port_rq)) File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/clients.py", line 107, in _create_ports response.text) openstack.exceptions.SDKException: Error when bulk creating ports: %s ^ | Version-Release number of selected component (if applicable): OCP 4.6.0-0.nightly-2020-10-20-101225 OSP13 2020-10-06.2 How reproducible: always Steps to Reproduce: 1. Make a subnet runs out of available ports so no new ports can be created $ oc new-project test $ oc new-app kuryr/demo Scale up until the max number of ports are reached (for a /23 subnet it will be ~508 ports) $ oc scale deployment/demo --replicas 50 $ oc scale deployment/demo --replicas 100 ... $ oc scale deployment/demo --replicas 510 2. Check Kuryr controller's logs Actual results: exception printed without the creation request response text. Expected results: exception correctly printed