| Summary: | PortBindingFailed - not able to launch VM after OSP major upgrade | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | kamil.rogon |
| Component: | openstack-nova | Assignee: | Lee Yarwood <lyarwood> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Prasanth Anbalagan <panbalag> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 9.0 (Mitaka) | CC: | berrange, dasmith, eglynn, jdonohue, kamil.rogon, kchamart, lyarwood, sbauza, sferdjao, sgordon, srevivo, vromanso |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-01-11 10:11:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | |||
| Bug Blocks: | 1373538, 1400606 | ||
I was able to restore functionality be restarting those services on all computes, but root cause is not identified. # systemctl restart openvswitch # systemctl restart neutron-openvswitch-agent # openstack-service restart nova (In reply to kamil.rogon from comment #0) > Description of problem: I have performed an upgrade of OSP8 with Cisco > nexus/ucsm plugins to OSP9. After all steps I cannot start any new instance > with PortBindingFailed error in nova-compute logs. > > [..] > > How reproducible: > always > > Steps to Reproduce: > 1.Complete all steps from current upgrade documentation (Chapter 3. > Director-based environments: performing upgrades to major versions). > 2. Try to boot VM: > $ nova boot --flavor 2 --image rhel7 --nic > net-id=cc5c67be-1ce0-4606-9d63-5f31fe361ffa after_all_steps > 3. Watch for VM stats using "nova show after_all_steps" > > Actual results: > 1. VM is going into building status. > 2. VM is going into ERROR with fault message as below. > {"message": "Exceeded maximum number of retries. Exceeded max scheduling > attempts 3 for instance a3859ede-4630-4b94-9aae-5583185aedf3. Last > exception: Binding failed for port 87f6470b-e73c-4b99-a61a-8cbdd60e9d36, > please check neutron logs for more information.", "code": 500, "details": " > File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line > 388, in build_instances > filter_properties, instances[0].uuid) > File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line > 186, in populate_retry > raise exception.MaxRetriesExceeded(reason=msg) > ", "created": "2016-11-21T09:32:14Z"} > Thanks for the report. This appears to be a known issue with the upgrade process where by the L2 agents may need to be restarted on the computes after an upgrade : https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Major-Upgrading_the_Overcloud-PostNotes """The Compute nodes might report a failure with neutron-openvswitch-agent. If this occurs, log into each Compute node and restart the service. For example: sudo systemctl restart neutron-openvswitch-agent """ Are you able to provide any example logs from the l2 agent or neutron server to confirm that this is the cause of the issue? Indeed, for same time as some VMs creation I can see errors in neutron-server.log indicating that neutron-openvswitch-agent are dead. I did not correlate my Nova error with L2 agent. I will validate this one more time, but for now it is OK for me to close this as duplicate (if there is a duplicate item for that). (In reply to kamil.rogon from comment #4) > Indeed, for same time as some VMs creation I can see errors in > neutron-server.log indicating that neutron-openvswitch-agent are dead. I did > not correlate my Nova error with L2 agent. > > I will validate this one more time, but for now it is OK for me to close > this as duplicate (if there is a duplicate item for that). Thank you for confirming, closing out as CURRENTRELEASE. |
Description of problem: I have performed an upgrade of OSP8 with Cisco nexus/ucsm plugins to OSP9. After all steps I cannot start any new instance with PortBindingFailed error in nova-compute logs. Version-Release number of selected component (if applicable): [root@overcloud-controller-0 ~]# rpm -qa | grep -e openstack -e cisco | sort fence-agents-cisco-mds-4.0.11-47.el7.x86_64 fence-agents-cisco-ucs-4.0.11-47.el7.x86_64 openstack-aodh-api-2.0.5-1.el7ost.noarch openstack-aodh-common-2.0.5-1.el7ost.noarch openstack-aodh-evaluator-2.0.5-1.el7ost.noarch openstack-aodh-listener-2.0.5-1.el7ost.noarch openstack-aodh-notifier-2.0.5-1.el7ost.noarch openstack-ceilometer-api-6.1.3-2.el7ost.noarch openstack-ceilometer-central-6.1.3-2.el7ost.noarch openstack-ceilometer-collector-6.1.3-2.el7ost.noarch openstack-ceilometer-common-6.1.3-2.el7ost.noarch openstack-ceilometer-compute-6.1.3-2.el7ost.noarch openstack-ceilometer-notification-6.1.3-2.el7ost.noarch openstack-ceilometer-polling-6.1.3-2.el7ost.noarch openstack-cinder-8.1.1-1.el7ost.noarch openstack-dashboard-9.0.1-2.el7ost.noarch openstack-dashboard-theme-9.0.1-2.el7ost.noarch openstack-glance-12.0.0-1.el7ost.noarch openstack-gnocchi-api-2.1.3-3.el7ost.noarch openstack-gnocchi-carbonara-2.1.3-3.el7ost.noarch openstack-gnocchi-common-2.1.3-3.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-2.1.3-3.el7ost.noarch openstack-gnocchi-metricd-2.1.3-3.el7ost.noarch openstack-gnocchi-statsd-2.1.3-3.el7ost.noarch openstack-heat-api-6.0.0-11.el7ost.noarch openstack-heat-api-cfn-6.0.0-11.el7ost.noarch openstack-heat-api-cloudwatch-6.0.0-11.el7ost.noarch openstack-heat-common-6.0.0-11.el7ost.noarch openstack-heat-engine-6.0.0-11.el7ost.noarch openstack-keystone-9.0.2-1.el7ost.noarch openstack-manila-2.0.0-6.el7ost.noarch openstack-manila-share-2.0.0-6.el7ost.noarch openstack-neutron-8.1.2-5.el7ost.noarch openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch openstack-neutron-common-8.1.2-5.el7ost.noarch openstack-neutron-lbaas-8.0.0-1.el7ost.noarch openstack-neutron-metering-agent-8.1.2-5.el7ost.noarch openstack-neutron-ml2-8.1.2-5.el7ost.noarch openstack-neutron-openvswitch-8.1.2-5.el7ost.noarch openstack-nova-api-13.1.1-7.el7ost.noarch openstack-nova-cert-13.1.1-7.el7ost.noarch openstack-nova-common-13.1.1-7.el7ost.noarch openstack-nova-compute-13.1.1-7.el7ost.noarch openstack-nova-conductor-13.1.1-7.el7ost.noarch openstack-nova-console-13.1.1-7.el7ost.noarch openstack-nova-novncproxy-13.1.1-7.el7ost.noarch openstack-nova-scheduler-13.1.1-7.el7ost.noarch openstack-puppet-modules-8.1.8-2.el7ost.noarch openstack-sahara-4.0.1-2.el7ost.noarch openstack-sahara-api-4.0.1-2.el7ost.noarch openstack-sahara-common-4.0.1-2.el7ost.noarch openstack-sahara-engine-4.0.1-2.el7ost.noarch openstack-selinux-0.7.3-3.el7ost.noarch openstack-swift-2.7.0-2.el7ost.noarch openstack-swift-account-2.7.0-2.el7ost.noarch openstack-swift-container-2.7.0-2.el7ost.noarch openstack-swift-object-2.7.0-2.el7ost.noarch openstack-swift-plugin-swift3-1.10-1.el7ost.noarch openstack-swift-proxy-2.7.0-2.el7ost.noarch openstack-utils-2015.2-1.el7ost.noarch python-django-openstack-auth-2.2.0-1.el7ost.noarch python-networking-cisco-3.0.0-1.el7ost.noarch python-openstackclient-2.2.0-1.el7ost.noarch python-openstacksdk-0.8.3-1.el7ost.noarch [root@overcloud-compute-3 ~]# rpm -qa | grep -e openstack -e cisco | sort fence-agents-cisco-mds-4.0.11-47.el7.x86_64 fence-agents-cisco-ucs-4.0.11-47.el7.x86_64 openstack-aodh-api-2.0.5-1.el7ost.noarch openstack-aodh-common-2.0.5-1.el7ost.noarch openstack-aodh-compat-2.0.5-1.el7ost.noarch openstack-aodh-evaluator-2.0.5-1.el7ost.noarch openstack-aodh-expirer-2.0.5-1.el7ost.noarch openstack-aodh-listener-2.0.5-1.el7ost.noarch openstack-aodh-notifier-2.0.5-1.el7ost.noarch openstack-ceilometer-api-6.1.3-2.el7ost.noarch openstack-ceilometer-central-6.1.3-2.el7ost.noarch openstack-ceilometer-collector-6.1.3-2.el7ost.noarch openstack-ceilometer-common-6.1.3-2.el7ost.noarch openstack-ceilometer-compute-6.1.3-2.el7ost.noarch openstack-ceilometer-notification-6.1.3-2.el7ost.noarch openstack-ceilometer-polling-6.1.3-2.el7ost.noarch openstack-cinder-8.1.1-1.el7ost.noarch openstack-dashboard-9.0.1-2.el7ost.noarch openstack-dashboard-theme-9.0.1-2.el7ost.noarch openstack-glance-12.0.0-1.el7ost.noarch openstack-heat-api-6.0.0-11.el7ost.noarch openstack-heat-api-cfn-6.0.0-11.el7ost.noarch openstack-heat-api-cloudwatch-6.0.0-11.el7ost.noarch openstack-heat-common-6.0.0-11.el7ost.noarch openstack-heat-engine-6.0.0-11.el7ost.noarch openstack-keystone-9.0.2-1.el7ost.noarch openstack-manila-2.0.0-6.el7ost.noarch openstack-manila-share-2.0.0-6.el7ost.noarch openstack-neutron-8.1.2-5.el7ost.noarch openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch openstack-neutron-common-8.1.2-5.el7ost.noarch openstack-neutron-lbaas-8.0.0-1.el7ost.noarch openstack-neutron-metering-agent-8.1.2-5.el7ost.noarch openstack-neutron-ml2-8.1.2-5.el7ost.noarch openstack-neutron-openvswitch-8.1.2-5.el7ost.noarch openstack-nova-api-13.1.1-7.el7ost.noarch openstack-nova-cert-13.1.1-7.el7ost.noarch openstack-nova-common-13.1.1-7.el7ost.noarch openstack-nova-compute-13.1.1-7.el7ost.noarch openstack-nova-conductor-13.1.1-7.el7ost.noarch openstack-nova-console-13.1.1-7.el7ost.noarch openstack-nova-novncproxy-13.1.1-7.el7ost.noarch openstack-nova-scheduler-13.1.1-7.el7ost.noarch openstack-puppet-modules-8.1.8-2.el7ost.noarch openstack-selinux-0.7.3-3.el7ost.noarch openstack-swift-2.7.0-2.el7ost.noarch openstack-swift-account-2.7.0-2.el7ost.noarch openstack-swift-container-2.7.0-2.el7ost.noarch openstack-swift-object-2.7.0-2.el7ost.noarch openstack-swift-plugin-swift3-1.10-1.el7ost.noarch openstack-swift-proxy-2.7.0-2.el7ost.noarch openstack-utils-2015.2-1.el7ost.noarch python-django-openstack-auth-2.2.0-1.el7ost.noarch python-networking-cisco-3.0.0-1.el7ost.noarch python-openstackclient-2.2.0-1.el7ost.noarch python-openstacksdk-0.8.3-1.el7ost.noarch How reproducible: always Steps to Reproduce: 1.Complete all steps from current upgrade documentation (Chapter 3. Director-based environments: performing upgrades to major versions). 2. Try to boot VM: $ nova boot --flavor 2 --image rhel7 --nic net-id=cc5c67be-1ce0-4606-9d63-5f31fe361ffa after_all_steps 3. Watch for VM stats using "nova show after_all_steps" Actual results: 1. VM is going into building status. 2. VM is going into ERROR with fault message as below. {"message": "Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance a3859ede-4630-4b94-9aae-5583185aedf3. Last exception: Binding failed for port 87f6470b-e73c-4b99-a61a-8cbdd60e9d36, please check neutron logs for more information.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 388, in build_instances filter_properties, instances[0].uuid) File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 186, in populate_retry raise exception.MaxRetriesExceeded(reason=msg) ", "created": "2016-11-21T09:32:14Z"} Expected results: VM is starting properly and is going into active state. Additional info: Stacktrace from nova-compute logs: 2016-11-21 09:31:35.640 9881 INFO nova.compute.claims [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4 630-4b94-9aae-5583185aedf3] Attempting claim: memory 2048 MB, disk 20 GB, vcpus 1 CPU 2016-11-21 09:31:35.640 9881 INFO nova.compute.claims [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4 630-4b94-9aae-5583185aedf3] Total memory: 261878 MB, used: 4096.00 MB 2016-11-21 09:31:35.640 9881 INFO nova.compute.claims [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] memory limit: 261878.00 MB, free: 257782.00 MB 2016-11-21 09:31:35.641 9881 INFO nova.compute.claims [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Total disk: 131952 GB, used: 20.00 GB 2016-11-21 09:31:35.641 9881 INFO nova.compute.claims [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] disk limit: 131952.00 GB, free: 131932.00 GB 2016-11-21 09:31:35.641 9881 INFO nova.compute.claims [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Total vcpu: 40 VCPU, used: 1.00 VCPU 2016-11-21 09:31:35.641 9881 INFO nova.compute.claims [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] vcpu limit not specified, defaulting to unlimited 2016-11-21 09:31:35.654 9881 INFO nova.compute.claims [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Claim successful 2016-11-21 09:31:36.229 9881 WARNING nova.virt.osinfo [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty) 2016-11-21 09:31:36.268 9881 WARNING nova.virt.osinfo [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty) 2016-11-21 09:31:36.522 9881 WARNING nova.virt.osinfo [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty) 2016-11-21 09:31:36.523 9881 INFO nova.virt.libvirt.driver [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Creating image 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] Instance failed network setup after 1 attempt(s) 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager Traceback (most recent call last): 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1570, in _allocate_network_async 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager bind_host_id=bind_host_id) 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 729, in allocate_for_instance 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids) 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager self.force_reraise() 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb) 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 718, in allocate_for_instance 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts) 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 305, in _create_port 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager raise exception.PortBindingFailed(port_id=port_id) 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 01efa7c3-0a80-47bf-bd5b-3bc5d26c6d44, please check neutron logs for more information. 2016-11-21 09:31:37.913 9881 ERROR nova.compute.manager 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Instance failed to spawn 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Traceback (most recent call last): 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] yield resources 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] block_device_info=block_device_info) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2780, in spawn 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] write_to_disk=True) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4719, in _get_guest_xml 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] network_info_str = str(network_info) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 521, in __str__ 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] return self._sync_wrapper(fn, *args, **kwargs) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 504, in _sync_wrapper 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] self.wait() 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 536, in wait 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] self[:] = self._gt.wait() 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] return self._exit_event.wait() 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] current.throw(*self._exc) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] result = function(*args, **kwargs) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1145, in context_wrapper 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] return func(*args, **kwargs) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1587, in _allocate_network_async 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] six.reraise(*exc_info) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1570, in _allocate_network_async 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] bind_host_id=bind_host_id) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 729, in allocate_for_instance 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] self._delete_ports(neutron, instance, created_port_ids) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] self.force_reraise() 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] six.reraise(self.type_, self.value, self.tb) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 718, in allocate_for_instance 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] security_group_ids, available_macs, dhcp_opts) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 305, in _create_port 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] raise exception.PortBindingFailed(port_id=port_id) 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] PortBindingFailed: Binding failed for port 01efa7c3-0a80-47bf-bd5b-3bc5d26c6d44, please check neutron logs for more information. 2016-11-21 09:31:47.439 9881 ERROR nova.compute.manager [instance: a3859ede-4630-4b94-9aae-5583185aedf3] 2016-11-21 09:31:47.442 9881 INFO nova.compute.manager [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Terminating instance 2016-11-21 09:31:47.447 9881 INFO nova.virt.libvirt.driver [-] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] During wait destroy, instance disappeared. 2016-11-21 09:31:47.726 9881 INFO nova.virt.libvirt.driver [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Deleting instance files /var/lib/nova/instances/a3859ede-4630-4b94-9aae-5583185aedf3_del 2016-11-21 09:31:47.727 9881 INFO nova.virt.libvirt.driver [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Deletion of /var/lib/nova/instances/a3859ede-4630-4b94-9aae-5583185aedf3_del complete 2016-11-21 09:31:47.850 9881 INFO nova.compute.manager [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Took 0.40 seconds to destroy the instance on the hypervisor. 2016-11-21 09:31:47.912 9881 INFO nova.compute.manager [req-b032e048-5ddd-46e8-a028-b144cc0e6d6d 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: a3859ede-4630-4b94-9aae-5583185aedf3] Took 0.06 seconds to deallocate network for instance.