Attachment 730649 Details for Bug 947351 – Error when network is recreated

Error when network is recreated

file_947351.txt (text/plain), 11.84 KB, created by Ofer Blaut on 2013-04-02 08:09:09 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Ofer Blaut

Created: 2013-04-02 08:09:09 UTC

Size: 11.84 KB

patch

obsolete

>[root@puma04 ~(keystone_admin)]$ tail -100 /var/log/nova/compute.log
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 191, in decorated_function
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 839, in run_instance
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     do_run_instance()
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 804, in inner
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 838, in do_run_instance
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     admin_password, is_first_time, instance)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 529, in _run_instance
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     self._set_instance_error_state(context, instance['uuid'])
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 517, in _run_instance
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     is_first_time, request_spec, filter_properties)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 547, in _reschedule_or_reraise
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     self._deallocate_network(context, instance)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 783, in _deallocate_network
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     self.network_api.deallocate_for_instance(context, instance)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 261, in deallocate_for_instance
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     'args': args})
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 108, in call
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 554, in call
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     rpc_amqp.get_connection_pool(conf, Connection))
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 369, in call
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     rv = list(rv)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 337, in __iter__
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     raise result
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError_Remote: Unexpected error while running command.
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf"] NETWORK_ID=6 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br190.pid --listen-address=10.35.175.200 --except-interface=lo --dhcp-range=set:'net_vlan_190',10.35.175.3,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br190.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Exit code: 2
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Stdout: ''
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Stderr: '\ndnsmasq: failed to bind listening socket for 10.35.175.200: Address already in use\n'
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 266, in wrapped
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     return func(self, context, *args, **kwargs)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 396, in deallocate_for_instance
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     super(FloatingIP, self).deallocate_for_instance(context, **kwargs)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 266, in wrapped
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     return func(self, context, *args, **kwargs)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1058, in deallocate_for_instance
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     host=kwargs.get('host'))
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 239, in deallocate_fixed_ip
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     address)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1376, in deallocate_fixed_ip
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     self._teardown_network_on_host(context, network)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 2162, in _teardown_network_on_host
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     self.driver.update_dhcp(context, dev, network)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 792, in update_dhcp
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     restart_dhcp(context, dev, network_ref)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 804, in inner
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 883, in restart_dhcp
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     _execute(*cmd, run_as_root=True)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 975, in _execute
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     return utils.execute(*cmd, **kwargs)
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 210, in execute
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf"] NETWORK_ID=6 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br190.pid --listen-address=10.35.175.200 --except-interface=lo --dhcp-range=set:'net_vlan_190',10.35.175.3,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br190.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Exit code: 2
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Stdout: ''
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp Stderr: '\ndnsmasq: failed to bind listening socket for 10.35.175.200: Address already in use\n'
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:23:36 9182 TRACE nova.openstack.common.rpc.amqp 
>2013-04-02 10:24:10 9182 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 31075
>2013-04-02 10:24:10 9182 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
>2013-04-02 10:24:10 9182 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 11
>2013-04-02 10:24:10 9182 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma04.scl.lab.tlv.redhat.com 
>2013-04-02 10:24:10 9182 INFO nova.compute.manager [-] Updating host status
>2013-04-02 10:25:10 9182 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 31075
>2013-04-02 10:25:10 9182 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
>2013-04-02 10:25:10 9182 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 11
>2013-04-02 10:25:10 9182 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma04.scl.lab.tlv.redhat.com 
>2013-04-02 10:26:10 9182 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 31075
>2013-04-02 10:26:10 9182 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
>2013-04-02 10:26:10 9182 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 11
>2013-04-02 10:26:11 9182 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma04.scl.lab.tlv.redhat.com 
>2013-04-02 10:26:11 9182 INFO nova.compute.manager [-] Updating host status

Actions: View

Attachments on bug 947351: 730649