865924 – terminate instance doesn't - "Not Authorized" from network/manager.py

Bug 865924 - terminate instance doesn't - "Not Authorized" from network/manager.py

Summary: terminate instance doesn't - "Not Authorized" from network/manager.py

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	1.0 (Essex)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	RHOS Maint
QA Contact:	Ofer Blaut
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-10-12 20:06 UTC by Dan Yocum
Modified:	2013-02-06 20:21 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2013-02-06 20:21:23 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Dan Yocum 2012-10-12 20:06:31 UTC

Description of problem:

Occasionally a user can not terminate a one of their own running instance because the network manager doesn't think the IP is associated with the project that the user is in.  See the Notes section for the entire trace.

Version-Release number of selected component (if applicable):

Essex

How reproducible:

intermittent.

Steps to Reproduce:
1. Launch a VM
2. Terminate the VM
3.
  
Actual results:

Instance remains running and the web UI shows "deleting" forever.

Expected results:

Intance should terminate.

Additional info:

Here's the trace from the compute.log:

2012-10-12 10:09:07 INFO nova.compute.manager [req-3daf7bac-2b5c-490c-9210-da724e5ac93f 1c09f9141fa9476f9a1de852d34867f0 882ee643d2b046b3b0b356928e321f43] check_instance_lock: decorating: |<
function terminate_instance at 0x2aadc80>|
2012-10-12 10:09:07 INFO nova.compute.manager [req-3daf7bac-2b5c-490c-9210-da724e5ac93f 1c09f9141fa9476f9a1de852d34867f0 882ee643d2b046b3b0b356928e321f43] check_instance_lock: arguments: |<n
ova.compute.manager.ComputeManager object at 0x2939e90>| |<nova.rpc.amqp.RpcContext object at 0x4ef6150>| |404553c4-802e-4426-871c-2aab510fb3be|
2012-10-12 10:09:07 INFO nova.compute.manager [req-3daf7bac-2b5c-490c-9210-da724e5ac93f 1c09f9141fa9476f9a1de852d34867f0 882ee643d2b046b3b0b356928e321f43] check_instance_lock: locked: |False
|
2012-10-12 10:09:07 INFO nova.compute.manager [req-3daf7bac-2b5c-490c-9210-da724e5ac93f 1c09f9141fa9476f9a1de852d34867f0 882ee643d2b046b3b0b356928e321f43] check_instance_lock: admin: |False|
2012-10-12 10:09:07 INFO nova.compute.manager [req-3daf7bac-2b5c-490c-9210-da724e5ac93f 1c09f9141fa9476f9a1de852d34867f0 882ee643d2b046b3b0b356928e321f43] check_instance_lock: executing: |<f
unction terminate_instance at 0x2aadc80>|
2012-10-12 10:09:07 AUDIT nova.compute.manager [req-3daf7bac-2b5c-490c-9210-da724e5ac93f 1c09f9141fa9476f9a1de852d34867f0 882ee643d2b046b3b0b356928e321f43] [instance: 404553c4-802e-4426-871c-2aab510fb3be] Terminating instance
2012-10-12 10:09:07 ERROR nova.rpc.amqp [req-3daf7bac-2b5c-490c-9210-da724e5ac93f 1c09f9141fa9476f9a1de852d34867f0 882ee643d2b046b3b0b356928e321f43] Exception during message handling
2012-10-12 10:09:07 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 253, in _process_data
2012-10-12 10:09:07 TRACE nova.rpc.amqp     rval = node_func(context=ctxt, **node_args)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
2012-10-12 10:09:07 TRACE nova.rpc.amqp     return f(*args, **kw)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 159, in decorated_function
2012-10-12 10:09:07 TRACE nova.rpc.amqp     function(self, context, instance_uuid, *args, **kwargs)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 183, in decorated_function
2012-10-12 10:09:07 TRACE nova.rpc.amqp     sys.exc_info())
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-10-12 10:09:07 TRACE nova.rpc.amqp     self.gen.next()
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 177, in decorated_function
2012-10-12 10:09:07 TRACE nova.rpc.amqp     return function(self, context, instance_uuid, *args, **kwargs)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 771, in terminate_instance
2012-10-12 10:09:07 TRACE nova.rpc.amqp     do_terminate_instance()
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 946, in inner
2012-10-12 10:09:07 TRACE nova.rpc.amqp     retval = f(*args, **kwargs)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 764, in do_terminate_instance
2012-10-12 10:09:07 TRACE nova.rpc.amqp     self._delete_instance(context, instance)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 742, in _delete_instance
2012-10-12 10:09:07 TRACE nova.rpc.amqp     self._shutdown_instance(context, instance, 'Terminating')
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 704, in _shutdown_instance
2012-10-12 10:09:07 TRACE nova.rpc.amqp     self._deallocate_network(context, instance)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 636, in _deallocate_network
2012-10-12 10:09:07 TRACE nova.rpc.amqp     self.network_api.deallocate_for_instance(context, instance)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 190, in deallocate_for_instance
2012-10-12 10:09:07 TRACE nova.rpc.amqp     'args': args})
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/__init__.py", line 68, in call
2012-10-12 10:09:07 TRACE nova.rpc.amqp     return _get_impl().call(context, topic, msg, timeout)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 526, in call
2012-10-12 10:09:07 TRACE nova.rpc.amqp     return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 343, in call
2012-10-12 10:09:07 TRACE nova.rpc.amqp     rv = list(rv)
2012-10-12 10:09:07 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 311, in __iter__
2012-10-12 10:09:07 TRACE nova.rpc.amqp     raise result
2012-10-12 10:09:07 TRACE nova.rpc.amqp RemoteError: Remote error: NotAuthorized Not authorized.
2012-10-12 10:09:07 TRACE nova.rpc.amqp [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 253, in _process_data\n    rval = node_func(context=ctxt, **node_args)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 257, in wrapped\n    return func(self, context, *args, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 370, in deallocate_for_instance\n    affect_auto_assigned=True)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 257, in wrapped\n    return func(self, context, *args, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 514, in disassociate_floating_ip\n    self._floating_ip_owned_by_project(context, floating_ip)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 387, in _floating_ip_owned_by_project\n    raise exception.NotAuthorized()\n', u'NotAuthorized: Not authorized.\n'].
2012-10-12 10:09:07 TRACE nova.rpc.amqp

Comment 2 Nikola Dipanov 2013-01-04 18:58:04 UTC

Dan,

Can you make sure you were not using the admin credentials when testing this with Essex? There is a patch that addresses a similar issue that made it to Folsom and it happened due to admin user not having a tenanat set (https://bugs.launchpad.net/nova/+bug/1045508).

Since we will not be backporting stuff to Essex - I will move this bug to Folsom to make sure it is not happening again.

Please re test and close if it is not reproduceable anymore.

Comment 3 Dan Yocum 2013-02-03 20:47:16 UTC

Sorry for the delay in replying - I will try w/o admin privs.  Right now, I can't see anything in the "Instances and Volumes" screen because of a wedged volume in the DB - someone shutdown an instance before detatching the persistent volume from the VM.  (This is a different bug... I need to search BZ to see if it has been reported).

Comment 4 Rami Vaknin 2013-02-06 20:21:23 UTC

This bug does not reproduce on my Folsom on RHEL6.4 (openstack-nova-2012.2.2-9.el6ost).

I've booted an instance from a non-privileged user, then I've allocated a floating IP by the admin user and assigned it to the non-privileged user's instance using the admin user (so maybe this way a non-privileged user's instance will get a floating IP of other tenant), then I've stopped the non-privileged user's instance, operation passed successfully, I even deleted the instance by the non-privileged user.

In addition, I haven't encountered with the described issue while playing a bit with the floating IPs feature.

Note You need to log in before you can comment on or make changes to this bug.