Bug 977895 - nova [Negative]: issues with print of nova show on instance that failed to boot due to nfs server connectivity issues
nova [Negative]: issues with print of nova show on instance that failed to bo...
Status: CLOSED WONTFIX
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
unspecified
x86_64 Linux
low Severity low
: ---
: 6.0 (Juno)
Assigned To: Russell Bryant
Ami Jeain
: Reopened, Triaged, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-25 10:26 EDT by Dafna Ron
Modified: 2016-04-26 17:28 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-04-15 12:20:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs and full show output (79.21 KB, application/x-gzip)
2013-06-25 10:26 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1253629 None None None Never

  None (edit)
Description Dafna Ron 2013-06-25 10:26:41 EDT
Created attachment 765115 [details]
logs and full show output

Description of problem:

i tried booting an instance from a volume while i blocked the connectivity to the nfs server. 
once the instance failed booting on timeout and changed status to ERROR, I ran nova show <instance UUID>.  

the print shows stack trace 

Version-Release number of selected component (if applicable):

[root@opens-vdsb tmp(keystone_admin)]# rpm -qa |grep nova
python-novaclient-2.13.0-1.el6ost.noarch
openstack-nova-scheduler-2013.1.2-2.el6ost.noarch
openstack-nova-compute-2013.1.2-2.el6ost.noarch
python-nova-2013.1.2-2.el6ost.noarch
openstack-nova-api-2013.1.2-2.el6ost.noarch
openstack-nova-console-2013.1.2-2.el6ost.noarch
openstack-nova-conductor-2013.1.2-2.el6ost.noarch
openstack-nova-novncproxy-0.4-6.el6ost.noarch
openstack-nova-common-2013.1.2-2.el6ost.noarch
openstack-nova-cert-2013.1.2-2.el6ost.noarch
openstack-nova-network-2013.1.2-2.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. create an nfs server
2. create a volume 
3. block connectivity to the server using iptables: iptables -A OUTPUT -d >Server>  -j DROP
4. boot an instance from the volume: nova boot --flavor 2  --block_device_mapping=vda=<volume_uuid>::: <name>
5. run nova show <instance_UUID>

Actual results:

the results of the show command appear with stack trace 

Expected results:

even if an instance is in error the show command should be able to supply the information 

Additional info: full output and logs are attached 


fault                               | {u'message': u'ClientException', u'code': 500, u'details': u'The server has either erred or is incapable of performing the requested ope
ration. (HTTP 500) (Request-ID: req-6cfa3ce9-312b-42f8-a3a2-846d87ae63bb) |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 224, in decorated_function                                      
                                                                          |
|                                     |     return function(self, context, *args, **kwargs)                                                                                     
                                                                          |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1224, in run_instance                                           
                                                                          |
|                                     |     do_run_instance()                                                                                                                   
                                                                          |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 242, in inner                                        
                                                                          |
|                                     |     retval = f(*args, **kwargs)                                                                                                         
                                                                          |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1223, in do_run_instance                                        
                                                                          |
|                                     |     admin_password, is_first_time, node, instance)                                                                                      
                                                                          |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 879, in _run_instance                                           
                                                                          |
|                                     |     self._set_instance_error_state(context, instance[\'uuid\'])                                                                         
                                                                          |
|                                     |   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__                                                                       
                                                                          |
|                                     |     self.gen.next()                                                                                                                     
                                                                          |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 871, in _run_instance                                           
                                                                          |
|                                     |     filter_properties, bdms)                                                                                                            
                                                                          |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 911, in _reschedule_or_reraise                                  
                                                                          |
|                                     |     self._log_original_error(exc_info, instance_uuid)                                                                                   
                                                                          |
:
Comment 1 Dafna Ron 2013-06-26 07:44:19 EDT
same issue for permission denied error: 

[root@opens-vdsb lib(keystone_admin)]# nova show bdfc260e-14b9-490b-befc-293afda2d4b9
+-------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| Property                            | Value                                                                                                                          |
+-------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| status                              | ERROR                                                                                                                          |
| updated                             | 2013-06-26T11:27:57Z                                                                                                           |
| OS-EXT-STS:task_state               | None                                                                                                                           |
| OS-EXT-SRV-ATTR:host                | nott-vdsa.qa.lab.tlv.redhat.com                                                                                                |
| key_name                            | None                                                                                                                           |
| image                               | rhel (a53e43d6-b956-4695-b4d9-abf195c49fea)                                                                                    |
| hostId                              | af8d2b7ddca044031949039f1fd556e34bd498d052f888fa704b6e3e                                                                       |
| OS-EXT-STS:vm_state                 | error                                                                                                                          |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000014                                                                                                              |
| OS-EXT-SRV-ATTR:hypervisor_hostname | nott-vdsa.qa.lab.tlv.redhat.com                                                                                                |
| flavor                              | m1.tiny (1)                                                                                                                    |
| id                                  | bdfc260e-14b9-490b-befc-293afda2d4b9                                                                                           |
| security_groups                     | [{u'name': u'default'}]                                                                                                        |
| user_id                             | 4e8268c19c2143a0b3cf978afab45fea                                                                                               |
| name                                | blae                                                                                                                           |
| created                             | 2013-06-26T11:27:53Z                                                                                                           |
| tenant_id                           | f372ca53f0484f589413148b6c9ad39c                                                                                               |
| OS-DCF:diskConfig                   | MANUAL                                                                                                                         |
| metadata                            | {}                                                                                                                             |
| novanetwork network                 |                                                                                                                                |
| accessIPv4                          |                                                                                                                                |
| accessIPv6                          |                                                                                                                                |
| fault                               | {u'message': u'IOError', u'code': 500, u'details': u'[Errno 13] Permission denied: u\'/var/lib/nova/networks/nova-br100.conf\' |
|                                     | Traceback (most recent call last):                                                                                             |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data                        |
|                                     |     rval = self.proxy.dispatch(ctxt, version, method, **args)                                                                  |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch                       |
|                                     |     return getattr(proxyobj, method)(ctxt, **kwargs)                                                                           |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 187, in deallocate_for_instance                   |
|                                     |     super(FloatingIP, self).deallocate_for_instance(context, **kwargs)                                                         |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 550, in deallocate_for_instance                        |
|                                     |     self.deallocate_fixed_ip(context, fixed_ip[\'address\'], host=host)                                                        |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 243, in deallocate_fixed_ip                            |
|                                     |     address)                                                                                                                   |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 938, in deallocate_fixed_ip                            |
|                                     |     self._teardown_network_on_host(context, network)                                                                           |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1632, in _teardown_network_on_host                     |
|                                     |     self.driver.update_dhcp(elevated, dev, network)                                                                            |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 940, in update_dhcp                                  |
|                                     |     write_to_file(conffile, get_dhcp_hosts(context, network_ref))                                                              |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 592, in write_to_file                                |
|                                     |     with open(file, mode) as f:                                                                                                |
|                                     |                                                                                                                                |
|                                     | IOError: [Errno 13] Permission denied: u\'/var/lib/nova/networks/nova-br100.conf\'                                             |
|                                     |                                                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 224, in decorated_function                             |
|                                     |     return function(self, context, *args, **kwargs)                                                                            |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1224, in run_instance                                  |
|                                     |     do_run_instance()                                                                                                          |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 242, in inner                               |
|                                     |     retval = f(*args, **kwargs)                                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1223, in do_run_instance                               |
|                                     |     admin_password, is_first_time, node, instance)                                                                             |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 879, in _run_instance                                  |
|                                     |     self._set_instance_error_state(context, instance[\'uuid\'])                                                                |
|                                     |   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__                                                              |
|                                     |     self.gen.next()                                                                                                            |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 871, in _run_instance                                  |
|                                     |     filter_properties, bdms)                                                                                                   |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 911, in _reschedule_or_reraise                         |
|                                     |     self._log_original_error(exc_info, instance_uuid)                                                                          |
|                                     |   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__                                                              |
|                                     |     self.gen.next()                                                                                                            |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 906, in _reschedule_or_reraise                         |
|                                     |     self._shutdown_instance(context, instance, bdms)                                                                           |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1247, in _shutdown_instance                            |
|                                     |     self._set_instance_error_state(context, instance[\'uuid\'])                                                                |
|                                     |   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__                                                              |
|                                     |     self.gen.next()                                                                                                            |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1242, in _shutdown_instance                            |
|                                     |     self._deallocate_network(context, instance)                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1159, in _deallocate_network                           |
|                                     |     self.network_api.deallocate_for_instance(context, instance)                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 89, in wrapped                                             |
|                                     |     return func(self, context, *args, **kwargs)                                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 285, in deallocate_for_instance                            |
|                                     |     self.network_rpcapi.deallocate_for_instance(context, **args)                                                               |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/network/rpcapi.py", line 180, in deallocate_for_instance                         |
|                                     |     topic=topic)                                                                                                               |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/proxy.py", line 80, in call                                 |
|                                     |     return rpc.call(context, self._get_topic(topic), msg, timeout)                                                             |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 140, in call                             |
|                                     |     return _get_impl().call(CONF, context, topic, msg, timeout)                                                                |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 610, in call                            |
|                                     |     rpc_amqp.get_connection_pool(conf, Connection))                                                                            |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 612, in call                                 |
|                                     |     rv = list(rv)                                                                                                              |
|                                     |   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 508, in __iter__                             |
|                                     |     raise result                                                                                                               |
|                                     | ', u'created': u'2013-06-26T11:27:56Z'}                                                                                        |
| OS-EXT-STS:power_state              | 0                                                                                                                              |
| OS-EXT-AZ:availability_zone         | nova                                                                                                                           |
| config_drive                        |                                                                                                                                |
+-------------------------------------+-------------------------------------------------------
Comment 2 Solly Ross 2013-10-17 14:16:40 EDT
This was misfiled under novnc.  It should be python-novaclient.
Comment 3 Jakub Ruzicka 2013-11-08 10:28:53 EST
novaclient is just printing what nova sent it and the backtrace is from nova so I'm changing component to nova.
Comment 4 Vladan Popovic 2013-11-12 10:25:23 EST
The traceback is printed when admin does the request, when you do it with a non-admin user (demo) the traceback isn't printed at all, nova prints the following output:
$ nova show ee46f79d-a1c0-4e7f-804a-be79bef5a35e
ERROR: No server with a name or ID of 'ee46f79d-a1c0-4e7f-804a-be79bef5a35e' exists.

This is a debugging/troubleshooting feature for the admin.
Closing this as NOTABUG.
Comment 5 Dafna Ron 2013-11-15 03:40:18 EST
same user (which is admin user) is used on all my requests. 

The bug is that nova show will give results with "garbage" output if cinder remote storage is down and will give a normal result if storage is accessible.
Comment 6 Vladan Popovic 2013-11-15 11:01:25 EST
The "garbage" is actually a traceback so that it would be easier for the administrator to troubleshoot the problem.
As you can see the output is properly formatted, just like when cinder is accessible, it's just that more info is included (the traceback). It's intended to work this way, that's why I closed it as NOTABUG.
Comment 7 Russell Bryant 2013-11-21 00:36:54 EST
Yes, Vladan is right here.  This is intended and correct behavior.  The traceback is visible to the cloud admin.
Comment 8 Haim 2013-11-21 01:49:29 EST
(In reply to Vladan Popovic from comment #6)
> The "garbage" is actually a traceback so that it would be easier for the
> administrator to troubleshoot the problem.
> As you can see the output is properly formatted, just like when cinder is
> accessible, it's just that more info is included (the traceback). It's
> intended to work this way, that's why I closed it as NOTABUG.

I don't really follow, why print this "traceback" into the standard output? just have it properly logged in the log and show valid error message to the user.
Comment 10 Vladan Popovic 2013-11-21 07:10:39 EST
Haim: The only thing that falls on mind is that it's quicker to discover the actual error like this instead of searching the logs for tracebacks. If it were a critical operational error (out of space, unreachable node ...) the admin, and only the admin would see the error and be able to immediately troubleshoot and fix the problem.

We discussed about this and it would be OK for the admin to see the traceback if debug=True, but not otherwise. This isn't implemented in nova and would require opening the bug upstream and waiting for it's approval for this to be resolved.
Comment 11 Dafna Ron 2013-11-21 07:47:28 EST
I was sure I opened one downstream as well but I could not find it 
Opened a new one and attached to external links
Comment 12 Vladan Popovic 2013-11-22 12:35:49 EST
This is planned to be changed in the next release, or at least the output is to be taken care of and the API will be upgraded to v3.
Some references mentioning what may seem relevant:
https://etherpad.openstack.org/p/IcehouseTaskAPI
https://blueprints.launchpad.net/nova/+spec/nova-v3-api

Couldn't find the exact blueprint that addresses exactly this issue, when I find something or when it comes out I'll surely post it here.
We'll keep an eye on this so if something comes up we'll get right on it.

Note You need to log in before you can comment on or make changes to this bug.