Bug 977895
| Summary: | nova [Negative]: issues with print of nova show on instance that failed to boot due to nfs server connectivity issues | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Dafna Ron <dron> | ||||
| Component: | openstack-nova | Assignee: | Russell Bryant <rbryant> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | Ami Jeain <ajeain> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | unspecified | CC: | jruzicka, ndipanov, sgordon, srevivo, sross, yeylon | ||||
| Target Milestone: | --- | Keywords: | Reopened, Triaged, ZStream | ||||
| Target Release: | 6.0 (Juno) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-04-15 16:20:47 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
same issue for permission denied error:
[root@opens-vdsb lib(keystone_admin)]# nova show bdfc260e-14b9-490b-befc-293afda2d4b9
+-------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+-------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| status | ERROR |
| updated | 2013-06-26T11:27:57Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | nott-vdsa.qa.lab.tlv.redhat.com |
| key_name | None |
| image | rhel (a53e43d6-b956-4695-b4d9-abf195c49fea) |
| hostId | af8d2b7ddca044031949039f1fd556e34bd498d052f888fa704b6e3e |
| OS-EXT-STS:vm_state | error |
| OS-EXT-SRV-ATTR:instance_name | instance-00000014 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | nott-vdsa.qa.lab.tlv.redhat.com |
| flavor | m1.tiny (1) |
| id | bdfc260e-14b9-490b-befc-293afda2d4b9 |
| security_groups | [{u'name': u'default'}] |
| user_id | 4e8268c19c2143a0b3cf978afab45fea |
| name | blae |
| created | 2013-06-26T11:27:53Z |
| tenant_id | f372ca53f0484f589413148b6c9ad39c |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| novanetwork network | |
| accessIPv4 | |
| accessIPv6 | |
| fault | {u'message': u'IOError', u'code': 500, u'details': u'[Errno 13] Permission denied: u\'/var/lib/nova/networks/nova-br100.conf\' |
| | Traceback (most recent call last): |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data |
| | rval = self.proxy.dispatch(ctxt, version, method, **args) |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch |
| | return getattr(proxyobj, method)(ctxt, **kwargs) |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 187, in deallocate_for_instance |
| | super(FloatingIP, self).deallocate_for_instance(context, **kwargs) |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 550, in deallocate_for_instance |
| | self.deallocate_fixed_ip(context, fixed_ip[\'address\'], host=host) |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 243, in deallocate_fixed_ip |
| | address) |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 938, in deallocate_fixed_ip |
| | self._teardown_network_on_host(context, network) |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1632, in _teardown_network_on_host |
| | self.driver.update_dhcp(elevated, dev, network) |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 940, in update_dhcp |
| | write_to_file(conffile, get_dhcp_hosts(context, network_ref)) |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 592, in write_to_file |
| | with open(file, mode) as f: |
| | |
| | IOError: [Errno 13] Permission denied: u\'/var/lib/nova/networks/nova-br100.conf\' |
| | |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 224, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1224, in run_instance |
| | do_run_instance() |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 242, in inner |
| | retval = f(*args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1223, in do_run_instance |
| | admin_password, is_first_time, node, instance) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 879, in _run_instance |
| | self._set_instance_error_state(context, instance[\'uuid\']) |
| | File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ |
| | self.gen.next() |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 871, in _run_instance |
| | filter_properties, bdms) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 911, in _reschedule_or_reraise |
| | self._log_original_error(exc_info, instance_uuid) |
| | File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ |
| | self.gen.next() |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 906, in _reschedule_or_reraise |
| | self._shutdown_instance(context, instance, bdms) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1247, in _shutdown_instance |
| | self._set_instance_error_state(context, instance[\'uuid\']) |
| | File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ |
| | self.gen.next() |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1242, in _shutdown_instance |
| | self._deallocate_network(context, instance) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1159, in _deallocate_network |
| | self.network_api.deallocate_for_instance(context, instance) |
| | File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 89, in wrapped |
| | return func(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 285, in deallocate_for_instance |
| | self.network_rpcapi.deallocate_for_instance(context, **args) |
| | File "/usr/lib/python2.6/site-packages/nova/network/rpcapi.py", line 180, in deallocate_for_instance |
| | topic=topic) |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/proxy.py", line 80, in call |
| | return rpc.call(context, self._get_topic(topic), msg, timeout) |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 140, in call |
| | return _get_impl().call(CONF, context, topic, msg, timeout) |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 610, in call |
| | rpc_amqp.get_connection_pool(conf, Connection)) |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 612, in call |
| | rv = list(rv) |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 508, in __iter__ |
| | raise result |
| | ', u'created': u'2013-06-26T11:27:56Z'} |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+-------------------------------------+-------------------------------------------------------
This was misfiled under novnc. It should be python-novaclient. novaclient is just printing what nova sent it and the backtrace is from nova so I'm changing component to nova. The traceback is printed when admin does the request, when you do it with a non-admin user (demo) the traceback isn't printed at all, nova prints the following output: $ nova show ee46f79d-a1c0-4e7f-804a-be79bef5a35e ERROR: No server with a name or ID of 'ee46f79d-a1c0-4e7f-804a-be79bef5a35e' exists. This is a debugging/troubleshooting feature for the admin. Closing this as NOTABUG. same user (which is admin user) is used on all my requests. The bug is that nova show will give results with "garbage" output if cinder remote storage is down and will give a normal result if storage is accessible. The "garbage" is actually a traceback so that it would be easier for the administrator to troubleshoot the problem. As you can see the output is properly formatted, just like when cinder is accessible, it's just that more info is included (the traceback). It's intended to work this way, that's why I closed it as NOTABUG. Yes, Vladan is right here. This is intended and correct behavior. The traceback is visible to the cloud admin. (In reply to Vladan Popovic from comment #6) > The "garbage" is actually a traceback so that it would be easier for the > administrator to troubleshoot the problem. > As you can see the output is properly formatted, just like when cinder is > accessible, it's just that more info is included (the traceback). It's > intended to work this way, that's why I closed it as NOTABUG. I don't really follow, why print this "traceback" into the standard output? just have it properly logged in the log and show valid error message to the user. Haim: The only thing that falls on mind is that it's quicker to discover the actual error like this instead of searching the logs for tracebacks. If it were a critical operational error (out of space, unreachable node ...) the admin, and only the admin would see the error and be able to immediately troubleshoot and fix the problem. We discussed about this and it would be OK for the admin to see the traceback if debug=True, but not otherwise. This isn't implemented in nova and would require opening the bug upstream and waiting for it's approval for this to be resolved. I was sure I opened one downstream as well but I could not find it Opened a new one and attached to external links This is planned to be changed in the next release, or at least the output is to be taken care of and the API will be upgraded to v3. Some references mentioning what may seem relevant: https://etherpad.openstack.org/p/IcehouseTaskAPI https://blueprints.launchpad.net/nova/+spec/nova-v3-api Couldn't find the exact blueprint that addresses exactly this issue, when I find something or when it comes out I'll surely post it here. We'll keep an eye on this so if something comes up we'll get right on it. |
Created attachment 765115 [details] logs and full show output Description of problem: i tried booting an instance from a volume while i blocked the connectivity to the nfs server. once the instance failed booting on timeout and changed status to ERROR, I ran nova show <instance UUID>. the print shows stack trace Version-Release number of selected component (if applicable): [root@opens-vdsb tmp(keystone_admin)]# rpm -qa |grep nova python-novaclient-2.13.0-1.el6ost.noarch openstack-nova-scheduler-2013.1.2-2.el6ost.noarch openstack-nova-compute-2013.1.2-2.el6ost.noarch python-nova-2013.1.2-2.el6ost.noarch openstack-nova-api-2013.1.2-2.el6ost.noarch openstack-nova-console-2013.1.2-2.el6ost.noarch openstack-nova-conductor-2013.1.2-2.el6ost.noarch openstack-nova-novncproxy-0.4-6.el6ost.noarch openstack-nova-common-2013.1.2-2.el6ost.noarch openstack-nova-cert-2013.1.2-2.el6ost.noarch openstack-nova-network-2013.1.2-2.el6ost.noarch How reproducible: 100% Steps to Reproduce: 1. create an nfs server 2. create a volume 3. block connectivity to the server using iptables: iptables -A OUTPUT -d >Server> -j DROP 4. boot an instance from the volume: nova boot --flavor 2 --block_device_mapping=vda=<volume_uuid>::: <name> 5. run nova show <instance_UUID> Actual results: the results of the show command appear with stack trace Expected results: even if an instance is in error the show command should be able to supply the information Additional info: full output and logs are attached fault | {u'message': u'ClientException', u'code': 500, u'details': u'The server has either erred or is incapable of performing the requested ope ration. (HTTP 500) (Request-ID: req-6cfa3ce9-312b-42f8-a3a2-846d87ae63bb) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 224, in decorated_function | | | return function(self, context, *args, **kwargs) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1224, in run_instance | | | do_run_instance() | | | File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 242, in inner | | | retval = f(*args, **kwargs) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1223, in do_run_instance | | | admin_password, is_first_time, node, instance) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 879, in _run_instance | | | self._set_instance_error_state(context, instance[\'uuid\']) | | | File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ | | | self.gen.next() | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 871, in _run_instance | | | filter_properties, bdms) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 911, in _reschedule_or_reraise | | | self._log_original_error(exc_info, instance_uuid) | :