Bug 981418 - horizon: no alert for failing to attach a volume when the rpcbind is down on the compute the instance is running on
horizon: no alert for failing to attach a volume when the rpcbind is down on ...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-django-horizon (Show other bugs)
unspecified
x86_64 Linux
unspecified Severity low
: ---
: ---
Assigned To: Angus Thomas
Ami Jeain
storage
: Triaged
Depends On: 1029919
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-04 12:06 EDT by Dafna Ron
Modified: 2015-06-04 17:52 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-04-29 03:03:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
log (188.38 KB, application/x-xz)
2013-07-04 12:06 EDT, Dafna Ron
no flags Details

  None (edit)
Description Dafna Ron 2013-07-04 12:06:33 EDT
Created attachment 768865 [details]
log

Description of problem:

I have two computes working with a cinder remote nfs storage. 
I did not get a failure alert in horizon when trying to attach a volume to an instance that was running on a compute in which the rpcbind was down

Version-Release number of selected component (if applicable):

python-django-horizon-2013.1.2-1.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. install with packstack on a setup that has two computes 
2. change cinder to work with a remove nfs share
3.

Actual results:

we fail to attach the volume but have no error alert in horizon

Expected results:

we should have an error alert in horizon

Additional info:

2013-07-04 18:26:40.793 ERROR nova.openstack.common.rpc.amqp [req-8886c85d-6fb0-4041-8cd4-46633d5a87fb 4e8268c19c2143a0b3cf978afab45fea f372ca53f0484f589413148b6c9ad39c] Except
ion during message handling
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 94, in wrapped
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 209, in decorated_function
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     pass
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 195, in decorated_function
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 237, in decorated_function
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 224, in decorated_function
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2852, in attach_volume
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     context, instance, mountpoint)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2847, in attach_volume
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     mountpoint, instance)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2887, in _attach_volume
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     connector)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2878, in _attach_volume
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     mountpoint)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 957, in attach_volume
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     disk_info)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 943, in volume_driver_method
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     return method(connection_info, *args, **kwargs)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 477, in connect_volume
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     path = self._ensure_mounted(connection_info['data']['export'], options)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 490, in _ensure_mounted
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     self._mount_nfs(mount_path, nfs_export, options, ensure=True)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 506, in _mount_nfs
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     utils.execute(*nfs_cmd, run_as_root=True)
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 239, in execute
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -t nfs orion.qa.lab.tlv.redhat.com:/share /var/lib/nova/mnt/fde24a16f24b6c83ed7c586f4982c88b
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp Exit code: 32
2013-07-04 18:26:40.793 8736 TRACE nova.openstack.common.rpc.amqp Stdout: ''
Comment 1 Julie Pichon 2013-07-17 06:29:40 EDT
I agree it would be nice to give a message back to the user. At the moment there isn't much we can do on the Horizon side in this case, since the attach call is run asynchronously by cinder and no information is sent back. The same thing happens if trying to attach using the command line tool.

If we do get a message back though, it probably shouldn't be anything more than "Failed to attach volume". rpcbind being down is something only the admin can act on, and is not actually useful to the user.
Comment 2 Julie Pichon 2014-05-27 07:25:54 EDT
Matching target versions to bug depended on.
Comment 4 Matthias Runge 2015-04-29 03:03:58 EDT
While I agree this is a bug, I would expect rcpbind to be monitored by nagios or whatever.

Horizon relies on a functioning infrastructure. Currently, there's not much we can do about this to work around.

Note You need to log in before you can comment on or make changes to this bug.