Description of problem: NetAppNFSDriver does not create new snapshots and returns error, the volume.log traceback is: 2013-10-28 14:45:57 ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data rval = self.proxy.dispatch(ctxt, version, method, **args) File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch return getattr(proxyobj, method)(ctxt, **kwargs) File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 482, in create_snapshot {'status': 'error'}) File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ self.gen.next() File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 473, in create_snapshot model_update = self.driver.create_snapshot(snapshot_ref) File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/netapp/nfs.py", line 87, in create_snapshot snapshot['volume_id']) File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/netapp/nfs.py", line 132, in _clone_volume host_id = self._get_host_id(volume_id) File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/netapp/nfs.py", line 203, in _get_host_id resp = server.HostListInfoIterStart(ObjectNameOrId=host_ip) File "/usr/lib/python2.6/site-packages/suds/client.py", line 542, in __call__ return client.invoke(args, kwargs) File "/usr/lib/python2.6/site-packages/suds/client.py", line 602, in invoke result = self.send(soapenv) File "/usr/lib/python2.6/site-packages/suds/client.py", line 657, in send result = self.failed(binding, e) File "/usr/lib/python2.6/site-packages/suds/client.py", line 718, in failed raise Exception((status, reason)) Exception: (404, u'Not Found') Version-Release number of selected component (if applicable): openstack-cinder-2013.1.4-2.el6ost.noarch Steps to Reproduce: 1. configure Cinder with NetAppNFSDriver and the NetApp software accordingly 2. create a new volume 3. create a volume snapshot Actual results: snapshot creation fails and the snapshot is found in "error" state
It looks like this may happen if the volume's provider_location doesn't match the IP address of the NetApp machine. Have you reconfigured networking parameters etc., lately for the NetApp machine? What does # mysql cinder -e "select id,provider_location from volumes" show, versus "netapp_server_hostname" in cinder.conf (or the known address).
hi Eric, that shouldn't be the case as the netapp vfiler didn't change nor I changed the hostname/dns/ip of it but I noticed the table has some NULL values: | 07374fd2-af24-4b17-930a-1cd0cc33477c | vfiler02:/vol/vfilers/vfiler02 | | 07bcd6e8-d681-4456-b8fb-08eab138b4f6 | vfiler02:/vol/vfilers/vfiler02 | | 07d557b5-fd0b-4875-ab59-321ba131c65f | NULL | | 09d2fe05-c727-42be-8255-7e3b9bf5374a | NULL | | 09e3eaf0-e68e-4c83-9698-625d7e4554e5 | vfiler02:/vol/vfilers/vfiler02 | | 0a0fcbe4-6e74-4a94-ab43-c94a5e760c41 | NULL | | 0c26216b-e6ea-414f-8d6f-d925778dcc5c | vfiler02:/vol/vfilers/vfiler02 | does that help?
I assume that one of those showing NULL is the volume that snapshot creation is failing on? _get_host_id() calls _get_host_ip() which does: return self._get_provider_location(volume_id).split(':')[0] So, host_id will be NULL in that case which is probably why the server returns a 404 for locating that object.
hi Eric, actually no, snapshotted volumes (volumes for which the snapshot creation failed) do report correctly the remote share path they could probably be volumes where creation with type and extra spec failed, but this is probably not relevant to this bug
Giulio: Have you tested for or seen this issue on RHOS 4 at all?
We have not tested this with RHOS4 as we are waiting on hardware to use the newer unified driver from NetApp shipped in Havana.
I tried this out briefly w/ 3.0.z and this NetApp server and wasn't able to produce any problem -- need to dig more.
I haven't been able to reproduce this issue... can you please send it back my way if you can still produce it somehow?
In accordance with the Red Hat Enterprise Linux OpenStack Platform Support Policy, the one-year life cycle of Production Support for version 3 will end on July 31, 2014. On August 1, 2014, Red Hat Enterprise Linux OpenStack Platform version 3 will enter an inactive state and will no longer receive updated packages, including Critical-impact security patches or urgent-priority bug fixes. In addition, technical support through Red Hat's Global Support Services will no longer be provided after this date. We encourage customers to plan their migration from Red Hat Enterprise Linux OpenStack Platform 3.0 to a supported version of Red Hat Enterprise Linux OpenStack Platform. To upgrade to Red Hat Enterprise Linux OpenStack Platform version 4, see Chapter "Upgrading" in the Release Notes document linked to in the References section. Full details of the Red Hat Enterprise Linux OpenStack Platform Life Cycle can be found at https://access.redhat.com/support/policy/updates/openstack/platform/ https://rhn.redhat.com/errata/RHSA-2014-0995.html