Bug 1481121

Description of problem:

After providing a new cinder package to the customer coming from (BZ#1437556) customer has new netapp issues.

Hi All,

We provided openstack-cinder-9.1.3-3.el7ost to the customer but he is still experiencing issues.

It seems to be https://bugs.launchpad.net/cinder/+bug/1694579
"""
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 222, in invoke_successfully
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode raise NaApiError(code, msg)
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode NaApiError: NetApp API failed. Reason - 13001:Object "system:constituent" was not found. <======
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode
"""

This is customer's trace:

2017-08-10 08:25:19.586 170638 WARNING cinder.volume.drivers.nfs [req-901d7b32-4155-4649-924e-3acb4735c1ba - - - - -] The NAS file operations will be run as root: allowing root level access at the storage backend. This is considered an insecure NAS environment. Please see http://docs.openstack.org/admin-guide/blockstorage_nfs_backend.html for information on a secure NAS configuration.
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode [req-901d7b32-4155-4649-924e-3acb4735c1ba - - - - -] Could not get performance base counter name. Performance-based scheduler functions may not be available.
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode Traceback (most recent call last):
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/performance/perf_cmode.py", line 47, in _init_counter_info
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode     'avg_processor_busy'))
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/performance/perf_base.py", line 224, in _get_base_counter_name
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode     object_name, counter_name)
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method_logging_wrapper
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode     return f(*args, **kwargs)
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 292, in get_performance_counter_info
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode     enable_tunneling=False)
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method_logging_wrapper
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode     return f(*args, **kwargs)
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 93, in send_request
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode     return self.connection.invoke_successfully(request, enable_tunneling)
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 222, in invoke_successfully
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode     raise NaApiError(code, msg)
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode NaApiError: NetApp API failed. Reason - 13001:Object "system:constituent" was not found.
2017-08-10 08:25:19.673 170638 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager [req-901d7b32-4155-4649-924e-3acb4735c1ba - - - - -] Failed to initialize driver.
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager Traceback (most recent call last):
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 415, in init_host
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager     self.driver.check_for_setup_error()
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method_logging_wrapper
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method_logging_wrapper
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method_logging_wrapper
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py", line 99, in check_for_setup_error
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager     self.ssc_library.check_api_permissions()
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/utils/capabilities.py", line 79, in check_api_permissions
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager     raise exception.VolumeBackendAPIException(data=msg)
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: User not permitted to query Data ONTAP volumes.
2017-08-10 08:25:19.753 170638 ERROR cinder.volume.manager



And this is the network communication between the driver and netapp, at least is what was captured:

We establish connection:

- HTTP Request:

POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1
Accept-Encoding: identity
Content-Length: 102
Charset: utf-8
Host: 126.190.62.124:80
User-Agent: Python-urllib/2.7
Connection: close
Content-Type: text/xml

<netapp xmlns="http://www.netapp.com/filer/admin" version="1.15"><system-get-ontapi-version/></netapp>


- Response: 

HTTP/1.1 401 Unauthorized
Date: Thu, 10 Aug 2017 11:50:49 GMT
Server: Apache
WWW-Authenticate: Basic realm="Remote Administrative API Support"
Content-Length: 381
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Unauthorized</title>
</head><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested.  Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>

Connection Closed



We establish another connextion

- Second HTTP Request:

POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1
Accept-Encoding: identity
Content-Length: 102
Charset: utf-8
Connection: close
User-Agent: Python-urllib/2.7
Host: 126.190.62.124:80
Content-Type: text/xml
Authorization: Basic b3BlbnN0YWNrX2FkbWluOk9TdGFja1NLMTIj

<netapp xmlns="http://www.netapp.com/filer/admin" version="1.15"><system-get-ontapi-version/></netapp>


Second Response:

|^EEY@@~>|~;CPu
A^yHTTP/1.1 200 OK
Date: Thu, 10 Aug 2017 11:50:49 GMT
Server: libzapid-httpd
Content-Length: 268
Content-Type: text/xml
Connection: close

<?xml version='1.0' encoding='UTF-8' ?>
<!DOCTYPE netapp SYSTEM 'file:/etc/netapp_gx.dtd'>
<netapp version='1.110' xmlns='http://www.netapp.com/filer/admin'>
<results status="passed"><major-version>1</major-version><minor-version>110</minor-version></results></netapp>

Connection Closed


This is the workaround in upstream:
"""
I solve the problem in my environment.
It looks like the driver does not work at all with "Vserver Scoped" permissions.
Vserver scoped user can not get object system:constituent.
With Cluster permissions scoped user everything work.
"""

This is the documentation customer is using (Please note that there is a 8.0 a " OpenStack @ NetAppOpenStack Deployment and Operations Guide  - Version 8.0 " but customer is using On the storage site, we have ONTAP 9.1P5 . :
http://netapp.github.io/openstack-deploy-ops-guide/newton/content/cinder.fas.configuration.html

Answer from customer said when I told them to apply the workaround:

 SVM scoped account should work correctly. We cannot use cluster scope due to security reason.



Version-Release number of selected component (if applicable):
RHOS 10

How reproducible:

always

Steps to Reproduce:
1. Install package openstack-cinder-9.1.3-3.el7ost that solves https://bugzilla.redhat.com/show_bug.cgi?id=1437556
2. Setup a netapp backed
3. restart cinder

Actual results:
Netapp driver can initialize

Expected results:
Netapp able to initialize

Additional info: