Bug 1640443 - volume_extend while attached - Policy doesn't allow os_compute_api:os-server-external-events:create
Summary: volume_extend while attached - Policy doesn't allow os_compute_api:os-server-...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 14.0 (Rocky)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 14.0 (Rocky)
Assignee: Alan Bishop
QA Contact: Avi Avraham
Kim Nylander
URL:
Whiteboard:
: 1622006 (view as bug list)
Depends On:
Blocks: 1315799 1640754
TreeView+ depends on / blocked
 
Reported: 2018-10-18 04:43 UTC by bkopilov
Modified: 2023-10-06 18:00 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-9.0.1-0.20181013060885.el7ost puppet-tripleo-9.3.1-0.20181010034749.157eaab.el7ost puppet-cinder-13.3.1-0.20181013114721.25b1ba3.el7ost
Doc Type: Bug Fix
Doc Text:
The OpenStack Platform director was not configuring authentication data required for the Block Storage service (cinder) to access privileged portions of the nova API. Because of this, operations on volumes that use nova's privileged API (e.g., migrating an in-use volume) would fail. The director now configures cinder with nova's authentication data. As a result, operations on volumes that require privileges work.
Clone Of:
Environment:
Last Closed: 2019-01-11 11:54:07 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
cinder-volume (3.24 MB, text/plain)
2018-10-18 04:50 UTC, bkopilov 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1802347 0 None None None 2018-11-08 18:54:33 UTC
OpenStack gerrit 616666 0 'None' MERGED Include ::cinder::nova in cinder's configuration 2020-11-12 01:32:31 UTC
OpenStack gerrit 616669 0 'None' MERGED Configure cinder's access to the nova API 2020-11-12 01:32:30 UTC
OpenStack gerrit 618047 0 'None' MERGED Improve cinder::nova support for auth_type password 2020-11-12 01:32:31 UTC
Red Hat Bugzilla 1622006 1 None None None 2023-09-14 04:33:36 UTC
Red Hat Bugzilla 1624744 0 unspecified CLOSED NFS assisted snapshot fails 2023-10-06 17:56:32 UTC
Red Hat Bugzilla 1641825 0 high CLOSED RFE: Populate authentication information in [nova] section in cinder.conf enabling migration of in-use bootable volumes ... 2023-10-06 17:57:47 UTC
Red Hat Issue Tracker OSP-29338 0 None None None 2023-10-06 18:00:26 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:54:18 UTC

Internal Links: 1622006 1624744 1641825

Description bkopilov 2018-10-18 04:43:34 UTC 
Description of problem:

RHOS14 , ALL in one setup .

Trying to extend volume size while attached to nova instance ,
The action has failed to to policy .

2018-10-17 15:25:28.492 31 DEBUG novaclient.v2.client [req-4345d7d4-2427-4ea9-8391-187027591468 91c621500d564429bab682a378cec799 9c8b3a9623614a0a889e7a9c4c625ac0 - default default] RESP BODY: {"forbidden": {"message": "Policy doesn't allow os_compute_api:os-server-external-events:create to be performed.", "code": 403}} _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:511
2018-10-17 15:25:28.493 31 DEBUG novaclient.v2.client [req-4345d7d4-2427-4ea9-8391-187027591468 91c621500d564429bab682a378cec799 9c8b3a9623614a0a889e7a9c4c625ac0 - default default] POST call to compute for http://10.0.0.107:8774/v2.1/os-server-external-events used request id req-21d497fc-4118-4f90-8054-72a307cfc8aa request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:844
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova [req-4345d7d4-2427-4ea9-8391-187027591468 91c621500d564429bab682a378cec799 9c8b3a9623614a0a889e7a9c4c625ac0 - default default] Failed to notify nova on events: [{'tag': u'ca1e1eb6-401b-4468-8de5-e0b2ea25f0e9', 'name': 'volume-extended', 'server_uuid': u'cd7a83cd-195b-492f-b16d-3d36e7d802cd'}].: Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-21d497fc-4118-4f90-8054-72a307cfc8aa)
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova Traceback (most recent call last):
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/cinder/compute/nova.py", line 148, in _send_events
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova     response = nova.server_external_events.create(events)
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/novaclient/v2/server_external_events.py", line 39, in create
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova     return_raw=True)
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 366, in _create
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova     resp, body = self.api.client.post(url, body=body)
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova     return self.request(url, 'POST', **kwargs)
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova     raise exceptions.from_response(resp, body, url, method)
2018-10-17 15:25:28.493 31 ERROR cinder.compute.nova Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-21d497fc-4118-4f90-8054-72a307cfc8aa)



https://specs.openstack.org/openstack/cinder-specs/specs/pike/extend-attached-volume.html


The same test works on devstack.

Version-Release number of selected component (if applicable):


How reproducible:
Create a volume (non-admin)
create an instance (non-admin)
Attach the volume to the instance.
run volume_extend +1 (size) while attached .


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 bkopilov 2018-10-18 04:50:29 UTC 
Created attachment 1495136 [details]
cinder-volume
Comment 2 Eric Harney 2018-10-18 16:36:28 UTC 
The most likely cause for this is the Cinder service accessing the Nova public API for this call instead of the admin API, and Nova applying the default policy for os-server-external-events:create, which is admin API only.

This can likely be fixed by setting cinder to point to the Nova admin API, but we need to verify whether that could cause issues with other Cinder operations.

The setting would be
[nova]
interface=admin


in cinder.conf if you would like to try it for this case.
Comment 3 Eric Harney 2018-11-02 16:45:24 UTC 
test case:

$ openstack project create demo
$ openstack user create --domain default --password password user1
$ openstack role add --project demo --user user1 member
$ glance image-create --file cirros-0.4.0-x86_64-disk.img --name cirros --progress --container-format bare --disk-format qcow2
$ nova flavor-create small small 64 1 1
$ nova boot --flavor small --image cirros vm --nic none
$ cinder create 1
$ nova volume-attach vm b90298c2-7a68-4fae-8275-1189f0cb4bfd
$ export OS_VOLUME_API_VERSION=3.50
$ cinder extend b90298c2-7a68-4fae-8275-1189f0cb4bfd 2




changing cinder.conf's setting to hit the Nova admin endpoint does fix this.


before:

2018-11-02 16:28:30.002 67 DEBUG novaclient.v2.client [req-6369a9e2-2bee-4a68-b023-68483d472269 f56cf37c54d841308d86b1b1001448e1 53bba5b137a3442e8d639a1b0525a10a - default default] REQ: curl -g -i -X POST http://10.0.0.102:8774/v2.1/os-server-external-events -H "Accept: application/json" -H "Content-Type: application/json" -H "OpenStack-API-Version: compute 2.51" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}8c722b1ee9fef1ab6b314359130bf1d17bb6e25e" -H "X-OpenStack-Nova-API-Version: 2.51" -H "X-OpenStack-Request-ID: req-c73bc4bf-12bf-4dbb-bc01-77d67bb1028a" -d '{"events": [{"tag": "b90298c2-7a68-4fae-8275-1189f0cb4bfd", "name": "volume-extended", "server_uuid": "78823cdc-9ab7-4e57-bbab-733dee6bc55f"}]}' _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:448
2018-11-02 16:28:30.412 67 DEBUG novaclient.v2.client [req-6369a9e2-2bee-4a68-b023-68483d472269 f56cf37c54d841308d86b1b1001448e1 53bba5b137a3442e8d639a1b0525a10a - default default] RESP: [403] Content-Length: 128 Content-Type: application/json; charset=UTF-8 Date: Fri, 02 Nov 2018 16:28:30 GMT OpenStack-API-Version: compute 2.51 Server: Apache Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.51 x-compute-request-id: req-d21d0b61-efd1-412f-8948-3dd6fcb0683d x-openstack-request-id: req-d21d0b61-efd1-412f-8948-3dd6fcb0683d _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:479
2018-11-02 16:28:30.412 67 DEBUG novaclient.v2.client [req-6369a9e2-2bee-4a68-b023-68483d472269 f56cf37c54d841308d86b1b1001448e1 53bba5b137a3442e8d639a1b0525a10a - default default] RESP BODY: {"forbidden": {"message": "Policy doesn't allow os_compute_api:os-server-external-events:create to be performed.", "code": 403}} _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:511
2018-11-02 16:28:30.413 67 DEBUG novaclient.v2.client [req-6369a9e2-2bee-4a68-b023-68483d472269 f56cf37c54d841308d86b1b1001448e1 53bba5b137a3442e8d639a1b0525a10a - default default] POST call to compute for http://10.0.0.102:8774/v2.1/os-server-external-events used request id req-d21d0b61-efd1-412f-8948-3dd6fcb0683d request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:844
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova [req-6369a9e2-2bee-4a68-b023-68483d472269 f56cf37c54d841308d86b1b1001448e1 53bba5b137a3442e8d639a1b0525a10a - default default] Failed to notify nova on events: [{'tag': u'b90298c2-7a68-4fae-8275-1189f0cb4bfd', 'name': 'volume-extended', 'server_uuid': u'78823cdc-9ab7-4e57-bbab-733dee6bc55f'}].: Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-d21d0b61-efd1-412f-8948-3dd6fcb0683d)
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova Traceback (most recent call last):
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/cinder/compute/nova.py", line 148, in _send_events
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova     response = nova.server_external_events.create(events)
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/novaclient/v2/server_external_events.py", line 39, in create
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova     return_raw=True)
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 366, in _create
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova     resp, body = self.api.client.post(url, body=body)
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova     return self.request(url, 'POST', **kwargs)
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova     raise exceptions.from_response(resp, body, url, method)
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-d21d0b61-efd1-412f-8948-3dd6fcb0683d)
2018-11-02 16:28:30.413 67 ERROR cinder.compute.nova



after setting [nova] interface=admin in cinder.conf:


2018-11-02 16:40:30.203 41 DEBUG novaclient.v2.client [req-ca6fae5e-b1b6-478b-89ed-5604a9084844 d2827b1ae2e54b7e884f435cbe539791 ffa239e9ab804dca97aa96b1ed217247 - default default] REQ: curl -g -i -X POST http://172.17.1.10:8774/v2.1/os-server-external-events -H "Accept: application/json" -H "Content-Type: application/json" -H "OpenStack-API-Version: compute 2.51" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}e5d44a19a710cf4db64927afae99db87a215210a" -H "X-OpenStack-Nova-API-Version: 2.51" -H "X-OpenStack-Request-ID: req-b0b8f2ff-c4cc-4939-853c-8d9414aa30d5" -d '{"events": [{"tag": "b90298c2-7a68-4fae-8275-1189f0cb4bfd", "name": "volume-extended", "server_uuid": "78823cdc-9ab7-4e57-bbab-733dee6bc55f"}]}' _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:448
2018-11-02 16:40:30.657 41 DEBUG novaclient.v2.client [req-ca6fae5e-b1b6-478b-89ed-5604a9084844 d2827b1ae2e54b7e884f435cbe539791 ffa239e9ab804dca97aa96b1ed217247 - default default] RESP: [200] Content-Encoding: gzip Content-Length: 159 Content-Type: application/json Date: Fri, 02 Nov 2018 16:40:30 GMT OpenStack-API-Version: compute 2.51 Server: Apache Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding X-OpenStack-Nova-API-Version: 2.51 x-compute-request-id: req-8f4e97c9-91c2-43f7-b66a-a5b13bae8ab8 x-openstack-request-id: req-8f4e97c9-91c2-43f7-b66a-a5b13bae8ab8 _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:479
2018-11-02 16:40:30.658 41 DEBUG novaclient.v2.client [req-ca6fae5e-b1b6-478b-89ed-5604a9084844 d2827b1ae2e54b7e884f435cbe539791 ffa239e9ab804dca97aa96b1ed217247 - default default] RESP BODY: {"events": [{"status": "completed", "tag": "b90298c2-7a68-4fae-8275-1189f0cb4bfd", "name": "volume-extended", "server_uuid": "78823cdc-9ab7-4e57-bbab-733dee6bc55f", "code": 200}]} _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:511
2018-11-02 16:40:30.658 41 DEBUG novaclient.v2.client [req-ca6fae5e-b1b6-478b-89ed-5604a9084844 d2827b1ae2e54b7e884f435cbe539791 ffa239e9ab804dca97aa96b1ed217247 - default default] POST call to compute for http://172.17.1.10:8774/v2.1/os-server-external-events used request id req-8f4e97c9-91c2-43f7-b66a-a5b13bae8ab8 request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:844
2018-11-02 16:40:30.658 41 INFO cinder.compute.nova [req-ca6fae5e-b1b6-478b-89ed-5604a9084844 d2827b1ae2e54b7e884f435cbe539791 ffa239e9ab804dca97aa96b1ed217247 - default default] Nova event response: {u'status': u'completed', u'tag': u'b90298c2-7a68-4fae-8275-1189f0cb4bfd', u'name': u'volume-extended', u'server_uuid': u'78823cdc-9ab7-4e57-bbab-733dee6bc55f', u'code': 200}.
2018-11-02 16:40:30.667 41 DEBUG oslo_db.sqlalchemy.engines [req-ca6fae5e-b1b6-478b-89ed-5604a9084844 d2827b1ae2e54b7e884f435cbe539791 ffa239e9ab804dca97aa96b1ed217247 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:308
2018-11-02 16:40:30.684 41 INFO cinder.volume.manager [req-ca6fae5e-b1b6-478b-89ed-5604a9084844 d2827b1ae2e54b7e884f435cbe539791 ffa239e9ab804dca97aa96b1ed217247 - default default] Extend volume completed successfully.



This confirms that the issue is the default Nova policy for this call, as expected.  Now, we need to determine whether we should change the cinder config or the Nova policy to support this.
Comment 4 Keigo Noha 2018-11-07 08:13:25 UTC 
Hi,

Can you try the configuration in https://access.redhat.com/solutions/3675991?
Comment 5 Alan Bishop 2018-11-08 17:20:00 UTC 
(In reply to Keigo Noha from comment #4)
> Hi,
> 
> Can you try the configuration in https://access.redhat.com/solutions/3675991?

That article describes the need to properly configure the nova endpoint used by cinder. As Eric mentions in comment #3, the critical thing is making sure cinder doesn't use the public endpoint, which is the default configuration. The fix for this is having TripleO configure cinder to configure the proper interface.
Comment 22 Avi Avraham 2018-11-27 14:21:09 UTC 
We heat the following error while testing the new package 
==== compute-1 computes ====
/var/log/containers/nova/nova-compute.log:308:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server [req-5e221be6-dd6e-4dfe-a68b-dd539ad3ae25 8941550cbc2a41549e1fcfa5a3df34f3 a884e1d8126848b799a0339de060a06c - default default] Exception during message handling: ExtendVolumeNotSupported: Volume size extension is not supported by the hypervisor.
/var/log/containers/nova/nova-compute.log:309:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log:310:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
/var/log/containers/nova/nova-compute.log:311:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
/var/log/containers/nova/nova-compute.log:312:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
/var/log/containers/nova/nova-compute.log:313:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
/var/log/containers/nova/nova-compute.log:314:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
/var/log/containers/nova/nova-compute.log:315:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
/var/log/containers/nova/nova-compute.log:316:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
/var/log/containers/nova/nova-compute.log:317:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
/var/log/containers/nova/nova-compute.log:318:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:319:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:320:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:321:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:322:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
/var/log/containers/nova/nova-compute.log:323:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
/var/log/containers/nova/nova-compute.log:324:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8123, in external_instance_event
/var/log/containers/nova/nova-compute.log:325:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     self.extend_volume(context, instance, event.tag)
/var/log/containers/nova/nova-compute.log:326:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function
/var/log/containers/nova/nova-compute.log:327:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:328:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
/var/log/containers/nova/nova-compute.log:329:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
/var/log/containers/nova/nova-compute.log:330:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:331:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:332:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:333:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:334:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
/var/log/containers/nova/nova-compute.log:335:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:336:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8081, in extend_volume
/var/log/containers/nova/nova-compute.log:337:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     instance)
/var/log/containers/nova/nova-compute.log:338:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1630, in extend_volume
/var/log/containers/nova/nova-compute.log:339:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server     raise exception.ExtendVolumeNotSupported()
/var/log/containers/nova/nova-compute.log:340:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server ExtendVolumeNotSupported: Volume size extension is not supported by the hypervisor.
/var/log/containers/nova/nova-compute.log:341:2018-11-26 16:55:38.802 1 ERROR oslo_messaging.rpc.server
Comment 24 Tzach Shefi 2018-11-28 09:45:13 UTC 
What am I missing here, used a netapp iscsi 1G vol attached to an instance.


$  export OS_VOLUME_API_VERSION=3.50
$ cinder extend 33bc1b92-a412-40a0-b120-9e4e944c0880 2
$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name         | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 33bc1b92-a412-40a0-b120-9e4e944c0880 | in-use    | -            | 1    | tripleo     | false    | 02a0def9-b109-42b1-8ff5-489302964233 |

Remains 1G 


Tested on Puddle  2018-11-22.2
All the needed fixed-in are included. 
openstack-tripleo-heat-templates-9.0.1-0.20181013060891.el7ost.noarch
puppet-tripleo-9.3.1-0.20181010034751.157eaab.el7ost.noarch
puppet-cinder-13.3.1-0.20181013114721.25b1ba3.el7ost.noarch


/var/log/containers/cinder/cinder-scheduler.log:2018-11-28 09:22:05.936 1 ERROR cinder.scheduler.manager [req-0360ecb3-6bab-4800-8b29-9fe83ab61a38 ac4b9116aafc4502ac1c95d18ae8c2f7 c1d6a0720f844f11a637be69ca465311 - default default] Failed to schedule_extend_volume: Cannot place volume 33bc1b92-a412-40a0-b120-9e4e944c0880 on hostgroup@tripleo_netapp#rhos_cinder.: NoValidBackend: Cannot place volume 33bc1b92-a412-40a0-b120-9e4e944c0880 on hostgroup@tripleo_netapp#rhos_cinder.

OK great figured it out, not supported on -> 
NetApp Data ONTAP Driver (iSCSI, NFS, FC): missing 

On the plus side
HPE 3PAR Storage Driver (FC, iSCSI): complete
Kaminario Storage Driver (iSCSI, FC): complete
Logical Volume Manager (LVM) Reference Driver (iSCSI): complete

Avi you can only test this on LVM or K2, 3par hasn't been used yet not operational.
Comment 25 Tzach Shefi 2018-11-30 16:21:48 UTC 
Verified on:
openstack-tripleo-heat-templates-9.0.1-0.20181013060894.el7ost.noarch
puppet-tripleo-9.3.1-0.20181010034751.157eaab.el7ost.noarch
puppet-cinder-13.3.1-0.20181013114721.25b1ba3.el7ost.noarch

All operations below were done on demo project, with demo (member) user. 

#nova boot inst2 --flavor tiny --image cirros --nic net-id=bfd3c5ba-dd40-41cb-b7b4-53f9b425bd32



Created a volume
cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID                                   | Status    | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 1a003f59-9c65-4b58-8f6e-d46b7b9e8018 | available 


Attach volume to instance:
nova volume-attach 4eef2582-9d21-44ba-ade8-f36a8d80ab05 1a003f59-9c65-4b58-8f6e-d46b7b9e8018 auto 
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 1a003f59-9c65-4b58-8f6e-d46b7b9e8018 |
| serverId | 4eef2582-9d21-44ba-ade8-f36a8d80ab05 |
| volumeId | 1a003f59-9c65-4b58-8f6e-d46b7b9e8018 |
+----------+--------------------------------------+


Now extend volume 
export OS_VOLUME_API_VERSION=3.50
cinder extend 1a003f59-9c65-4b58-8f6e-d46b7b9e8018 3

Review cinder volumes new size, it should be 3G

cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID                                   | Status | Name | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 1a003f59-9c65-4b58-8f6e-d46b7b9e8018 | in-use | -    | 3    | tripleo     | false    | 4eef2582-9d21-44ba-ade8-f36a8d80ab05 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+

As see the volume was extended from 1G to 3G with a none admin user. 
Looking good to verify.
Comment 26 Alan Bishop 2018-12-03 19:02:17 UTC 
*** Bug 1622006 has been marked as a duplicate of this bug. ***
Comment 29 errata-xmlrpc 2019-01-11 11:54:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045
Note You need to log in before you can comment on or make changes to this bug.