Created attachment 1736653 [details] keystone get tokens answer Description of problem: OpenStack Cinder oVirt client use wrong uri (without project_id). Version-Release number of selected component (if applicable): 4.4.3.12 How reproducible: Steps to Reproduce: 1. Administration -> Providers 2. Add 3. Connect OpenStack Block Storage Domain Type Actual results: Cinder integration don't work. Expected results: Cinder integration work. Additional info: I was upgrade our ovirt-engine from 4.3 to latest 4.4.3.12. Currently our clusters can't start VM, create disks, resize disks, etc. Only migration works. The root cause: ovirt missing project_id in API call, reproducer is: 1. Go to Storage -> Disks 2. New 3. Select Cinder 4. Volume-type is empty (list of '/types' should be present) ovirt 4.3 call (see '07f5bf3f6dc64b85988c3779654e175e'): 17910:2020-12-02 20:43:32.087 1949 INFO eventlet.wsgi.server [req-7267b4fd-9659-4380-9297-4582ece3fe23 - - - - -] 192.168.101.10 "POST /v2/07f5bf3f6dc64b85988c3779654e175e/volumes/e3df2f84-2206-4165-9001-bcace8613315/action HTTP/1.1" status: 200 len: 777 time: 0.4810839 ovirt 4.4 call (don't have project_id string in query): 2020-12-05 14:09:24.155 2031 INFO eventlet.wsgi.server [req-71b47b34-8ed7-410a-812f-8f662f9f4037 - - - - -] 192.168.101.10 "POST /v2/volumes/e3df2f84-2206-4165-9001-bcace8613315/action HTTP/1.1" status: 404 len: 333 time: 0.2505009 provider configuration in engine database: engine=# select * from providers where name='cinder_ceph_backened'; -[ RECORD 1 ]---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | 9c5800e6-5a88-403a-9c57-a501714fe816 name | cinder_ceph_backened description | OpenStack Cinder with Ceph Backend url | http://192.168.101.20:8776 provider_type | OPENSTACK_VOLUME auth_required | t auth_username | admin auth_password | OUDZtXsI4eOT69UKYI2DNqTFmN3c08XNAbbi3PQHq2Np319yURcIjhOJ81lKUo+T+pa/e6d5XUbPZmwulCK21fU5UrY2uJBSg8GXaVH23os7BmZzx+7V0V82LLBFYWUAAACeXY0hu9UGgQiMd0L7wPS0hU23iSib/BWnCcxY6h4ooQ0/pfKNZ10so5tKin/mAgMHNmX2YtqiYaQgZTYpDcIf9JnfqsiJKUW3xekPzTJQCIUEDbX/1Jpp5sJCW5aFHDSiy1I9CU/etAcqrzf6JMN8Mfn6X4VZjXqrg4YQ+QD6TiTwOAS7u7oJwCYopdRHvGNspc2YbPykN62NgFWwmg== _create_date | 2017-04-05 19:30:57.045699+07 _update_date | 2020-12-05 13:27:31.320646+07 custom_properties | tenant_name | admin plugin_type | auth_url | http://192.168.101.20:5000/v2.0 additional_properties | read_only | f is_unmanaged | f auto_sync | f user_domain_name | project_name | project_domain_name | I also try to switch to API v3.0: engine=# select * from providers where name='cinder_ceph_backened'; -[ RECORD 1 ]---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | 9c5800e6-5a88-403a-9c57-a501714fe816 name | cinder_ceph_backened description | OpenStack Cinder with Ceph Backend url | http://192.168.101.20:8776 provider_type | OPENSTACK_VOLUME auth_required | t auth_username | admin auth_password | XC3eTjt9okk5aqOPB1brOP6BgwXqkbNvquNTZ55nWzY33nJnlVEE3V1YabgTfXcswzH8sPdBxj97cjxKXMKYJoF0hOVV/+O8LeCYATuGRAiNEWUfrJxevcpn9I0g4bi18REjHfk8zBZT1IM6KbD5Fmohx8O3torBTpKO2zZu4gu0LX1shzltCpgp4jGhtChMvsNeW42bmW4S3zA1JVZPY7kyKZPJcvOmhap0FpBurVFpAquiB6fs6ZIK/WmgcfX5bJO5GEDLX1TRVBj6W/6xm4kmxrp/XrGuWBhetS6iGYXlAc17nBqJ4p0k3V6PMzaWX8uwpP5bKQp3UFU33Sku4Q== _create_date | 2017-04-05 19:30:57.045699+07 _update_date | 2020-12-05 18:33:17.722833+07 custom_properties | tenant_name | plugin_type | auth_url | http://192.168.101.20:35357/v3 additional_properties | read_only | f is_unmanaged | f auto_sync | f user_domain_name | Default project_name | admin project_domain_name | Default Result the same. Currently we stop upgrade, because if something wrong we can't even restart our VM's. This Storage Domain contains 196 Disks for a 109 VM's. P.S.: attached `auth/tokens` of our "openstack for ovirt" answer (where project_id for 'admin' is present).
Did you check cinderlib integration for use with ceph (https://www.ovirt.org/develop/release-management/features/storage/cinderlib-integration.html)? It's going to replace the aging cinder integration which doesn't work with recent Openstack versions
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.
Michal, yes I know about cinderlib. Exactly for this feature was I start EL8 + oVirt 4.4 upgrade. We (oVirt admin's with large Cinder integrations) already mentioned almost two years ago, about "we need migration guide" before remove old external provider for OpenStack [1]. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1539837#c6
1. I found 1797528 [1] where disk can't be deleted from Managed Storage Block pool. 2. AFAIU current Managed Storage Block realization support only krbd (kernel rbd) driver - it's also not a option, because kernel client is always lagging behind librbd, and every update\bugfix we should reboot whole host instead simple migration of all vms and then migrate it back. 3. Also with krbd host will be use kernel page cache, and will not be unmounted if VM will crash (qemu with librbd is one userland process). So I see only regressions for now, hope we'll found some code owner who can catch this oVirt 4.4 only bugs. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1797528
We still need patch for this blocker. Thanks
I was found a bunch of patches in BZ#1704349, where tenantId just dropped from the code and tests was perfromed without auth... ``` - OpenStackRequest<VolumeTypes> listRequest = getClient(getTenantId()).volumeTypes().list(); + OpenStackRequest<VolumeTypes> listRequest = getClient().volumeTypes().list(); ``` This exact our problem, I CC'ed Dominik Holler...
(In reply to Konstantin Shalygin from comment #6) > I was found a bunch of patches in BZ#1704349, where tenantId just dropped > from the code and tests was perfromed without auth... > > ``` > - OpenStackRequest<VolumeTypes> listRequest = > getClient(getTenantId()).volumeTypes().list(); > + OpenStackRequest<VolumeTypes> listRequest = > getClient().volumeTypes().list(); > ``` > > This exact our problem, I CC'ed Dominik Holler... Konstantin, thanks for your analysis, I just can acknowledge that the change provider: Support OpenStack Identity API v3 introduced the regression of missing the {project_id} during the construction of the url of the OpenStack Block Storage API.
(In reply to Dominik Holler from comment #7) > Konstantin, thanks for your analysis, I just can acknowledge that the change > provider: Support OpenStack Identity API v3 > introduced the regression of missing the {project_id} during the > construction of the url of the OpenStack Block Storage API. Thank you Dominik, glad to hear that the regression acknowledged.
Any news for this?
(In reply to Konstantin Shalygin from comment #9) > Any news for this? The "old" way of using Cinder as an external provider is now deprecated. Please see the new Cinder integration using cinderlib AKA as Managed Block Storage. For more info, you can see - https://www.ovirt.org/develop/release-management/features/storage/cinderlib-integration.html
(In reply to Eyal Shenitzky from comment #10) > (In reply to Konstantin Shalygin from comment #9) > > Any news for this? > > The "old" way of using Cinder as an external provider is now deprecated. Please open a proper deprecation bug and get it documented as deprecated feature. > > Please see the new Cinder integration using cinderlib AKA as Managed Block > Storage. > > For more info, you can see - > https://www.ovirt.org/develop/release-management/features/storage/cinderlib- > integration.html This feature page is obsolete and not a valid user documentation. I'm working with doc team to get it properly added to oVirt Installation Guide.
(In reply to Sandro Bonazzola from comment #11) > (In reply to Eyal Shenitzky from comment #10) > > (In reply to Konstantin Shalygin from comment #9) > > > Any news for this? > > > > The "old" way of using Cinder as an external provider is now deprecated. > > Please open a proper deprecation bug and get it documented as deprecated > feature. Already opened by Tal - bug 1899453.