Bug 1889628

Summary: [RHOSP 16.1] [cinder] PowerStore Cinder Driver does not have ‘get_volume_stats’
Product: Red Hat OpenStack Reporter: Sam Wan <Sam.Wan>
Component: openstack-cinderAssignee: Alan Bishop <abishop>
Status: CLOSED CURRENTRELEASE QA Contact: Tzach Shefi <tshefi>
Severity: urgent Docs Contact: Chuck Copello <ccopello>
Priority: urgent    
Version: 16.1 (Train)CC: abishop, arkady_kanevsky, a.stripeikis, gael_rehault, gcharot, geguileo, Ivan.Pchelintsev, joflynn, kholtz, kurt_hey, ltoscano, morazi, pcaruana, pmannidi, rajini.karthik, rcernin, sam.wan, scohen, tom.zhao, vladislav.belogrudov
Target Milestone: z3Keywords: OtherQA, Triaged, ZStream
Target Release: 16.1 (Train on RHEL 8.2)Flags: tshefi: automate_bug-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-15.3.1-4.el8ost Doc Type: Bug Fix
Doc Text:
Before this update, the driver for the Block Storage service (cinder) Dell EMC PowerStore back end failed to periodically report statistics for its storage pools. This stopped the cinder-scheduler from placing volumes on the back end. With this update, the PowerStore driver periodically reports its storage pool statistics and you can create volumes on a PowerStore back end.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-12-15 19:11:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1862541    
Attachments:
Description Flags
Add get_volume_stats method to PowerStore driver none

Description Sam Wan 2020-10-20 08:50:58 UTC 
Description of problem:

The backported powerstore cinder driver does not have ‘get_volume_stats’which causes it being filtered out by scheduler.


Version-Release number of selected component (if applicable):
VERSION = "1.0.0

How reproducible:


Steps to Reproduce:
1. deploy with latest RHOSP16.1.2 rc container images: rh-osbs/rhosp16-openstack-cinder-volume:16.1_20201003.1 

2. enable powerstore cinder backend.
$ cinder service-list
+------------------+-------------------------------+------+----------+-------+----------------------------+-----------------+
| Binary           | Host                          | Zone | Status   | State | Updated_at                 | Disabled Reason |
+------------------+-------------------------------+------+----------+-------+----------------------------+-----------------+
| cinder-backup    | elabrhosp85ctl0               | nova | enabled  | up    | 2020-10-20T08:45:57.000000 | -               |
| cinder-scheduler | elabrhosp85ctl0               | nova | enabled  | up    | 2020-10-20T08:45:56.000000 | -               |
| cinder-volume    | elabrhosp85ctl0@powerstorefc1 | nova | enabled  | up    | 2020-10-20T08:46:00.000000 | -               |
| cinder-volume    | elabrhosp85ctl0@powerstorefc2 | nova | enabled  | up    | 2020-10-20T08:46:00.000000 | -               |

3.  cinder get-pools to check pool stats.

Actual results:
no pool stats reported by powerstore cinder driver.

Expected results:
pool stats should be reported by powerstore cinder driver like other drivers,


Additional info:

=====================================
Before Victoria release, all drivers report stats using ‘get_volume_stats’ at initialization.
This has been changed in Victoria, 
since all drivers need this, it's been moved to the base volume driver.
https://review.opendev.org/#/c/656888/ Move get_volume_stats impl to the base volume driver

so powerstore driver does not need this function in Victoria.
However when it's been backported to train(RHOSP16.1.2), it causes issue.

------------------------------------------------
()[root@elabrhosp85ctl0 dell_emc]# ls -l
total 88
-rw-r--r--. 1 root root     0 Sep 14 17:37 __init__.py
drwxr-xr-x. 3 root root   178 Oct  3 01:59 powermax
drwxr-xr-x. 1 root root    28 Oct 20 01:12 powerstore
-rw-r--r--. 1 root root 28840 Sep 14 17:37 ps.py
drwxr-xr-x. 2 root root   196 Oct  3 01:59 __pycache__
drwxr-xr-x. 3 root root   160 Oct  3 01:59 sc
drwxr-xr-x. 3 root root   134 Oct  3 01:59 unity
drwxr-xr-x. 3 root root   187 Oct  3 01:59 vnx
drwxr-xr-x. 3 root root   139 Oct  3 01:59 vxflexos
-rw-r--r--. 1 root root 54556 Sep 14 17:37 xtremio.py

()[root@elabrhosp85ctl0 dell_emc]# grep -r 'def get_volume_stats' *
powermax/fc.py:    def get_volume_stats(self, refresh=False):
powermax/iscsi.py:    def get_volume_stats(self, refresh=False):
sc/storagecenter_common.py:    def get_volume_stats(self, refresh=False):
unity/driver.py:    def get_volume_stats(self, refresh=False):
vnx/driver.py:    def get_volume_stats(self, refresh=False):
vxflexos/driver.py:    def get_volume_stats(self, refresh=False):
xtremio.py:    def get_volume_stats(self, refresh=False):
()[root@elabrhosp85ctl0 dell_emc]#
-----------------------------

we can see all dell emc cinder drivers have this get_volume_status except powerstore.
Comment 1 Sam Wan 2020-10-20 09:07:15 UTC 
This prevents powerstore from serving any request because it's filtered out by scheduler.

==============================
2020-10-19 15:12:39.763 7 INFO cinder.scheduler.base_filter [req-44a573bc-9866-45b9-997f-bf0685245eba a798b89fcb1541fa8d2627139815620b fab7b689c75d4543ae700de44dcd1441 - default default] Filtering removed all hosts for the request with volume ID 'cc87b0bf-8d31-440c-85bf-012525cd4498'. Filter results: AvailabilityZoneFilter: (start: 0, end: 0), CapacityFilter: (start: 0, end: 0), CapabilitiesFilter: (start: 0, end: 0)
2020-10-19 15:12:39.763 7 WARNING cinder.scheduler.filter_scheduler [req-44a573bc-9866-45b9-997f-bf0685245eba a798b89fcb1541fa8d2627139815620b fab7b689c75d4543ae700de44dcd1441 - default default] No weighed backend found for volume with properties: {'id': '1a2e6a5e-6ad2-45c8-8845-7634892211f6', 'name': 'powerstorefc1', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'powerstorefc1'}, 'qos_specs_id': None, 'created_at': '2020-10-19T15:11:28.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}
2020-10-19 15:12:39.764 7 INFO cinder.message.api [req-44a573bc-9866-45b9-997f-bf0685245eba a798b89fcb1541fa8d2627139815620b fab7b689c75d4543ae700de44dcd1441 - default default] Creating message record for request_id = req-44a573bc-9866-45b9-997f-bf0685245eba
2020-10-19 15:12:39.792 7 ERROR cinder.scheduler.flows.create_volume [req-44a573bc-9866-45b9-997f-bf0685245eba a798b89fcb1541fa8d2627139815620b fab7b689c75d4543ae700de44dcd1441 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available
==============================
Comment 2 Luigi Toscano 2020-10-20 09:08:44 UTC 
*** Bug 1889627 has been marked as a duplicate of this bug. ***
Comment 3 Luigi Toscano 2020-10-20 09:13:47 UTC 
As a fix/follow-up for the backport, I guess we can expect to receive a patch soon?
Comment 4 Sam Wan 2020-10-20 11:50:43 UTC 
Created attachment 1722873 [details]
Add get_volume_stats method to PowerStore driver
Comment 5 arkady kanevsky 2020-10-20 12:25:45 UTC 
Can this be pulled into z2?
Without it driver does not work.
Comment 6 Luigi Toscano 2020-10-20 12:29:58 UTC 
(In reply to arkady kanevsky from comment #5)
> Can this be pulled into z2?
> Without it driver does not work.

Unfortunately it's a bit too late for 16.1.2.
Comment 7 arkady kanevsky 2020-10-20 12:31:40 UTC 
Can this be released async?
Comment 8 Gregory Charot 2020-10-20 14:11:13 UTC 
async are for critical bugs / CVE so unfortunately it's most likely going to be 16.1.3

If we have a business impact we can potentially ship a hot fix when the fix is ready and bits are available downstream
Comment 11 Alan Bishop 2020-10-27 21:16:19 UTC 
The BZ is on track to be fixed in the 16.1.3 release.
Comment 28 Luigi Toscano 2020-12-15 19:11:17 UTC 
Released together with openstack-cinder-15.3.1-5.