Bug 1889628 - [RHOSP 16.1] [cinder] PowerStore Cinder Driver does not have ‘get_volume_stats’
Summary: [RHOSP 16.1] [cinder] PowerStore Cinder Driver does not have ‘get_volume_stats’
Status: ON_QA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
Target Milestone: z3
: 16.1 (Train on RHEL 8.2)
Assignee: Alan Bishop
QA Contact: Tzach Shefi
Chuck Copello
: 1889627 (view as bug list)
Depends On:
Blocks: 1862541
TreeView+ depends on / blocked
Reported: 2020-10-20 08:50 UTC by Sam Wan
Modified: 2020-11-18 16:50 UTC (History)
19 users (show)

Fixed In Version: openstack-cinder-15.3.1-4.el8ost
Doc Type: Bug Fix
Doc Text:
Cause: The driver for cinder's Dell EMC PowerStore back end failed to periodically report statistics for its storage pools, and this prevented the cinder-secheduler from placing volumes on the back end. Consequence: Volumes cannot be created on a PowerStore cinder backend. Fix: Now the PowerStore driver periodically reports its storage pool statistics. Result: Volumes can be created on a PowerStore backend.
Clone Of:
Last Closed:
Target Upstream Version:
tshefi: automate_bug-

Attachments (Terms of Use)
Add get_volume_stats method to PowerStore driver (1.32 KB, patch)
2020-10-20 11:50 UTC, Sam Wan
no flags Details | Diff

System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1862541 medium ON_QA [RHOSP 16.1] [cinder] Backport: PowerStore Cinder Driver 2020-11-05 13:50:41 UTC

Internal Links: 1862541

Description Sam Wan 2020-10-20 08:50:58 UTC
Description of problem:

The backported powerstore cinder driver does not have ‘get_volume_stats’which causes it being filtered out by scheduler.

Version-Release number of selected component (if applicable):
VERSION = "1.0.0

How reproducible:

Steps to Reproduce:
1. deploy with latest RHOSP16.1.2 rc container images: rh-osbs/rhosp16-openstack-cinder-volume:16.1_20201003.1 

2. enable powerstore cinder backend.
$ cinder service-list
| Binary           | Host                          | Zone | Status   | State | Updated_at                 | Disabled Reason |
| cinder-backup    | elabrhosp85ctl0               | nova | enabled  | up    | 2020-10-20T08:45:57.000000 | -               |
| cinder-scheduler | elabrhosp85ctl0               | nova | enabled  | up    | 2020-10-20T08:45:56.000000 | -               |
| cinder-volume    | elabrhosp85ctl0@powerstorefc1 | nova | enabled  | up    | 2020-10-20T08:46:00.000000 | -               |
| cinder-volume    | elabrhosp85ctl0@powerstorefc2 | nova | enabled  | up    | 2020-10-20T08:46:00.000000 | -               |

3.  cinder get-pools to check pool stats.

Actual results:
no pool stats reported by powerstore cinder driver.

Expected results:
pool stats should be reported by powerstore cinder driver like other drivers,

Additional info:

Before Victoria release, all drivers report stats using ‘get_volume_stats’ at initialization.
This has been changed in Victoria, 
since all drivers need this, it's been moved to the base volume driver.
https://review.opendev.org/#/c/656888/ Move get_volume_stats impl to the base volume driver

so powerstore driver does not need this function in Victoria.
However when it's been backported to train(RHOSP16.1.2), it causes issue.

()[root@elabrhosp85ctl0 dell_emc]# ls -l
total 88
-rw-r--r--. 1 root root     0 Sep 14 17:37 __init__.py
drwxr-xr-x. 3 root root   178 Oct  3 01:59 powermax
drwxr-xr-x. 1 root root    28 Oct 20 01:12 powerstore
-rw-r--r--. 1 root root 28840 Sep 14 17:37 ps.py
drwxr-xr-x. 2 root root   196 Oct  3 01:59 __pycache__
drwxr-xr-x. 3 root root   160 Oct  3 01:59 sc
drwxr-xr-x. 3 root root   134 Oct  3 01:59 unity
drwxr-xr-x. 3 root root   187 Oct  3 01:59 vnx
drwxr-xr-x. 3 root root   139 Oct  3 01:59 vxflexos
-rw-r--r--. 1 root root 54556 Sep 14 17:37 xtremio.py

()[root@elabrhosp85ctl0 dell_emc]# grep -r 'def get_volume_stats' *
powermax/fc.py:    def get_volume_stats(self, refresh=False):
powermax/iscsi.py:    def get_volume_stats(self, refresh=False):
sc/storagecenter_common.py:    def get_volume_stats(self, refresh=False):
unity/driver.py:    def get_volume_stats(self, refresh=False):
vnx/driver.py:    def get_volume_stats(self, refresh=False):
vxflexos/driver.py:    def get_volume_stats(self, refresh=False):
xtremio.py:    def get_volume_stats(self, refresh=False):
()[root@elabrhosp85ctl0 dell_emc]#

we can see all dell emc cinder drivers have this get_volume_status except powerstore.

Comment 1 Sam Wan 2020-10-20 09:07:15 UTC
This prevents powerstore from serving any request because it's filtered out by scheduler.

2020-10-19 15:12:39.763 7 INFO cinder.scheduler.base_filter [req-44a573bc-9866-45b9-997f-bf0685245eba a798b89fcb1541fa8d2627139815620b fab7b689c75d4543ae700de44dcd1441 - default default] Filtering removed all hosts for the request with volume ID 'cc87b0bf-8d31-440c-85bf-012525cd4498'. Filter results: AvailabilityZoneFilter: (start: 0, end: 0), CapacityFilter: (start: 0, end: 0), CapabilitiesFilter: (start: 0, end: 0)
2020-10-19 15:12:39.763 7 WARNING cinder.scheduler.filter_scheduler [req-44a573bc-9866-45b9-997f-bf0685245eba a798b89fcb1541fa8d2627139815620b fab7b689c75d4543ae700de44dcd1441 - default default] No weighed backend found for volume with properties: {'id': '1a2e6a5e-6ad2-45c8-8845-7634892211f6', 'name': 'powerstorefc1', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'powerstorefc1'}, 'qos_specs_id': None, 'created_at': '2020-10-19T15:11:28.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}
2020-10-19 15:12:39.764 7 INFO cinder.message.api [req-44a573bc-9866-45b9-997f-bf0685245eba a798b89fcb1541fa8d2627139815620b fab7b689c75d4543ae700de44dcd1441 - default default] Creating message record for request_id = req-44a573bc-9866-45b9-997f-bf0685245eba
2020-10-19 15:12:39.792 7 ERROR cinder.scheduler.flows.create_volume [req-44a573bc-9866-45b9-997f-bf0685245eba a798b89fcb1541fa8d2627139815620b fab7b689c75d4543ae700de44dcd1441 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available

Comment 2 Luigi Toscano 2020-10-20 09:08:44 UTC
*** Bug 1889627 has been marked as a duplicate of this bug. ***

Comment 3 Luigi Toscano 2020-10-20 09:13:47 UTC
As a fix/follow-up for the backport, I guess we can expect to receive a patch soon?

Comment 4 Sam Wan 2020-10-20 11:50:43 UTC
Created attachment 1722873 [details]
Add get_volume_stats method to PowerStore driver

Comment 5 arkady kanevsky 2020-10-20 12:25:45 UTC
Can this be pulled into z2?
Without it driver does not work.

Comment 6 Luigi Toscano 2020-10-20 12:29:58 UTC
(In reply to arkady kanevsky from comment #5)
> Can this be pulled into z2?
> Without it driver does not work.

Unfortunately it's a bit too late for 16.1.2.

Comment 7 arkady kanevsky 2020-10-20 12:31:40 UTC
Can this be released async?

Comment 8 Gregory Charot 2020-10-20 14:11:13 UTC
async are for critical bugs / CVE so unfortunately it's most likely going to be 16.1.3

If we have a business impact we can potentially ship a hot fix when the fix is ready and bits are available downstream

Comment 11 Alan Bishop 2020-10-27 21:16:19 UTC
The BZ is on track to be fixed in the 16.1.3 release.

Note You need to log in before you can comment on or make changes to this bug.