Bug 1416920 - storops library missing for Cinder VNX driver
Summary: storops library missing for Cinder VNX driver
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: RDO
Classification: Community
Component: Package Review
Version: trunk
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: trunk
Assignee: Alan Pevec
QA Contact: hguemar
URL:
Whiteboard:
Depends On: 1416902 1418428 1418429 1422203
Blocks: RDO-OCATA 1412759
TreeView+ depends on / blocked
 
Reported: 2017-01-26 19:31 UTC by Eric Harney
Modified: 2017-03-30 00:44 UTC (History)
10 users (show)

Fixed In Version: python-storops-0.4.7-2.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-30 00:44:48 UTC


Attachments (Terms of Use)
python-storops.spec (2.04 KB, text/plain)
2017-02-09 21:15 UTC, Eric Harney
no flags Details
python-storops.spec (2.10 KB, text/plain)
2017-02-09 21:22 UTC, Eric Harney
no flags Details
python-storops.spec (3.65 KB, text/plain)
2017-02-14 20:36 UTC, Eric Harney
no flags Details

Description Eric Harney 2017-01-26 19:31:42 UTC
Description of problem:
The storops library (https://github.com/emc-openstack/storops) is now required by the EMC VNX cinder driver to manage VNX arrays, as of Newton.  If it is not installed, the VNX driver fails with:

2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager [req-cf47020a-cb83-4bc7-8b4a-573fda411de2 - - - - -] Failed to initialize driver.
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager Traceback (most recent call last):
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 414, in init_host
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager     self.driver.do_setup(ctxt)
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/vnx/driver.py", line 98, in do_setup
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager     self.adapter.do_setup()
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/vnx/adapter.py", line 1214, in do_setup
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager     super(ISCSIAdapter, self).do_setup()
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/vnx/adapter.py", line 71, in do_setup
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager     self.config.storage_vnx_security_file_dir)
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/vnx/client.py", line 86, in __init__
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager     raise exception.VolumeBackendAPIException(message=msg)
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager VolumeBackendAPIException: storops Python library is not installed.
2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager 


The storops library needs to be packaged, and has at least a few more missing subdependencies:
  - python-bitmath
  - python-cachez
  - python-queuelib
  - python-retryz


Version-Release number of selected component (if applicable):
Cinder Newton


Steps to Reproduce:
1.  Configure EMC VNX driver
2.  Start cinder volume service

Comment 1 Alan Pevec 2017-01-26 21:58:27 UTC
Here is preliminary analysis of the current RPM packaging state of the web-of-deps:

>  - python-bitmath
   1.3.1 in Fedora and EPEL7, can be imported to CBS Koji for RDO deps repo
   Missing BR: Error: No Package found for python-progressbar

>  - python-queuelib
   1.4.2 in Fedora, needs to be forked in github.com/rdo-common/ to make it build on EL7 (conditionalize py3 at least)

>  - python-cachez
   not in Fedora, needs Fedora Package Review

>  - python-retryz
   not in Fedora, needs Fedora Package Review

Comment 2 Alan Pevec 2017-01-26 22:35:52 UTC
Built in CBS:
* progressbar http://cbs.centos.org/koji/buildinfo?buildID=15343
* bitmath http://cbs.centos.org/koji/buildinfo?buildID=15332
* queuelib http://cbs.centos.org/koji/buildinfo?buildID=15345
  RDO EL7 fix https://github.com/rdo-common/python-queuelib/commit/d30a88bb953ad536935bfea1dc0db46547aaa6db

> >  - python-cachez
>    not in Fedora, needs Fedora Package Review
> 
> >  - python-retryz
>    not in Fedora, needs Fedora Package Review

Eric, would like to open Fedora/PackageReview for those two? You can use pyp2rpm for the initial spec or start with example spec in https://fedoraproject.org/wiki/Packaging:Python

When you have review bz please add them in Depends On here.

Comment 3 Alan Pevec 2017-01-26 23:10:37 UTC
BR for cachez and retryz:
* hamcrest http://cbs.centos.org/koji/buildinfo?buildID=15346

Comment 4 Alan Pevec (Fedora) 2017-02-09 05:54:20 UTC
Deps should be all set now, please post link to the initial spec/SRPM here.

Comment 5 Eric Harney 2017-02-09 21:15:37 UTC
Created attachment 1248903 [details]
python-storops.spec

Comment 6 Eric Harney 2017-02-09 21:16:44 UTC
Comment on attachment 1248903 [details]
python-storops.spec

Wrong version.

Comment 7 Eric Harney 2017-02-09 21:22:58 UTC
Created attachment 1248905 [details]
python-storops.spec

Comment 8 Haïkel Guémar 2017-02-10 16:53:55 UTC
SPEC URL: https://hguemar.fedorapeople.org/reviews/python-storops.spec
SRPM URL: https://hguemar.fedorapeople.org/reviews/python-storops-0.4.4-1.el7.src.rpm

I fixed download URL since the one in spec file doesn't work, I also started a scratch build in CBS.
https://cbs.centos.org/koji/taskinfo?taskID=159597

Comment 9 Haïkel Guémar 2017-02-10 17:10:52 UTC
Package looks sane, but it's not running unit tests.
From quick glance:
- missing BR to run test: python-pytest-xdist, python-future, python-fasteners, python-hamcrest
- potential conflict with requests versions (does not prevent tests running)

Comment 10 Eric Harney 2017-02-10 17:24:30 UTC
(In reply to Haïkel Guémar from comment #9)
> Package looks sane, but it's not running unit tests.

I don't think that the unit tests will currently function in our package build environment.

Running them locally shows lots of failures to open files like "storops-0.4.4/test/vnx/testdata/block_output/security_-certificate_-getLevel.txt" which do not exist.

This data is present here, but not in the source on PyPI:
https://github.com/emc-openstack/storops/tree/develop/test/vnx/testdata/block_output

Comment 11 Alan Pevec (Fedora) 2017-02-14 01:18:22 UTC
Initial package could be w/o unit tests but give we don't have CI coverage for this HW specific library, it should be added so we have at least minimal coverage on the unittesting level. Please open upstream issue to fix the release tarball and reference it as TODO in commented-out %check section.

Also update to the latest version https://github.com/emc-openstack/storops/releases/tag/r0.4.6

Comment 12 Eric Harney 2017-02-14 02:02:04 UTC
The 0.4.6 update to the storops library has added a requirement for persist-queue, which is not yet packaged for Fedora.

https://pypi.python.org/pypi/persist-queue/

Comment 13 Eric Harney 2017-02-14 02:47:57 UTC
Opened https://github.com/emc-openstack/storops/issues/92 for unit test issues.

Comment 14 Eric Harney 2017-02-14 20:36:40 UTC
Created attachment 1250329 [details]
python-storops.spec

Updated to 0.4.7 and now runs unit tests.

Comment 15 Alan Pevec (Fedora) 2017-02-15 00:30:14 UTC
Built in CBS:
* persist-queue https://cbs.centos.org/koji/buildinfo?buildID=15690

Comment 16 Alan Pevec (Fedora) 2017-02-15 00:34:08 UTC
> Source0:        https://tarballs.openstack.org/%{pypi_name}/%{pypi_name}-%{version}.tar.gz

That's 404, storops is not hosted in OpenStack infra.

Comment 17 Alan Pevec (Fedora) 2017-02-15 00:38:34 UTC
This works (it was also fixed in comment 8):
Source0:        https://pypi.io/packages/source/%(n=%{pypi_name}; echo ${n:0:1})/%{pypi_name}/%{pypi_name}-%{version}.tar.gz

Comment 18 Alan Pevec (Fedora) 2017-02-15 00:39:45 UTC
Please provide updated spec/srpm links formatted like in comment 8

Comment 19 Alan Pevec (Fedora) 2017-02-15 00:42:15 UTC
CBS scratch build: https://cbs.centos.org/koji/taskinfo?taskID=160424

Comment 20 Alan Pevec (Fedora) 2017-02-15 01:28:16 UTC
failed, BR missing: python-pytest-xdist

Built in CBS but it's missing further two more deps:
Error: Package: python2-pytest-xdist-1.15.0-1.el7.noarch
           Requires: python2-execnet
Error: Package: python2-pytest-xdist-1.15.0-1.el7.noarch
           Requires: python2-py

Comment 21 Alan Pevec (Fedora) 2017-02-15 01:51:30 UTC
TODO execnet>=1.1 / Fedora python2-execnet-1.4.1 / EPEL python-execnet-1.2.0
CBS OK pytest>=2.4.2 / Fedora python2-pytest-3.0.6 / CBS RDO pytest-2.8.7 / EL7 base pytest-2.3.5
CBS OK py>=1.4.22 / Fedora python2-py-1.4.32 / CBS RDO python-py-1.4.30 / EL7 base python-py-1.4.14

TODO python2->python in python-pytest-xdist.spec

Comment 22 Alan Pevec (Fedora) 2017-02-15 02:26:13 UTC
TODOs done, scratch now fails during unittests execution: 
http://cbs.centos.org/kojifiles/work/tasks/554/160554/build.log
E   ImportError: No module named monotonic

storops_test/utils.py:25: in <module>
    import fasteners
/usr/lib/python2.7/site-packages/fasteners/__init__.py:23: in <module>
    from fasteners.lock import locked  # noqa
/usr/lib/python2.7/site-packages/fasteners/lock.py:24: in <module>
    from fasteners import _utils
/usr/lib/python2.7/site-packages/fasteners/_utils.py:22: in <module>
    from monotonic import monotonic as now  # noqa
E   ImportError: No module named monotonic

TODO fix missing dep in python-fasteners

Comment 24 Alan Pevec (Fedora) 2017-02-22 06:23:35 UTC
CBS build: http://cbs.centos.org/koji/buildinfo?buildID=15732

Comment 26 Alan Pevec 2017-03-30 00:44:48 UTC
FWIW this cannot be shipped in RDO b/c proprietary Navisphere is required, strops is wrapping proprietary CLI: https://github.com/emc-openstack/storops/blob/7da570abff63b1813b49e34cdc3ed2f03d6680b4/storops/vnx/navi_command.py#L96-L99

The current RPM build will stay available in CBS Koji but EMC will need to keep it updated[*] and ship it in their own package repository, along with all required proprietary packages.


[*] there were 3 new upstream releases since 0.4.7 in February: https://github.com/emc-openstack/storops/tags


Note You need to log in before you can comment on or make changes to this bug.