Description of problem: The storops library (https://github.com/emc-openstack/storops) is now required by the EMC VNX cinder driver to manage VNX arrays, as of Newton. If it is not installed, the VNX driver fails with: 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager [req-cf47020a-cb83-4bc7-8b4a-573fda411de2 - - - - -] Failed to initialize driver. 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager Traceback (most recent call last): 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 414, in init_host 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager self.driver.do_setup(ctxt) 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/vnx/driver.py", line 98, in do_setup 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager self.adapter.do_setup() 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/vnx/adapter.py", line 1214, in do_setup 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager super(ISCSIAdapter, self).do_setup() 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/vnx/adapter.py", line 71, in do_setup 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager self.config.storage_vnx_security_file_dir) 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/vnx/client.py", line 86, in __init__ 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager raise exception.VolumeBackendAPIException(message=msg) 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager VolumeBackendAPIException: storops Python library is not installed. 2017-01-11 18:57:31.413 127144 ERROR cinder.volume.manager The storops library needs to be packaged, and has at least a few more missing subdependencies: - python-bitmath - python-cachez - python-queuelib - python-retryz Version-Release number of selected component (if applicable): Cinder Newton Steps to Reproduce: 1. Configure EMC VNX driver 2. Start cinder volume service
Here is preliminary analysis of the current RPM packaging state of the web-of-deps: > - python-bitmath 1.3.1 in Fedora and EPEL7, can be imported to CBS Koji for RDO deps repo Missing BR: Error: No Package found for python-progressbar > - python-queuelib 1.4.2 in Fedora, needs to be forked in github.com/rdo-common/ to make it build on EL7 (conditionalize py3 at least) > - python-cachez not in Fedora, needs Fedora Package Review > - python-retryz not in Fedora, needs Fedora Package Review
Built in CBS: * progressbar http://cbs.centos.org/koji/buildinfo?buildID=15343 * bitmath http://cbs.centos.org/koji/buildinfo?buildID=15332 * queuelib http://cbs.centos.org/koji/buildinfo?buildID=15345 RDO EL7 fix https://github.com/rdo-common/python-queuelib/commit/d30a88bb953ad536935bfea1dc0db46547aaa6db > > - python-cachez > not in Fedora, needs Fedora Package Review > > > - python-retryz > not in Fedora, needs Fedora Package Review Eric, would like to open Fedora/PackageReview for those two? You can use pyp2rpm for the initial spec or start with example spec in https://fedoraproject.org/wiki/Packaging:Python When you have review bz please add them in Depends On here.
BR for cachez and retryz: * hamcrest http://cbs.centos.org/koji/buildinfo?buildID=15346
Deps should be all set now, please post link to the initial spec/SRPM here.
Created attachment 1248903 [details] python-storops.spec
Comment on attachment 1248903 [details] python-storops.spec Wrong version.
Created attachment 1248905 [details] python-storops.spec
SPEC URL: https://hguemar.fedorapeople.org/reviews/python-storops.spec SRPM URL: https://hguemar.fedorapeople.org/reviews/python-storops-0.4.4-1.el7.src.rpm I fixed download URL since the one in spec file doesn't work, I also started a scratch build in CBS. https://cbs.centos.org/koji/taskinfo?taskID=159597
Package looks sane, but it's not running unit tests. From quick glance: - missing BR to run test: python-pytest-xdist, python-future, python-fasteners, python-hamcrest - potential conflict with requests versions (does not prevent tests running)
(In reply to Haïkel Guémar from comment #9) > Package looks sane, but it's not running unit tests. I don't think that the unit tests will currently function in our package build environment. Running them locally shows lots of failures to open files like "storops-0.4.4/test/vnx/testdata/block_output/security_-certificate_-getLevel.txt" which do not exist. This data is present here, but not in the source on PyPI: https://github.com/emc-openstack/storops/tree/develop/test/vnx/testdata/block_output
Initial package could be w/o unit tests but give we don't have CI coverage for this HW specific library, it should be added so we have at least minimal coverage on the unittesting level. Please open upstream issue to fix the release tarball and reference it as TODO in commented-out %check section. Also update to the latest version https://github.com/emc-openstack/storops/releases/tag/r0.4.6
The 0.4.6 update to the storops library has added a requirement for persist-queue, which is not yet packaged for Fedora. https://pypi.python.org/pypi/persist-queue/
Opened https://github.com/emc-openstack/storops/issues/92 for unit test issues.
Created attachment 1250329 [details] python-storops.spec Updated to 0.4.7 and now runs unit tests.
Built in CBS: * persist-queue https://cbs.centos.org/koji/buildinfo?buildID=15690
> Source0: https://tarballs.openstack.org/%{pypi_name}/%{pypi_name}-%{version}.tar.gz That's 404, storops is not hosted in OpenStack infra.
This works (it was also fixed in comment 8): Source0: https://pypi.io/packages/source/%(n=%{pypi_name}; echo ${n:0:1})/%{pypi_name}/%{pypi_name}-%{version}.tar.gz
Please provide updated spec/srpm links formatted like in comment 8
CBS scratch build: https://cbs.centos.org/koji/taskinfo?taskID=160424
failed, BR missing: python-pytest-xdist Built in CBS but it's missing further two more deps: Error: Package: python2-pytest-xdist-1.15.0-1.el7.noarch Requires: python2-execnet Error: Package: python2-pytest-xdist-1.15.0-1.el7.noarch Requires: python2-py
TODO execnet>=1.1 / Fedora python2-execnet-1.4.1 / EPEL python-execnet-1.2.0 CBS OK pytest>=2.4.2 / Fedora python2-pytest-3.0.6 / CBS RDO pytest-2.8.7 / EL7 base pytest-2.3.5 CBS OK py>=1.4.22 / Fedora python2-py-1.4.32 / CBS RDO python-py-1.4.30 / EL7 base python-py-1.4.14 TODO python2->python in python-pytest-xdist.spec
TODOs done, scratch now fails during unittests execution: http://cbs.centos.org/kojifiles/work/tasks/554/160554/build.log E ImportError: No module named monotonic storops_test/utils.py:25: in <module> import fasteners /usr/lib/python2.7/site-packages/fasteners/__init__.py:23: in <module> from fasteners.lock import locked # noqa /usr/lib/python2.7/site-packages/fasteners/lock.py:24: in <module> from fasteners import _utils /usr/lib/python2.7/site-packages/fasteners/_utils.py:22: in <module> from monotonic import monotonic as now # noqa E ImportError: No module named monotonic TODO fix missing dep in python-fasteners
SPEC URL: https://eharney.fedorapeople.org/python-storops/2/python-storops.spec SRPM URL: https://eharney.fedorapeople.org/python-storops/2/python-storops-0.4.7-2.el7.src.rpm
CBS build: http://cbs.centos.org/koji/buildinfo?buildID=15732
FWIW this cannot be shipped in RDO b/c proprietary Navisphere is required, strops is wrapping proprietary CLI: https://github.com/emc-openstack/storops/blob/7da570abff63b1813b49e34cdc3ed2f03d6680b4/storops/vnx/navi_command.py#L96-L99 The current RPM build will stay available in CBS Koji but EMC will need to keep it updated[*] and ship it in their own package repository, along with all required proprietary packages. [*] there were 3 new upstream releases since 0.4.7 in February: https://github.com/emc-openstack/storops/tags