@Arie, any ideas? I imagine the recent OSP 9 Neutron rebase is relevant here.

Unable to reproduce:

[abregman@eve71c openstack-tempest-10.0.0]$ rpm -q openstack-tempest
openstack-tempest-10.0.0-1.b4a056dgit.el7ost.noarch

[abregman@eve71c openstack-tempest-10.0.0]$ testr list-tests
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list 
2016-05-17 16:04:03.453 405 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf
tempest.api.baremetal.admin.test_api_discovery.TestApiDiscovery.test_api_versions[id-a3c27e94-f56c-42c4-8600-d6790650b9c5]
tempest.api.baremetal.admin.test_api_discovery.TestApiDiscovery.test_default_version[id-896283a6-488e-4f31-af78-6614286cbe0d]
tempest.api.baremetal.admin.test_api_discovery.TestApiDiscovery.test_version_1_resources[id-abc0b34d-e684-4546-9728-ab7a9ad9f174]
tempest.api.baremetal.admin.test_chassis.TestChassis.test_chassis_node_list[id-76305e22-a4e2-4ab3-855c-f4e2368b9335]
tempest.api.baremetal.admin.test_chassis.TestChassis.test_create_chassis[id-7c5a2e09-699c-44be-89ed-2bc189992d42]

Which version of python-testrepository are you using?

@Arie, below is the version of python-testrepository

python-testrepository.noarch          0.0.20-3.1.el7ost      @rhelosp-9.0-puddle

I'm using the same version.

Is it possible to provide access to the host for further investigation?

Hi Arie,

Will it be possible to send us your public key so that I can give you the access to the system.
Also you can find me on #rhos-dev and #rhos-qe as anshul. If not there I will be reachable over email.

Thanks,
Anshul Behl

Found the issue.

Installing either 'python-aodh' or 'python-ceilometer' will cause the issue by using wrong import path. Removing them causing testr to list neutron tests properly.

This is because ceilometer tempest tests are shipped in python-ceilometer-tests packages. But the python entry_points are shipped in python-ceilometer.


So to fix we can:
* Make python-ceilometer depends on python-ceilometer-tests (This looks weird because the package have been created to not ship tests if not needed)
* Fix tempest to not fail when an entry-point is unloadable.
* Make openstack-tempest depends on python-ceilometer-tests

Or:
* moves back tests into python-ceilometer package

Since this is not a ceilometer only issue but a issue for all tempest plugins shipped in python-XXXX-packages. Only the seconds solution looks reasonable.

s/python-XXXX-packages/python-XXXX-tests

Also installation python-ceilometer-tests and python-aodh-tests fixes the issue.

I have no knowledge on tempest, but something like this could fix the issue: https://review.openstack.org/318151.

Or a downstream approach could be to just have tempest depending on all python-XXXX-tests that contains tempest plugins.

Will close it for now as it's not a bug but we'll have a meeting about the possible dependencies upstream.

@Mehdi: I completely agree on comment 11 that this issue is for all tempest plugins however, installing python-ceilometer-tests and python-aodh-tests (mentioned in comment 13) wouldn't fix the issue.

This is because, it will then raise error for neutron tempest tests (refer to comment 8 by Arie).

@Daniel: Can you explain why this is not a bug as installing few packages to run individual component tempest tests raises error for tempest tests of other component

Upstream fix is in puppet-tempest which knows about -tests subpackages and installs them.
Since we don't use that downstream and have non-upstream config_tempest script, d/s solution could be to install corresponding -tests subpackages in that script.

And final proper upstream solution is to make all tempest plugins stand-alone projects as suggested by http://docs.openstack.org/developer/tempest/plugin.html#standalone-plugin-vs-in-repo-plugin

Hi Alan,

I was aware of the upstream solutions as well as for puppet-tempest. So in the downstream approach, I've some doubts about packaging. 1) How are the entry points assigned in the rpms that are split? (i.e. sahara + sahara tests)

As for the plugin split, it also depends on the project so you can have some of projects following that approach (designate) + some others that are not (neutron)

The workaround did worked. I was able to list the tests after installing python-ceilometer-tests and python-aodh-tests packages.

> I've some doubts about packaging. 1) How are the
> entry points assigned in the rpms that are split? (i.e. sahara + sahara
> tests)

sahara-tests is a separate project so tempest plugin entry point is in https://github.com/openstack/sahara-tests/blob/master/setup.cfg#L34 

Corresponding standalone RPM is under review in bug 1318765

> As for the plugin split, it also depends on the project so you can have some
> of projects following that approach (designate) + some others that are not
> (neutron)

Yeah, and now I think that really it's the best approach to have separate plugin project, as suggested by tempest plugin docs:
* avoids the issue with entry point so it can be cleanly separately packaged
* can have different "branchless" schedule, same as Tempest

Sahara has some tests also in the sahara package (for now), so RHOP8 packages ships a python-sahara-tests packages, similar to aodh and ceilometer packaging.  

The question from Daniel is about how the entry points are split, and the answer is that entry points can't be split from the other definition in setup.cfg, so if the tests are part of the same repository, the only way is to make sure that the -tests package is installed, as other pointed out in the discussion.

While moving plugins in separate repository is a wise thing, we can't and I suspect we won't be able to rely on all projects using this schema. For sure not for Mitaka, most likely not for Newton either, so the backup solution (ensure that the -tests packages are installed) should be implemented and kept working.

I am observing below issue in latest RHOSP-8 puddles , 

ImportError: No module named tests.tempest.scenario.data_processing.plugin

[root@dhcp201-218 openstack-tempest-liberty]# testr list-tests
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list 
No handlers could be found for logger "tempest.test_discover.plugins"
--- import errors ---
Failed to call load_tests:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/unittest2/loader.py", line 136, in loadTestsFromModule
    return load_tests(self, tests, pattern)
  File "tempest/test_discover/test_discover.py", line 27, in load_tests
    ext_plugins = plugins.TempestTestPluginManager()
  File "tempest/lib/common/utils/misc.py", line 30, in getinstance
    instances[cls] = cls()
  File "tempest/test_discover/plugins.py", line 77, in __init__
    on_load_failure_callback=self.failure_hook)
  File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 92, in __init__
    verify_requirements)
  File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 170, in _load_plugins
    self._on_load_failure_callback(self, ep, err)
  File "tempest/test_discover/plugins.py", line 82, in failure_hook
    raise err
ImportError: No module named tests.tempest.scenario.data_processing.plugin
Non-zero exit code (2) from test listing.


[root@dhcp201-218 openstack-tempest-liberty]# rpm -q openstack-tempest
openstack-tempest-liberty-20160413.1.el7ost.noarch


[root@dhcp201-218 openstack-tempest-liberty]# rpm -qa | grep python-testrepository
python-testrepository-0.0.20-3.1.el7ost.noarch


Work Around suggested above was to install python-sahara-tests packages
which fixes this issue.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html