Bug 1574931 - [Docs][Tempest] Update the Integration Test Suite Guide for RHOSP 13 - changed plug-in names, tempest-container available in TP
Summary: [Docs][Tempest] Update the Integration Test Suite Guide for RHOSP 13 - change...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Alex McLeod
QA Contact: RHOS Documentation Team
URL:
Whiteboard:
Depends On: 1488115
Blocks: 1488095 1666376
TreeView+ depends on / blocked
 
Reported: 2018-05-04 11:23 UTC by Chandan Kumar
Modified: 2019-10-22 15:22 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-01 09:35:08 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Chandan Kumar 2018-05-04 11:23:04 UTC
We need to update the tempest documentation for RHOS-13

* The names of tempest plugins got changed:
 -> Here is the updated tempest plugins names: https://git.openstack.org/cgit/openstack/tripleo-docs/tree/doc/source/install/basic_deployment/tempest_plugins.rst?h=refs/changes/61/565161/5

* We have added tempest-container in Tech Preview.
Here is the blog post on how to use it: https://blogs.rdoproject.org/2018/05/consuming-kolla-tempest-container-image-for-running-tempest-tests/

Please let us know if more help is needed.

Comment 1 wes hayutin 2018-06-18 18:43:15 UTC
Updated doc still under review but could be ready for 13
https://review.openstack.org/#/c/565161/

Comment 3 David Paterson 2018-11-28 19:49:00 UTC
There are also some plugins mentioned that do not exists, glance for example.  I have some notes below:

There are a few issues with the OSP 13 integration test suite guide: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/openstack_integration_test_suite_guide

3.3.3 has
# yum install python-glance-tests python-keystone-tests python-horizon-tests-tempest python-neutron-tests python-cinder-tests python-nova-tests python-swift-tests python-ceilometer-tests python-gnocchi-tests python-aodh-tests

This command has a few issues. 

The following packages do not exist at all:
sudo yum install python-neutron-tests python-cinder-tests python-ceilometer-tests
Loaded plugins: product-id, search-disabled-repos, subscription-manager
No package python-neutron-tests available.
No package python-cinder-tests available.
No package python-ceilometer-tests available.

The following packages exist but they are not tempest plugin packages but unit and functional test packages:
python-glance-tests
python-nova-tests
python-swift-tests
python-gnocchi-tests
python-aodh-tests
python-keystone-tests 

A tempest plugin must follow the convention that there is a setup.cfg in the root of the repo, and it has an entry point that looks something like the horizon pluigin for example:
[entry_points]
tempest.test_plugins =
   horizon = tempest_horizon.plugin:HorizonTempestPlugin

In the case of keystone there actually IS a tempest plugin and it is package python-keystone-tests-tempest.  So there are two packages, a keystone unit test package (python-keystone-tests), and a tempest plugin (python-keystone-tests-tempest), the OSP 13 documentation is referring to the wrong package, the unit test package (python-keystone-tests).

Basically the only tests that will be discovered when ostestr runs are in the tempest tree or in the plugins discovered.  The upstream tempest plugin registry can be found here:
https://docs.openstack.org/tempest/latest/plugin-registry.html


After adjusting the command in 3.3.3 to:
sudo yum install python-neutron-tests-tempest python-cinder-tests-tempest python-telemetry-tests-tempest python-heat-tests-tempest python-keystone-tests-tempest python-horizon-tests-tempest

The list of tempest plugins now looks better:

$ tempest list-plugins
+-----------------+--------------------------------------------------------+
|       Name      |                       EntryPoint                       |
+-----------------+--------------------------------------------------------+
| telemetry_tests | telemetry_tempest_plugin.plugin:TelemetryTempestPlugin |
|       heat      |      heat_tempest_plugin.plugin:HeatTempestPlugin      |
|  keystone_tests |  keystone_tempest_plugin.plugin:KeystoneTempestPlugin  |
|     horizon     |      tempest_horizon.plugin:HorizonTempestPlugin       |
|  neutron_tests  |   neutron_tempest_plugin.plugin:NeutronTempestPlugin   |
|   cinder_tests  |    cinder_tempest_plugin.plugin:CinderTempestPlugin    |
+-----------------+--------------------------------------------------------+

Before correcting the yum install command listed in the OSP 13 doc the only plugin listed was the horizon one.

This also means the tempest plugin table (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/openstack_integration_test_suite_guide/index#sec-list-of-plugins) in the OSP 13 docs is pretty wrong.  All the package names should end in "-tempest" I believe.  Also the tempest plugin packages table should align with the actual tempest plugins (again see: https://docs.openstack.org/tempest/latest/plugin-registry.html).

For instance, glance, nova, swift, gnocchi or aodh, don't have tempest plugins at all, yet they are referenced in the OSP 13 doc as tempest plugins. I presume there may be others listed in the plugin package table that are also not tempest plugins, but unit test packages, and the table should be reconciled and corrected.

Comment 7 Michael Carden 2019-01-15 03:42:18 UTC
To find the tempest test package names for our RHOSP13 install, I've resorted to:

for service in $(openstack service list  -c Name -f value) \
; do yum search  ${service} 2>/dev/null \
| grep tests ; done | awk '{print $1}'
                                                                                                                                     
python-octavia-tests-golang.x86_64
python-octavia-tests-tempest-golang.x86_64
python2-octavia-tests-tempest.noarch
python-nova-tests.noarch
python2-novajoin-tests-tempest.noarch
python2-barbican-tests-tempest.noarch
python-neutron-fwaas-tests.noarch
python-neutron-lbaas-tests.noarch
python2-neutron-lib-tests.noarch
python2-neutron-tests-tempest.noarch
python2-networking-l2gw-tests-tempest.noarch
python-panko-tests.noarch
python2-cinder-tests-tempest.noarch
python-glance-tests.noarch
python-aodh-tests.noarch
python2-keystone-tests-tempest.noarch
python2-heat-tests-tempest.noarch
python-gnocchi-tests.noarch
python2-ironic-tests-tempest.noarch


Note that some package names begin with python2, some are suffixed by tempest,
and some have neither of those things.

While the naming remains messy like this, perhaps a dynamic means of discovery
like the one I used above, could be documented.

Comment 8 Michael Carden 2019-01-15 03:52:08 UTC
Or more succinctly:

yum search $(openstack service list -c Name -f value) 2>/dev/null | grep test | awk '{print $1}'


:)

Comment 11 Chuck Copello 2019-10-22 15:22:35 UTC
Clearing needinfo flag.


Note You need to log in before you can comment on or make changes to this bug.