Description of problem: DCI tempest execution failed while creating tempest environment. While listing subnetpool , tempest setup fails. Subnetpool extension is not a supported neutron plugin in contrail Version-Release number of selected component (if applicable): How reproducible: Run the DCI tempest setup creation Steps to Reproduce: 1. 2. 3. Actual results: 019-05-29 17:46:30 | + /usr/bin/tempest cleanup --init-saved-state 2019-05-29 17:46:38 | Traceback (most recent call last): 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 98, in take_action 2019-05-29 17:46:38 | self.init(parsed_args) 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 126, in init 2019-05-29 17:46:38 | self._init_state() 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 309, in _init_state 2019-05-29 17:46:38 | svc.run() 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 132, in run 2019-05-29 17:46:38 | self.save_state() 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 705, in save_state 2019-05-29 17:46:38 | pools = self.list() 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 680, in list 2019-05-29 17:46:38 | pools = client.list_subnetpools(**self.tenant_filter)['subnetpools'] 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnetpools_client.py", line 28, in list_subnetpools 2019-05-29 17:46:38 | return self.list_resources(uri, **filters) 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 34, in list_resources 2019-05-29 17:46:38 | resp, body = self.get(req_uri) 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 294, in get 2019-05-29 17:46:38 | return self.request('GET', url, extra_headers, headers) 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request 2019-05-29 17:46:38 | self._error_checker(resp, resp_body) 2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 842, in _error_checker 2019-05-29 17:46:38 | message=message) 2019-05-29 17:46:38 | NotImplemented: Got NotImplemented error 2019-05-29 17:46:38 | Details: 2019-05-29 17:46:38 | Initializing saved state. ³!@)stdout'No tests in repository' Expected results: Additional info: subnetpool not a supported extension
Created attachment 1576729 [details] tempest_output.log
with the latest run DCI failing way before tempest environment creation. It failed while pulling container image: Error ==== docker.errors.NotFound: 404 Client Error: Not Found ("{"message":"No such image: registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1 === non-zero return code [{'_ansible_no_log': False, 'attempts': 3, 'changed': False, 'msg': u'non-zero return code', 'rc': 1, 'stderr': u'Traceback (most recent call last):\n File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 141, in <module>\n main()\n File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 134, in main\n sync_image(image)\n File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 86, in sync_image\n tag_image(image, tag)\n File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 44, in tag_image\n docker_image_id = client.inspect_image(\'{origin_registry}/{project}/{name}:{tag}\'.format(**image))[\'Id\']\n File "/usr/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped\n return f(self, resource_id, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/docker/api/image.py", line 141, in inspect_image\n True\n File "/usr/lib/python2.7/site-packages/docker/client.py", line 178, in _result\n self._raise_for_status(response)\n File "/usr/lib/python2.7/site-packages/docker/client.py", line 173, in _raise_for_status\n raise errors.NotFound(e, response, explanation=explanation)\ndocker.errors.NotFound: 404 Client Error: Not Found ("{"message":"No such image: registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1"}")\n', 'stderr_lines': [u'Traceback (most recent call last):', u' File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 141, in <module>', u' main()', u' File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 134, in main', u' sync_image(image)', u' File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 86, in sync_image', u' tag_image(image, tag)', u' File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 44, in tag_image', u" docker_image_id = client.inspect_image('{origin_registry}/{project}/{name}:{tag}'.format(**image))['Id']", u' File "/usr/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped', u' return f(self, resource_id, *args, **kwargs)', u' File "/usr/lib/python2.7/site-packages/docker/api/image.py", line 141, in inspect_image', u' True', u' File "/usr/lib/python2.7/site-packages/docker/client.py", line 178, in _result', u' self._raise_for_status(response)', u' File "/usr/lib/python2.7/site-packages/docker/client.py", line 173, in _raise_for_status', u' raise errors.NotFound(e, response, explanation=explanation)', u'docker.errors.NotFound: 404 Client Error: Not Found ("{"message":"No such image: registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1"}")'], 'stdout': u'login onto registry.distributed-ci.io\nImage openstack-aodh-api:2019-05-31.1 is already on the registry\nImage openstack-aodh-base:2019-05-31.1 is already on the registry\nImage openstack-aodh-evaluator:2019-05-31.1 is already on the registry\nImage openstack-aodh-listener:2019-05-31.1 is already on the registry\nImage openstack-aodh-notifier:2019-05-31.1 is already on the registry\nImage openstack-barbican-api:2019-05-31.1 is already on the registry\nImage openstack-barbican-base:2019-05-31.1 is already on the registry\nImage openstack-barbican-keystone-listener:2019-05-31.1 is already on the registry\nImage openstack-barbican-worker:2019-05-31.1 is already on the registry\nImage openstack-base:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-base:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-central:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-compute:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-ipmi:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-notification:2019-05-31.1 is already on the registry\nImage openstack-cinder-api:2019-05-31.1 is already on the registry\nImage openstack-cinder-backup:2019-05-31.1 is already on the registry\nImage openstack-cinder-base:2019-05-31.1 is already on the registry\nImage openstack-cinder-scheduler:2019-05-31.1 is already on the registry\nImage openstack-cinder-volume:2019-05-31.1 is already on the registry\nImage openstack-collectd:2019-05-31.1 is already on the registry\nImage openstack-cron:2019-05-31.1 is already on the registry\nImage openstack-dependencies:2019-05-31.1 is already on the registry\nImage openstack-ec2-api:2019-05-31.1 is already on the registry\nImage openstack-etcd:2019-05-31.1 is already on the registry\nImage openstack-fluentd:2019-05-31.1 is already on the registry\nImage openstack-glance-api:2019-05-31.1 is already on the registry\nImage openstack-glance-base:2019-05-31.1 is already on the registry\nImage openstack-gnocchi-api:2019-05-31.1 is already on the registry\nImage openstack-gnocchi-base:2019-05-31.1 is already on the registry\nImage openstack-gnocchi-metricd:2019-05-31.1 is already on the registry\nImage openstack-gnocchi-statsd:2019-05-31.1 is already on the registry\nImage openstack-haproxy:2019-05-31.1 is already on the registry\nImage openstack-heat-all:2019-05-31.1 is already on the registry\nImage openstack-heat-api:2019-05-31.1 is already on the registry\nImage openstack-heat-api-cfn:2019-05-31.1 is already on the registry\nImage openstack-heat-base:2019-05-31.1 is already on the registry\nImage openstack-heat-engine:2019-05-31.1 is already on the registry\nImage openstack-horizon:2019-05-31.1 is already on the registry\nImage openstack-ironic-api:2019-05-31.1 is already on the registry\nImage openstack-ironic-base:2019-05-31.1 is already on the registry\nImage openstack-ironic-conductor:2019-05-31.1 is already on the registry\nImage openstack-ironic-inspector:2019-05-31.1 is already on the registry\nImage openstack-ironic-pxe:2019-05-31.1 is already on the registry\nImage openstack-iscsid:2019-05-31.1 is already on the registry\nImage openstack-keystone:2019-05-31.1 is already on the registry\nImage openstack-keystone-base:2019-05-31.1 is already on the registry\nImage openstack-kuryr-cni:2019-05-31.1 is already on the registry\nImage openstack-kuryr-controller:2019-05-31.1 is already on the registry\nImage openstack-manila-api:2019-05-31.1 is already on the registry\nImage openstack-manila-base:2019-05-31.1 is already on the registry\nImage openstack-manila-scheduler:2019-05-31.1 is already on the registry\nImage openstack-manila-share:2019-05-31.1 is already on the registry\nImage openstack-mariadb:2019-05-31.1 is already on the registry\nImage openstack-memcached:2019-05-31.1 is already on the registry\nImage openstack-mistral-api:2019-05-31.1 is already on the registry\nImage openstack-mistral-base:2019-05-31.1 is already on the registry\nImage openstack-mistral-engine:2019-05-31.1 is already on the registry\nImage openstack-mistral-event-engine:2019-05-31.1 is already on the registry\nImage openstack-mistral-executor:2019-05-31.1 is already on the registry\nImage openstack-multipathd:2019-05-31.1 is already on the registry\nImage openstack-neutron-base:2019-05-31.1 is already on the registry\ndocker pull registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1\n', 'stdout_lines': [u'login onto registry.distributed-ci.io', u'Image openstack-aodh-api:2019-05-31.1 is already on the registry', u'Image openstack-aodh-base:2019-05-31.1 is already on the registry', u'Image openstack-aodh-evaluator:2019-05-31.1 is already on the registry', u'Image openstack-aodh-listener:2019-05-31.1 is already on the registry', u'Image openstack-aodh-notifier:2019-05-31.1 is already on the registry', u'Image openstack-barbican-api:2019-05-31.1 is already on the registry', u'Image openstack-barbican-base:2019-05-31.1 is already on the registry', u'Image openstack-barbican-keystone-listener:2019-05-31.1 is already on the registry', u'Image openstack-barbican-worker:2019-05-31.1 is already on the registry', u'Image openstack-base:2019-05-31.1 is already on the registry', u'Image openstack-ceilometer-base:2019-05-31.1 is already on the registry', u'Image openstack-ceilometer-central:2019-05-31.1 is already on the registry', u'Image openstack-ceilometer-compute:2019-05-31.1 is already on the registry', u'Image openstack-ceilometer-ipmi:2019-05-31.1 is already on the registry', u'Image openstack-ceilometer-notification:2019-05-31.1 is already on the registry', u'Image openstack-cinder-api:2019-05-31.1 is already on the registry', u'Image openstack-cinder-backup:2019-05-31.1 is already on the registry', u'Image openstack-cinder-base:2019-05-31.1 is already on the registry', u'Image openstack-cinder-scheduler:2019-05-31.1 is already on the registry', u'Image openstack-cinder-volume:2019-05-31.1 is already on the registry', u'Image openstack-collectd:2019-05-31.1 is already on the registry', u'Image openstack-cron:2019-05-31.1 is already on the registry', u'Image openstack-dependencies:2019-05-31.1 is already on the registry', u'Image openstack-ec2-api:2019-05-31.1 is already on the registry', u'Image openstack-etcd:2019-05-31.1 is already on the registry', u'Image openstack-fluentd:2019-05-31.1 is already on the registry', u'Image openstack-glance-api:2019-05-31.1 is already on the registry', u'Image openstack-glance-base:2019-05-31.1 is already on the registry', u'Image openstack-gnocchi-api:2019-05-31.1 is already on the registry', u'Image openstack-gnocchi-base:2019-05-31.1 is already on the registry', u'Image openstack-gnocchi-metricd:2019-05-31.1 is already on the registry', u'Image openstack-gnocchi-statsd:2019-05-31.1 is already on the registry', u'Image openstack-haproxy:2019-05-31.1 is already on the registry', u'Image openstack-heat-all:2019-05-31.1 is already on the registry', u'Image openstack-heat-api:2019-05-31.1 is already on the registry', u'Image openstack-heat-api-cfn:2019-05-31.1 is already on the registry', u'Image openstack-heat-base:2019-05-31.1 is already on the registry', u'Image openstack-heat-engine:2019-05-31.1 is already on the registry', u'Image openstack-horizon:2019-05-31.1 is already on the registry', u'Image openstack-ironic-api:2019-05-31.1 is already on the registry', u'Image openstack-ironic-base:2019-05-31.1 is already on the registry', u'Image openstack-ironic-conductor:2019-05-31.1 is already on the registry', u'Image openstack-ironic-inspector:2019-05-31.1 is already on the registry', u'Image openstack-ironic-pxe:2019-05-31.1 is already on the registry', u'Image openstack-iscsid:2019-05-31.1 is already on the registry', u'Image openstack-keystone:2019-05-31.1 is already on the registry', u'Image openstack-keystone-base:2019-05-31.1 is already on the registry', u'Image openstack-kuryr-cni:2019-05-31.1 is already on the registry', u'Image openstack-kuryr-controller:2019-05-31.1 is already on the registry', u'Image openstack-manila-api:2019-05-31.1 is already on the registry', u'Image openstack-manila-base:2019-05-31.1 is already on the registry', u'Image openstack-manila-scheduler:2019-05-31.1 is already on the registry', u'Image openstack-manila-share:2019-05-31.1 is already on the registry', u'Image openstack-mariadb:2019-05-31.1 is already on the registry', u'Image openstack-memcached:2019-05-31.1 is already on the registry', u'Image openstack-mistral-api:2019-05-31.1 is already on the registry', u'Image openstack-mistral-base:2019-05-31.1 is already on the registry', u'Image openstack-mistral-engine:2019-05-31.1 is already on the registry', u'Image openstack-mistral-event-engine:2019-05-31.1 is already on the registry', u'Image openstack-mistral-executor:2019-05-31.1 is already on the registry', u'Image openstack-multipathd:2019-05-31.1 is already on the registry', u'Image openstack-neutron-base:2019-05-31.1 is already on the registry', u'docker pull registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1']}]
As explained through direct email, the issue with the missing container image is fixed, can you restart the agent again ?
Thanks for the help. Latest DCI didn't hit the docker image issue. But we still have the tempest environment issue Subnetpool. Could you please fix this too
Noticed below error in tempest_output.log ====== 2019-06-05 13:34:48 | 2019-06-05 13:34:48.521 31836 DEBUG config_tempest.constants [-] Setting [volume] max_microversion = 3.50 set /usr/lib/python2.7/site-packages/config_tempest/tempest_conf.py:103 2019-06-05 13:34:48 | 2019-06-05 13:34:48.834 31836 INFO config_tempest.constants [-] Service metering has no endpoints 2019-06-05 13:34:48 | 2019-06-05 13:34:48.835 31836 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /info: BadStatusLine: '' 2019-06-05 13:34:48 | 2019-06-05 13:34:48.837 31836 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /info: BadStatusLine: '' 2019-06-05 13:34:48 | 2019-06-05 13:34:48.838 31836 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /info: BadStatusLine: '' 2019-06-05 13:34:48 | Traceback (most recent call last): 2019-06-05 13:34:48 | File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit 2019-06-05 13:34:48 | msg = self.format(record) 2019-06-05 13:34:48 | File "/usr/lib/python2.7/site-packages/oslo_log/handlers.py", line 161, in format 2019-06-05 13:34:48 | return logging.StreamHandler.format(self, record) 2019-06-05 13:34:48 | File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format 2019-06-05 13:34:48 | return fmt.format(record) 2019-06-05 13:34:48 | File "/usr/lib/python2.7/site-packages/oslo_log/formatters.py", line 501, in format 2019-06-05 13:34:48 | return logging.Formatter.format(self, record) 2019-06-05 13:34:48 | File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format 2019-06-05 13:34:48 | record.message = record.getMessage() 2019-06-05 13:34:48 | File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage 2019-06-05 13:34:48 | msg = msg % self.args 2019-06-05 13:34:48 | TypeError: not all arguments converted during string formatting 2019-06-05 13:34:48 | Logged from file base.py, line 60 2019-06-05 13:34:49 | 2019-06-05 13:34:49.113 31836 DEBUG config_tempest.constants [-] Option '[compute-feature-enabled] console_output = true' was defined by user, NOT overwriting into value 'True' set /usr/lib/python2.7/site-packages/config_tempest/tempest_conf.py:
Created attachment 1577800 [details] tempest.log file from the latest run
Created attachment 1577801 [details] tempest_output.log file from the latest run
I added the latest logfiles to this BZ. Other logfiles for the overcloud servers/services are available from this page: https://www.distributed-ci.io/jobs/588fd57f-3d59-4e82-a291-e89122baf15a/files
The latest tempest logs have been added to the BZ, other logfiles for the overcloud servers/services are available from the following page: https://www.distributed-ci.io/jobs/588fd57f-3d59-4e82-a291-e89122baf15a/files
Looking at tempest history, this issue has been introduced on Feb 19th 2019 in https://github.com/openstack/tempest/commit/e681998023d9714da346d117e18ab673b5657f71 This is a blocking issue for Juniper as they can't run tempest at all anymore because this is not implemented in their Neutron plugin.
Is a Juniper Engineering going to send a patch to the Neutron Tempest plugin to make the subnetpool cleanup conditional on the subnetpool config option in tempest.conf? Or do they expect RH to send the patch? (if so why?)
How can we get this into a build for dci?
*** Bug 1726936 has been marked as a duplicate of this bug. ***
The issue is fixed only partially. tempest cleanup doesn't fail on NotImplemented error anymore and continues cleaning resources as expected, however, the return code of 'tempest cleanup --init-saved-state' is not appropriate and causes issues from automation perspective. This return code issue is tracked separately here https://bugzilla.redhat.com/show_bug.cgi?id=1737754
Created attachment 1606582 [details] test_fix.log
Tested with: - package with the bug (openstack-tempest-18.0.0-8.el7ost) - package with the fix (openstack-tempest-18.0.0-10.el7ost) The bug was not reproduced when tested with the package with the fix as shown in the attached log file (test_fix.log).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2623