Bug 1719940 - [RHOS 15] Tempest test failure: test_show_update_rebuild_list_server.ServerShowV263Test.test_show_update_rebuild_list_server
Summary: [RHOS 15] Tempest test failure: test_show_update_rebuild_list_server.ServerSh...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tempest
Version: 15.0 (Stein)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 15.0 (Stein)
Assignee: Lee Yarwood
QA Contact: Martin Kopec
URL:
Whiteboard:
Depends On:
Blocks: 1723796 1738330 1740708
TreeView+ depends on / blocked
 
Reported: 2019-06-12 19:07 UTC by Archit Modi
Modified: 2019-09-26 10:52 UTC (History)
14 users (show)

Fixed In Version: openstack-tempest-20.0.0-0.20190702111910.a955095.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1723796 1738330 (view as bug list)
Environment:
Last Closed: 2019-09-21 11:23:06 UTC
Target Upstream Version:


Attachments (Terms of Use)
verification output (246.55 KB, text/plain)
2019-08-13 14:44 UTC, Martin Kopec
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 663596 0 None None None 2019-06-17 08:33:37 UTC
OpenStack gerrit 667893 0 None None None 2019-06-27 11:28:30 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:23:31 UTC

Description Archit Modi 2019-06-12 19:07:10 UTC
Description of problem: Deploying RHOS 15 with mentioned topology and running the following tempest test, gives failures:

tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server

Version-Release number of selected component (if applicable):
core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190607.n.1

How reproducible: always

Topology: 1 controller, 2 compute

Steps to Reproduce:
1. Deploy RHOS 15 with mentioned topology
2. Run the following tempest test:
   tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server

Actual results:
Response - Headers: {'date': 'Tue, 11 Jun 2019 04:34:02 GMT', 'server': 'Apache', 'content-length': '1433', 'openstack-api-version': 'compute 2.63', 'x-openstack-nova-api-version': '2.63', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-5d4c6caa-8425-4354-91e3-7f6cd6e44e3b', 'x-compute-request-id': 'req-5d4c6caa-8425-4354-91e3-7f6cd6e44e3b', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e'}
        Body: b'{"server": {"id": "f897b0f8-76e5-48a1-a528-ccbe9a6f639e", "name": "tempest-ServerShowV263Test-server-1768484994", "status": "REBUILD", "tenant_id": "3b0372bbcad940819343f6c51cebbc4d", "user_id": "a7805e2b63774bb1bf0642e5a2da11aa", "metadata": {}, "hostId": "2af7c77f25137c1bc7de69fd8262ac5fffbedcaf3cda3d5a61de3377", "image": {"id": "9acc25ec-3f4f-4069-86d3-6ab7be17fe21", "links": [{"rel": "bookmark", "href": "http://10.0.0.129:8774/images/9acc25ec-3f4f-4069-86d3-6ab7be17fe21"}]}, "flavor": {"vcpus": 1, "ram": 64, "disk": 1, "ephemeral": 0, "swap": 0, "original_name": "m1.nano", "extra_specs": {"hw_rng:allowed": "True"}}, "created": "2019-06-11T04:33:33Z", "updated": "2019-06-11T04:33:40Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}, {"rel": "bookmark", "href": "http://10.0.0.129:8774/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-06-11T04:33:35.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": "rebuilding", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": ["test-cert-1", "test-cert-2"]}}'
2019-06-11 04:34:04,725 253872 INFO     [tempest.lib.common.rest_client] Request (ServerShowV263Test:test_show_update_rebuild_list_server): 200 GET http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e 0.121s
2019-06-11 04:34:04,726 253872 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.63', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 11 Jun 2019 04:34:04 GMT', 'server': 'Apache', 'content-length': '1439', 'openstack-api-version': 'compute 2.63', 'x-openstack-nova-api-version': '2.63', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-a1b94762-8b0e-4ab3-b1a5-57b08bca99df', 'x-compute-request-id': 'req-a1b94762-8b0e-4ab3-b1a5-57b08bca99df', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e'}
        Body: b'{"server": {"id": "f897b0f8-76e5-48a1-a528-ccbe9a6f639e", "name": "tempest-ServerShowV263Test-server-1768484994", "status": "REBUILD", "tenant_id": "3b0372bbcad940819343f6c51cebbc4d", "user_id": "a7805e2b63774bb1bf0642e5a2da11aa", "metadata": {}, "hostId": "2af7c77f25137c1bc7de69fd8262ac5fffbedcaf3cda3d5a61de3377", "image": {"id": "9acc25ec-3f4f-4069-86d3-6ab7be17fe21", "links": [{"rel": "bookmark", "href": "http://10.0.0.129:8774/images/9acc25ec-3f4f-4069-86d3-6ab7be17fe21"}]}, "flavor": {"vcpus": 1, "ram": 64, "disk": 1, "ephemeral": 0, "swap": 0, "original_name": "m1.nano", "extra_specs": {"hw_rng:allowed": "True"}}, "created": "2019-06-11T04:33:33Z", "updated": "2019-06-11T04:34:04Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}, {"rel": "bookmark", "href": "http://10.0.0.129:8774/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-06-11T04:33:35.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": "rebuild_spawning", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": ["test-cert-1", "test-cert-2"]}}'
2019-06-11 04:34:04,731 253872 INFO     [tempest.common.waiters] State transition "REBUILD/rebuilding" ==> "REBUILD/rebuild_spawning" after 25 second wait
2019-06-11 04:34:06,045 253872 INFO     [tempest.lib.common.rest_client] Request (ServerShowV263Test:test_show_update_rebuild_list_server): 200 GET http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e 0.312s
2019-06-11 04:34:06,046 253872 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.63', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 11 Jun 2019 04:34:05 GMT', 'server': 'Apache', 'content-length': '1655', 'openstack-api-version': 'compute 2.63', 'x-openstack-nova-api-version': '2.63', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-f334cdc7-61ad-4438-90ab-92a4a3ae82f5', 'x-compute-request-id': 'req-f334cdc7-61ad-4438-90ab-92a4a3ae82f5', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e'}
        Body: b'{"server": {"id": "f897b0f8-76e5-48a1-a528-ccbe9a6f639e", "name": "tempest-ServerShowV263Test-server-1768484994", "status": "ERROR", "tenant_id": "3b0372bbcad940819343f6c51cebbc4d", "user_id": "a7805e2b63774bb1bf0642e5a2da11aa", "metadata": {}, "hostId": "2af7c77f25137c1bc7de69fd8262ac5fffbedcaf3cda3d5a61de3377", "image": {"id": "9acc25ec-3f4f-4069-86d3-6ab7be17fe21", "links": [{"rel": "bookmark", "href": "http://10.0.0.129:8774/images/9acc25ec-3f4f-4069-86d3-6ab7be17fe21"}]}, "flavor": {"vcpus": 1, "ram": 64, "disk": 1, "ephemeral": 0, "swap": 0, "original_name": "m1.nano", "extra_specs": {"hw_rng:allowed": "True"}}, "created": "2019-06-11T04:33:33Z", "updated": "2019-06-11T04:34:05Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}, {"rel": "bookmark", "href": "http://10.0.0.129:8774/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2019-06-11T04:34:05Z", "message": "Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_uuid."}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-06-11T04:33:35.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": ["test-cert-1", "test-cert-2"]}}'
2019-06-11 04:34:06,052 253872 INFO     [tempest.common.waiters] State transition "REBUILD/rebuild_spawning" ==> "ERROR/None" after 27 second wait
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_servers.py", line 236, in test_show_update_rebuild_list_server
    server['id'], 'ACTIVE')
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server f897b0f8-76e5-48a1-a528-ccbe9a6f639e failed to build and is in ERROR status
Details: {'code': 500, 'created': '2019-06-11T04:34:05Z', 'message': 'Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_uuid.'}

Expected results:
Test passed

Additional info:

Comment 3 Matthew Booth 2019-06-13 16:15:09 UTC
This is a dup, but I can't find the link right now. Lee, can you close this for me?

Comment 4 Lee Yarwood 2019-06-13 16:21:46 UTC
(In reply to Matthew Booth from comment #3)
> This is a dup, but I can't find the link right now. Lee, can you close this
> for me?

Only of an upstream CIX for Rocky so I'll keep this open to track the issue.

Comment 12 Martin Kopec 2019-08-13 14:43:45 UTC
Tested on:
- core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190807.n.0
- topology: controller:1,compute:1

The test is skipped until compute.certified_image_ref and compute.certified_image_trusted_certs options are provided.
When I tried to provide those options to test if the test is working it failed with a very similar error as is described in this BZ.

This BZ is an automation blocker because our automated jobs are failing on this, however, their intention is not to run the test. The test is wrong because it doesn't use required resources as it is supposed to (an image with img_signature_* properties) - the changes proposed in the BZ introduced new tempest options which will provide required resources to the test. If the options are not set, the test is skipped.
To conclude, as the proposed changes fix the automation problem, I'm gonna verify it. The changes required for actually running the test are gonna be tracked in a new BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1740708

I'll also provide an output from my testing.
Note: the output shows my test workflow with a newer openstack-tempest package: openstack-tempest-21.0.0-0.20190726140456.702b21c.el8ost.noarch

Comment 13 Martin Kopec 2019-08-13 14:44:23 UTC
Created attachment 1603406 [details]
verification output

Comment 15 errata-xmlrpc 2019-09-21 11:23:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.