Bug 1719940

Summary: [RHOS 15] Tempest test failure: test_show_update_rebuild_list_server.ServerShowV263Test.test_show_update_rebuild_list_server
Product: Red Hat OpenStack Reporter: Archit Modi <amodi>
Component: openstack-tempestAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: Martin Kopec <mkopec>
Severity: medium Docs Contact:
Priority: medium    
Version: 15.0 (Stein)CC: apevec, dasmith, egallen, jhakimra, kchamart, lhh, lyarwood, mkopec, mschuppe, sbauza, sgordon, slinaber, udesale, vromanso
Target Milestone: rcKeywords: AutomationBlocker, Triaged
Target Release: 15.0 (Stein)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tempest-20.0.0-0.20190702111910.a955095.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1723796 1738330 (view as bug list) Environment:
Last Closed: 2019-09-21 11:23:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1723796, 1738330, 1740708    
Attachments:
Description Flags
verification output none

Description Archit Modi 2019-06-12 19:07:10 UTC
Description of problem: Deploying RHOS 15 with mentioned topology and running the following tempest test, gives failures:

tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server

Version-Release number of selected component (if applicable):
core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190607.n.1

How reproducible: always

Topology: 1 controller, 2 compute

Steps to Reproduce:
1. Deploy RHOS 15 with mentioned topology
2. Run the following tempest test:
   tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server

Actual results:
Response - Headers: {'date': 'Tue, 11 Jun 2019 04:34:02 GMT', 'server': 'Apache', 'content-length': '1433', 'openstack-api-version': 'compute 2.63', 'x-openstack-nova-api-version': '2.63', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-5d4c6caa-8425-4354-91e3-7f6cd6e44e3b', 'x-compute-request-id': 'req-5d4c6caa-8425-4354-91e3-7f6cd6e44e3b', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e'}
        Body: b'{"server": {"id": "f897b0f8-76e5-48a1-a528-ccbe9a6f639e", "name": "tempest-ServerShowV263Test-server-1768484994", "status": "REBUILD", "tenant_id": "3b0372bbcad940819343f6c51cebbc4d", "user_id": "a7805e2b63774bb1bf0642e5a2da11aa", "metadata": {}, "hostId": "2af7c77f25137c1bc7de69fd8262ac5fffbedcaf3cda3d5a61de3377", "image": {"id": "9acc25ec-3f4f-4069-86d3-6ab7be17fe21", "links": [{"rel": "bookmark", "href": "http://10.0.0.129:8774/images/9acc25ec-3f4f-4069-86d3-6ab7be17fe21"}]}, "flavor": {"vcpus": 1, "ram": 64, "disk": 1, "ephemeral": 0, "swap": 0, "original_name": "m1.nano", "extra_specs": {"hw_rng:allowed": "True"}}, "created": "2019-06-11T04:33:33Z", "updated": "2019-06-11T04:33:40Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}, {"rel": "bookmark", "href": "http://10.0.0.129:8774/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-06-11T04:33:35.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": "rebuilding", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": ["test-cert-1", "test-cert-2"]}}'
2019-06-11 04:34:04,725 253872 INFO     [tempest.lib.common.rest_client] Request (ServerShowV263Test:test_show_update_rebuild_list_server): 200 GET http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e 0.121s
2019-06-11 04:34:04,726 253872 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.63', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 11 Jun 2019 04:34:04 GMT', 'server': 'Apache', 'content-length': '1439', 'openstack-api-version': 'compute 2.63', 'x-openstack-nova-api-version': '2.63', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-a1b94762-8b0e-4ab3-b1a5-57b08bca99df', 'x-compute-request-id': 'req-a1b94762-8b0e-4ab3-b1a5-57b08bca99df', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e'}
        Body: b'{"server": {"id": "f897b0f8-76e5-48a1-a528-ccbe9a6f639e", "name": "tempest-ServerShowV263Test-server-1768484994", "status": "REBUILD", "tenant_id": "3b0372bbcad940819343f6c51cebbc4d", "user_id": "a7805e2b63774bb1bf0642e5a2da11aa", "metadata": {}, "hostId": "2af7c77f25137c1bc7de69fd8262ac5fffbedcaf3cda3d5a61de3377", "image": {"id": "9acc25ec-3f4f-4069-86d3-6ab7be17fe21", "links": [{"rel": "bookmark", "href": "http://10.0.0.129:8774/images/9acc25ec-3f4f-4069-86d3-6ab7be17fe21"}]}, "flavor": {"vcpus": 1, "ram": 64, "disk": 1, "ephemeral": 0, "swap": 0, "original_name": "m1.nano", "extra_specs": {"hw_rng:allowed": "True"}}, "created": "2019-06-11T04:33:33Z", "updated": "2019-06-11T04:34:04Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}, {"rel": "bookmark", "href": "http://10.0.0.129:8774/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-06-11T04:33:35.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": "rebuild_spawning", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": ["test-cert-1", "test-cert-2"]}}'
2019-06-11 04:34:04,731 253872 INFO     [tempest.common.waiters] State transition "REBUILD/rebuilding" ==> "REBUILD/rebuild_spawning" after 25 second wait
2019-06-11 04:34:06,045 253872 INFO     [tempest.lib.common.rest_client] Request (ServerShowV263Test:test_show_update_rebuild_list_server): 200 GET http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e 0.312s
2019-06-11 04:34:06,046 253872 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.63', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 11 Jun 2019 04:34:05 GMT', 'server': 'Apache', 'content-length': '1655', 'openstack-api-version': 'compute 2.63', 'x-openstack-nova-api-version': '2.63', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-f334cdc7-61ad-4438-90ab-92a4a3ae82f5', 'x-compute-request-id': 'req-f334cdc7-61ad-4438-90ab-92a4a3ae82f5', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e'}
        Body: b'{"server": {"id": "f897b0f8-76e5-48a1-a528-ccbe9a6f639e", "name": "tempest-ServerShowV263Test-server-1768484994", "status": "ERROR", "tenant_id": "3b0372bbcad940819343f6c51cebbc4d", "user_id": "a7805e2b63774bb1bf0642e5a2da11aa", "metadata": {}, "hostId": "2af7c77f25137c1bc7de69fd8262ac5fffbedcaf3cda3d5a61de3377", "image": {"id": "9acc25ec-3f4f-4069-86d3-6ab7be17fe21", "links": [{"rel": "bookmark", "href": "http://10.0.0.129:8774/images/9acc25ec-3f4f-4069-86d3-6ab7be17fe21"}]}, "flavor": {"vcpus": 1, "ram": 64, "disk": 1, "ephemeral": 0, "swap": 0, "original_name": "m1.nano", "extra_specs": {"hw_rng:allowed": "True"}}, "created": "2019-06-11T04:33:33Z", "updated": "2019-06-11T04:34:05Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.129:8774/v2.1/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}, {"rel": "bookmark", "href": "http://10.0.0.129:8774/servers/f897b0f8-76e5-48a1-a528-ccbe9a6f639e"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2019-06-11T04:34:05Z", "message": "Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_uuid."}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-06-11T04:33:35.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": ["test-cert-1", "test-cert-2"]}}'
2019-06-11 04:34:06,052 253872 INFO     [tempest.common.waiters] State transition "REBUILD/rebuild_spawning" ==> "ERROR/None" after 27 second wait
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_servers.py", line 236, in test_show_update_rebuild_list_server
    server['id'], 'ACTIVE')
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server f897b0f8-76e5-48a1-a528-ccbe9a6f639e failed to build and is in ERROR status
Details: {'code': 500, 'created': '2019-06-11T04:34:05Z', 'message': 'Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_uuid.'}

Expected results:
Test passed

Additional info:

Comment 3 Matthew Booth 2019-06-13 16:15:09 UTC
This is a dup, but I can't find the link right now. Lee, can you close this for me?

Comment 4 Lee Yarwood 2019-06-13 16:21:46 UTC
(In reply to Matthew Booth from comment #3)
> This is a dup, but I can't find the link right now. Lee, can you close this
> for me?

Only of an upstream CIX for Rocky so I'll keep this open to track the issue.

Comment 12 Martin Kopec 2019-08-13 14:43:45 UTC
Tested on:
- core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190807.n.0
- topology: controller:1,compute:1

The test is skipped until compute.certified_image_ref and compute.certified_image_trusted_certs options are provided.
When I tried to provide those options to test if the test is working it failed with a very similar error as is described in this BZ.

This BZ is an automation blocker because our automated jobs are failing on this, however, their intention is not to run the test. The test is wrong because it doesn't use required resources as it is supposed to (an image with img_signature_* properties) - the changes proposed in the BZ introduced new tempest options which will provide required resources to the test. If the options are not set, the test is skipped.
To conclude, as the proposed changes fix the automation problem, I'm gonna verify it. The changes required for actually running the test are gonna be tracked in a new BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1740708

I'll also provide an output from my testing.
Note: the output shows my test workflow with a newer openstack-tempest package: openstack-tempest-21.0.0-0.20190726140456.702b21c.el8ost.noarch

Comment 13 Martin Kopec 2019-08-13 14:44:23 UTC
Created attachment 1603406 [details]
verification output

Comment 15 errata-xmlrpc 2019-09-21 11:23:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811