Bug 1694629 - [Regression] Server xxx failed to build "REVERT_RESIZE" test after upgrading from RHOS13 to RHOS14
Summary: [Regression] Server xxx failed to build "REVERT_RESIZE" test after upgrading ...
Keywords:
Status: CLOSED DUPLICATE of bug 1678681
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: OSP DFG:Compute
QA Contact: OSP DFG:Compute
URL:
Whiteboard: DFG:upgrades
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-01 09:52 UTC by Ronnie Rasouli
Modified: 2023-03-21 19:14 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-04 20:51:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Ronnie Rasouli 2019-04-01 09:52:34 UTC
Description of problem:

Tempest test failed:
 tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert

[tempest.api.compute.base] Failed to delete server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878

Failed to delete server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878
Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to build and is in ERROR status

Version-Release number of selected component (if applicable):
2019-03-18.1

How reproducible:
100%

Steps to Reproduce:
1. Deploy RHOS13
2. Upgrade undercloud and overcloud to RHOS14
3. Run tempest

Actual results:
failure on tempest tests

Expected results:
pass like it used to pass on previous builds

Additional info:

esponse - Headers: {'status': '200', u'content-length': '1729', 'content-location': 'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/v2.1/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878', u'x-compute-request-id': 'req-e5676815-2e24-43f9-8ae4-c14f5ae0b2ea', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Mon, 25 Mar 2019 16:41:49 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-e5676815-2e24-43f9-8ae4-c14f5ae0b2ea'}
        Body: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"tempest-ServerActionsTestJSON-187989746-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:4f:3a:ee", "version": 4, "addr": "10.100.0.10", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/v2.1/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878", "rel": "self"}, {"href": "https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878", "rel": "bookmark"}], "image": {"id": "769267a6-8db7-4b48-ad8f-400d8df155af", "links": [{"href": "https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/images/769267a6-8db7-4b48-ad8f-400d8df155af", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "error", "OS-SRV-USG:launched_at": "2019-03-25T16:36:36.000000", "flavor": {"id": "b2250279-2250-4c42-88a4-60bd95e4d764", "links": [{"href": "https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/flavors/b2250279-2250-4c42-88a4-60bd95e4d764", "rel": "bookmark"}]}, "id": "bfd1b2bd-2662-4aa4-97a0-e1a9adffb878", "security_groups": [{"name": "default"}], "user_id": "80d098d039b24ee7ab153cd7a6854972", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ERROR", "updated": "2019-03-25T16:41:48Z", "hostId": "7aeb5d4ca13e58c6b5cd23d4e39a3b2a7ad53de98cbb3d1fc3bd73e6", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest-ServerActionsTestJSON-server-408964725", "created": "2019-03-25T16:35:58Z", "tenant_id": "bb640ac03b494b8fa9f5cad2d72c8ea1", "os-extended-volumes:volumes_attached": [], "fault": {"message": "Virtual Interface creation failed", "code": 500, "created": "2019-03-25T16:41:48Z"}, "metadata": {}}}
2019-03-25 12:41:50,440 795439 INFO     [tempest.common.waiters] State transition "REVERT_RESIZE/resize_reverting" ==> "ERROR/None" after 10 second wait
2019-03-25 12:41:50,440 795439 ERROR    [tempest.api.compute.base] Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to build and is in ERROR status
Details: {u'message': u'Virtual Interface creation failed', u'code': 500, u'created': u'2019-03-25T16:41:48Z'}
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base Traceback (most recent call last):
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 201, in server_check_teardown
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base     cls.server_id, 'ACTIVE')
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base     server_id=server_id)
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base BuildErrorException: Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to build and is in ERROR status
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base Details: {u'message': u'Virtual Interface creation failed', u'code': 500, u'created': u'2019-03-25T16:41:48Z'}
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base 
2019-03-25 12:41:50,756 795439 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:tearDown): 204 DELETE https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/v2.1/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 0.314s
2019-03-25 12:41:50,757 795439 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}

        Body: None
    Response - Headers: {'status': '404', u'content-length': '111', 'content-location': 'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/v2.1/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878', u'x-compute-request-id': 'req-6a437388-506b-4d97-a601-472ae4a81b2a', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Mon, 25 Mar 2019 16:41:53 GMT', u'content-type': 'application/json; charset=UTF-8', u'x-openstack-request-id': 'req-6a437388-506b-4d97-a601-472ae4a81b2a'}
        Body: {"itemNotFound": {"message": "Instance bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 could not be found.", "code": 404}}
2019-03-25 12:41:53,249 795439 ERROR    [tempest.api.compute.base] Failed to delete server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base Traceback (most recent call last):
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 425, in delete_server
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     cls.servers_client.delete_server(server_id)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 150, in delete_server
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     resp, body = self.delete("servers/%s" % server_id)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 310, in delete
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     return self.request('DELETE', url, extra_headers, headers, body)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     method, url, extra_headers, headers, body, chunked)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 670, in request
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     self._error_checker(resp, resp_body)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 776, in _error_checker
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     raise exceptions.NotFound(resp_body, resp=resp)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base NotFound: Object not found
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base Details: {u'message': u'Instance bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 could not be found.', u'code': 404}
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base
}}}

traceback-1: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_actions.py", line 66, in tearDown
    self.server_check_teardown()
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 201, in server_check_teardown
    cls.server_id, 'ACTIVE')
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to build and is in ERROR status
Details: {u'message': u'Virtual Interface creation failed', u'code': 500, u'created': u'2019-03-25T16:41:48Z'}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_actions.py", line 370, in test_resize_server_revert
    waiters.wait_for_server_status(self.client, self.server_id, 'ACTIVE')
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
    raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (ServerActionsTestJSON:test_resize_server_revert) Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to reach ACTIVE status and task state "None" within the required time (300 s). Current status: REVERT_RESIZE. Current task state: resize_reverting.

Comment 2 Ronnie Rasouli 2019-04-01 10:13:54 UTC
Seems that the issue is also failing test: tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert_with_volume_attached[

Comment 3 Artom Lifshitz 2019-04-04 20:51:00 UTC
Given that I found the following in ./compute-0/var/log/containers/nova/nova-compute.log.1, I think we can safely close this as a duplicate of 1678681.

2019-03-25 16:41:47.956 1 WARNING nova.virt.libvirt.driver [req-41dcaefa-ad9f-4c07-8d8f-68583e060d6d 80d098d039b24ee7ab153cd7a6854972 bb640ac03b494b8fa9f5cad2d72c8ea1 - default default] [instance: bfd1b2bd-2662-
4aa4-97a0-e1a9adffb878] Timeout waiting for [('network-vif-plugged', u'f1b0f4aa-d018-4bed-a5e8-27a42c82d688')] for instance with vm_state resized and task_state resize_reverting.: Timeout: 300 seconds

and 

2019-03-25 16:41:48.287 1 ERROR nova.compute.manager [req-41dcaefa-ad9f-4c07-8d8f-68583e060d6d 80d098d039b24ee7ab153cd7a6854972 bb640ac03b494b8fa9f5cad2d72c8ea1 - default default] [instance: bfd1b2bd-2662-4aa4-97a0-e1a9adffb878] Setting instance vm_state to ERROR: VirtualInterfaceCreateException: Virtual Interface creation failed

*** This bug has been marked as a duplicate of bug 1678681 ***


Note You need to log in before you can comment on or make changes to this bug.