Created attachment 1568610 [details]
tempest-run.log

Description of problem: While running Nova tempest tests after deploying RHOS-15, following tempest tests are failing on the error: Cannot access storage file '/var/lib/nova/instances/<uuid>/disk' (as uid:107, gid:107): Permission denied"}


1. tempest.api.compute.admin.test_migrations.MigrationsAdminTest.test_cold_migration
2. tempest.api.compute.admin.test_migrations.MigrationsAdminTest.test_list_migrations_in_flavor_resize_situation
3. tempest.api.compute.admin.test_migrations.MigrationsAdminTest.test_resize_server_revert_deleted_flavor
4. tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_verify_resize_state
5. tempest.api.compute.servers.test_disk_config.ServerDiskConfigTestJSON.test_resize_server_from_auto_to_manual
6. tempest.api.compute.servers.test_disk_config.ServerDiskConfigTestJSON.test_resize_server_from_manual_to_auto
7. tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm
8. tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm_from_stopped
9. tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert_with_volume_attached
10. tempest.api.compute.volumes.test_attach_volume.AttachVolumeMultiAttachTest.test_resize_server_with_multiattached_volume
11. tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize

Version-Release number of selected component (if applicable):
core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1

How reproducible: Always

Topology: 1 Controller, 2 Compute

Steps to Reproduce:
1. Deploy RHOS-15 with 1 controller, 2 compute
2. Run tempest tests

Actual results:
Above 11 tempest tests fail with the same error:

    Response - Headers: {'date': 'Sat, 11 May 2019 21:05:37 GMT', 'server': 'Apache', 'content-length': '1354', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-4ab79d01-a6e8-4c9f-b1c2-6986fa5876a1', 'x-compute-request-id': 'req-4ab79d01-a6e8-4c9f-b1c2-6986fa5876a1', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/005987b4-0318-4ef3-a607-efd65be77368'}
        Body: b'{"server": {"id": "005987b4-0318-4ef3-a607-efd65be77368", "name": "tempest-ServerDiskConfigTestJSON-server-1134618018", "status": "RESIZE", "tenant_id": "59306e5cb42141438fbaf8bdf805022f", "user_id": "3ad868f1beed422782bd619e3540a764", "metadata": {}, "hostId": "30d428e07c9f3055b743e02197e96fffb4d43ea5882cef7371929efa", "image": {"id": "0e69750d-552c-4f0e-a9d6-797bc415aa7f", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/images/0e69750d-552c-4f0e-a9d6-797bc415aa7f"}]}, "flavor": {"id": "ac93de77-01ce-4e91-a172-71875c18c55e", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/flavors/ac93de77-01ce-4e91-a172-71875c18c55e"}]}, "created": "2019-05-11T21:05:15Z", "updated": "2019-05-11T21:05:37Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.137:8774/v2.1/servers/005987b4-0318-4ef3-a607-efd65be77368"}, {"rel": "bookmark", "href": "http://10.0.0.137:8774/servers/005987b4-0318-4ef3-a607-efd65be77368"}], "OS-DCF:diskConfig": "AUTO", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-05-11T21:05:17.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": "resize_migrated", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2019-05-11 21:05:37,575 215815 INFO     [tempest.common.waiters] State transition "RESIZE/resize_migrating" ==> "RESIZE/resize_migrated" after 17 second wait
2019-05-11 21:05:38,732 215815 INFO     [tempest.lib.common.rest_client] Request (ServerDiskConfigTestJSON:test_resize_server_from_manual_to_auto): 200 GET http://10.0.0.137:8774/v2.1/servers/005987b4-0318-4ef3-a607-efd65be77368 0.155s
2019-05-11 21:05:38,732 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:05:38 GMT', 'server': 'Apache', 'content-length': '1544', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-e32f414f-55ff-40da-b50b-6e7c39b67c0e', 'x-compute-request-id': 'req-e32f414f-55ff-40da-b50b-6e7c39b67c0e', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/005987b4-0318-4ef3-a607-efd65be77368'}
        Body: b'{"server": {"id": "005987b4-0318-4ef3-a607-efd65be77368", "name": "tempest-ServerDiskConfigTestJSON-server-1134618018", "status": "ERROR", "tenant_id": "59306e5cb42141438fbaf8bdf805022f", "user_id": "3ad868f1beed422782bd619e3540a764", "metadata": {}, "hostId": "30d428e07c9f3055b743e02197e96fffb4d43ea5882cef7371929efa", "image": {"id": "0e69750d-552c-4f0e-a9d6-797bc415aa7f", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/images/0e69750d-552c-4f0e-a9d6-797bc415aa7f"}]}, "flavor": {"id": "ac93de77-01ce-4e91-a172-71875c18c55e", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/flavors/ac93de77-01ce-4e91-a172-71875c18c55e"}]}, "created": "2019-05-11T21:05:15Z", "updated": "2019-05-11T21:05:38Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.137:8774/v2.1/servers/005987b4-0318-4ef3-a607-efd65be77368"}, {"rel": "bookmark", "href": "http://10.0.0.137:8774/servers/005987b4-0318-4ef3-a607-efd65be77368"}], "OS-DCF:diskConfig": "AUTO", "fault": {"code": 500, "created": "2019-05-11T21:05:38Z", "message": "Cannot access storage file \'/var/lib/nova/instances/005987b4-0318-4ef3-a607-efd65be77368/disk\' (as uid:107, gid:107): Permission denied"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2019-05-11T21:05:17.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": "resize_finish", "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2019-05-11 21:05:38,737 215815 INFO     [tempest.common.waiters] State transition "RESIZE/resize_migrated" ==> "ERROR/resize_finish" after 18 second wait
2019-05-11 21:05:38,941 215815 INFO     [tempest.lib.common.rest_client] Request (ServerDiskConfigTestJSON:_run_cleanups): 204 DELETE http://10.0.0.137:8774/v2.1/servers/005987b4-0318-4ef3-a607-efd65be77368 0.203s
2019-05-11 21:05:38,942 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:05:38 GMT', 'server': 'Apache', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-1e920273-5625-4bbf-9cfe-0416f40b8836', 'x-compute-request-id': 'req-1e920273-5625-4bbf-9cfe-0416f40b8836', 'connection': 'close', 'status': '204', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/005987b4-0318-4ef3-a607-efd65be77368'}
        Body: b''
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_disk_config.py", line 98, in test_resize_server_from_manual_to_auto
    disk_config='AUTO')
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/base.py", line 424, in resize_server
    'VERIFY_RESIZE')
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server 005987b4-0318-4ef3-a607-efd65be77368 failed to build and is in ERROR status
Details: {'code': 500, 'created': '2019-05-11T21:05:38Z', 'message': "Cannot access storage file '/var/lib/nova/instances/005987b4-0318-4ef3-a607-efd65be77368/disk' (as uid:107, gid:107): Permission denied"}

    Response - Headers: {'date': 'Sat, 11 May 2019 21:15:00 GMT', 'server': 'Apache', 'content-length': '1925', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-ce042601-5a3a-4c36-b239-c0e5e09c3cb4', 'x-compute-request-id': 'req-ce042601-5a3a-4c36-b239-c0e5e09c3cb4', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483'}
        Body: b'{"server": {"id": "e6c2fb9d-a0ae-45a3-b22c-45865303c483", "name": "tempest-ServerActionsTestJSON-server-1870120660", "status": "ERROR", "tenant_id": "59e8e567688941f89efae0bbf6a0d3a6", "user_id": "2e0cd1029f94430aa692fbaf536afbf1", "metadata": {"rebuild": "server"}, "hostId": "c3eac17a1808415e7d9c75be02baf4105d1e0e1e8221a8a29542c6e8", "image": {"id": "0e69750d-552c-4f0e-a9d6-797bc415aa7f", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/images/0e69750d-552c-4f0e-a9d6-797bc415aa7f"}]}, "flavor": {"id": "ac93de77-01ce-4e91-a172-71875c18c55e", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/flavors/ac93de77-01ce-4e91-a172-71875c18c55e"}]}, "created": "2019-05-11T21:06:49Z", "updated": "2019-05-11T21:14:58Z", "addresses": {"tempest-ServerActionsTestJSON-1252015544-network": [{"version": 4, "addr": "10.100.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:2c:ac:46"}, {"version": 4, "addr": "10.0.0.247", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:2c:ac:46"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}, {"rel": "bookmark", "href": "http://10.0.0.137:8774/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2019-05-11T21:14:58Z", "message": "Cannot access storage file \'/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk\' (as uid:107, gid:107): Permission denied"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-638249548", "OS-SRV-USG:launched_at": "2019-05-11T21:13:49.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "tempest-securitygroup--1644861636"}], "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2019-05-11 21:15:01,375 215815 ERROR    [tempest.api.compute.base] Server e6c2fb9d-a0ae-45a3-b22c-45865303c483 failed to build and is in ERROR status
Details: {'code': 500, 'created': '2019-05-11T21:14:58Z', 'message': "Cannot access storage file '/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk' (as uid:107, gid:107): Permission denied"}
2019-05-11 21:15:01.375 215815 ERROR tempest.api.compute.base Traceback (most recent call last):
2019-05-11 21:15:01.375 215815 ERROR tempest.api.compute.base   File "/usr/lib/python3.6/site-packages/tempest/api/compute/base.py", line 201, in server_check_teardown
2019-05-11 21:15:01.375 215815 ERROR tempest.api.compute.base     cls.server_id, 'ACTIVE')
2019-05-11 21:15:01.375 215815 ERROR tempest.api.compute.base   File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
2019-05-11 21:15:01.375 215815 ERROR tempest.api.compute.base     server_id=server_id)
2019-05-11 21:15:01.375 215815 ERROR tempest.api.compute.base tempest.exceptions.BuildErrorException: Server e6c2fb9d-a0ae-45a3-b22c-45865303c483 failed to build and is in ERROR status
2019-05-11 21:15:01.375 215815 ERROR tempest.api.compute.base Details: {'code': 500, 'created': '2019-05-11T21:14:58Z', 'message': "Cannot access storage file '/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk' (as uid:107, gid:107): Permission denied"}
2019-05-11 21:15:01.375 215815 ERROR tempest.api.compute.base 
2019-05-11 21:15:01,602 215815 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:tearDown): 204 DELETE http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483 0.226s
2019-05-11 21:15:01,603 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:15:01 GMT', 'server': 'Apache', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-a43808b1-ecca-487e-a052-b942e71ba0f9', 'x-compute-request-id': 'req-a43808b1-ecca-487e-a052-b942e71ba0f9', 'connection': 'close', 'status': '204', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483'}
        Body: b''
2019-05-11 21:15:02,505 215815 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:tearDown): 200 GET http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483 0.902s
2019-05-11 21:15:02,506 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:15:01 GMT', 'server': 'Apache', 'content-length': '1931', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-9b56d909-1ca2-454d-a3d6-30d4437cf311', 'x-compute-request-id': 'req-9b56d909-1ca2-454d-a3d6-30d4437cf311', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483'}
        Body: b'{"server": {"id": "e6c2fb9d-a0ae-45a3-b22c-45865303c483", "name": "tempest-ServerActionsTestJSON-server-1870120660", "status": "ERROR", "tenant_id": "59e8e567688941f89efae0bbf6a0d3a6", "user_id": "2e0cd1029f94430aa692fbaf536afbf1", "metadata": {"rebuild": "server"}, "hostId": "c3eac17a1808415e7d9c75be02baf4105d1e0e1e8221a8a29542c6e8", "image": {"id": "0e69750d-552c-4f0e-a9d6-797bc415aa7f", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/images/0e69750d-552c-4f0e-a9d6-797bc415aa7f"}]}, "flavor": {"id": "ac93de77-01ce-4e91-a172-71875c18c55e", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/flavors/ac93de77-01ce-4e91-a172-71875c18c55e"}]}, "created": "2019-05-11T21:06:49Z", "updated": "2019-05-11T21:15:01Z", "addresses": {"tempest-ServerActionsTestJSON-1252015544-network": [{"version": 4, "addr": "10.100.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:2c:ac:46"}, {"version": 4, "addr": "10.0.0.247", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:2c:ac:46"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}, {"rel": "bookmark", "href": "http://10.0.0.137:8774/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2019-05-11T21:14:58Z", "message": "Cannot access storage file \'/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk\' (as uid:107, gid:107): Permission denied"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-638249548", "OS-SRV-USG:launched_at": "2019-05-11T21:13:49.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "tempest-securitygroup--1644861636"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2019-05-11 21:15:04,199 215815 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:tearDown): 200 GET http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483 0.683s
2019-05-11 21:15:04,200 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:15:03 GMT', 'server': 'Apache', 'content-length': '1931', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-e1e43c56-ed66-4992-9698-a2ab2972806b', 'x-compute-request-id': 'req-e1e43c56-ed66-4992-9698-a2ab2972806b', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483'}
        Body: b'{"server": {"id": "e6c2fb9d-a0ae-45a3-b22c-45865303c483", "name": "tempest-ServerActionsTestJSON-server-1870120660", "status": "ERROR", "tenant_id": "59e8e567688941f89efae0bbf6a0d3a6", "user_id": "2e0cd1029f94430aa692fbaf536afbf1", "metadata": {"rebuild": "server"}, "hostId": "c3eac17a1808415e7d9c75be02baf4105d1e0e1e8221a8a29542c6e8", "image": {"id": "0e69750d-552c-4f0e-a9d6-797bc415aa7f", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/images/0e69750d-552c-4f0e-a9d6-797bc415aa7f"}]}, "flavor": {"id": "ac93de77-01ce-4e91-a172-71875c18c55e", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/flavors/ac93de77-01ce-4e91-a172-71875c18c55e"}]}, "created": "2019-05-11T21:06:49Z", "updated": "2019-05-11T21:15:01Z", "addresses": {"tempest-ServerActionsTestJSON-1252015544-network": [{"version": 4, "addr": "10.100.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:2c:ac:46"}, {"version": 4, "addr": "10.0.0.247", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:2c:ac:46"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}, {"rel": "bookmark", "href": "http://10.0.0.137:8774/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2019-05-11T21:14:58Z", "message": "Cannot access storage file \'/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk\' (as uid:107, gid:107): Permission denied"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-638249548", "OS-SRV-USG:launched_at": "2019-05-11T21:13:49.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "tempest-securitygroup--1644861636"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2019-05-11 21:15:04,391 215815 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:_run_cleanups): 204 DELETE http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483 0.186s
2019-05-11 21:15:04,392 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:15:04 GMT', 'server': 'Apache', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-96fe2f9b-c78e-4a74-b366-1599e4189532', 'x-compute-request-id': 'req-96fe2f9b-c78e-4a74-b366-1599e4189532', 'connection': 'close', 'status': '204', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483'}
        Body: b''
2019-05-11 21:15:04,981 215815 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:_run_cleanups): 200 GET http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483 0.589s
2019-05-11 21:15:04,982 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:15:04 GMT', 'server': 'Apache', 'content-length': '1931', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-92b9aa39-1af3-4dab-b12f-6b4c9a1c7975', 'x-compute-request-id': 'req-92b9aa39-1af3-4dab-b12f-6b4c9a1c7975', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483'}
        Body: b'{"server": {"id": "e6c2fb9d-a0ae-45a3-b22c-45865303c483", "name": "tempest-ServerActionsTestJSON-server-1870120660", "status": "ERROR", "tenant_id": "59e8e567688941f89efae0bbf6a0d3a6", "user_id": "2e0cd1029f94430aa692fbaf536afbf1", "metadata": {"rebuild": "server"}, "hostId": "c3eac17a1808415e7d9c75be02baf4105d1e0e1e8221a8a29542c6e8", "image": {"id": "0e69750d-552c-4f0e-a9d6-797bc415aa7f", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/images/0e69750d-552c-4f0e-a9d6-797bc415aa7f"}]}, "flavor": {"id": "ac93de77-01ce-4e91-a172-71875c18c55e", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/flavors/ac93de77-01ce-4e91-a172-71875c18c55e"}]}, "created": "2019-05-11T21:06:49Z", "updated": "2019-05-11T21:15:04Z", "addresses": {"tempest-ServerActionsTestJSON-1252015544-network": [{"version": 4, "addr": "10.100.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:2c:ac:46"}, {"version": 4, "addr": "10.0.0.247", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:2c:ac:46"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}, {"rel": "bookmark", "href": "http://10.0.0.137:8774/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2019-05-11T21:14:58Z", "message": "Cannot access storage file \'/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk\' (as uid:107, gid:107): Permission denied"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-638249548", "OS-SRV-USG:launched_at": "2019-05-11T21:13:49.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "tempest-securitygroup--1644861636"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2019-05-11 21:15:06,311 215815 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:_run_cleanups): 200 GET http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483 0.323s
2019-05-11 21:15:06,312 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:15:05 GMT', 'server': 'Apache', 'content-length': '1602', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-54a96f69-28f2-4e6e-874d-b1a44d55fcda', 'x-compute-request-id': 'req-54a96f69-28f2-4e6e-874d-b1a44d55fcda', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483'}
        Body: b'{"server": {"id": "e6c2fb9d-a0ae-45a3-b22c-45865303c483", "name": "tempest-ServerActionsTestJSON-server-1870120660", "status": "DELETED", "tenant_id": "59e8e567688941f89efae0bbf6a0d3a6", "user_id": "2e0cd1029f94430aa692fbaf536afbf1", "metadata": {"rebuild": "server"}, "hostId": "c3eac17a1808415e7d9c75be02baf4105d1e0e1e8221a8a29542c6e8", "image": {"id": "0e69750d-552c-4f0e-a9d6-797bc415aa7f", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/images/0e69750d-552c-4f0e-a9d6-797bc415aa7f"}]}, "flavor": {"id": "ac93de77-01ce-4e91-a172-71875c18c55e", "links": [{"rel": "bookmark", "href": "http://10.0.0.137:8774/flavors/ac93de77-01ce-4e91-a172-71875c18c55e"}]}, "created": "2019-05-11T21:06:49Z", "updated": "2019-05-11T21:15:05Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}, {"rel": "bookmark", "href": "http://10.0.0.137:8774/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2019-05-11T21:14:58Z", "message": "Cannot access storage file \'/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk\' (as uid:107, gid:107): Permission denied"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-638249548", "OS-SRV-USG:launched_at": "2019-05-11T21:13:49.000000", "OS-SRV-USG:terminated_at": "2019-05-11T21:15:05.000000", "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "deleted", "OS-EXT-STS:power_state": 0, "os-extended-volumes:volumes_attached": []}}'
2019-05-11 21:15:06,320 215815 INFO     [tempest.common.waiters] State transition "ERROR/deleting" ==> "DELETED/None" after 2 second wait
2019-05-11 21:15:07,404 215815 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:_run_cleanups): 404 GET http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483 0.081s
2019-05-11 21:15:07,404 215815 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sat, 11 May 2019 21:15:07 GMT', 'server': 'Apache', 'content-length': '111', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-67d6ce86-02e1-4296-a8bd-13e0a2477a08', 'x-compute-request-id': 'req-67d6ce86-02e1-4296-a8bd-13e0a2477a08', 'connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'status': '404', 'content-location': 'http://10.0.0.137:8774/v2.1/servers/e6c2fb9d-a0ae-45a3-b22c-45865303c483'}
        Body: b'{"itemNotFound": {"code": 404, "message": "Instance e6c2fb9d-a0ae-45a3-b22c-45865303c483 could not be found."}}'
}}}

traceback-1: {{{
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/base.py", line 201, in server_check_teardown
    cls.server_id, 'ACTIVE')
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server e6c2fb9d-a0ae-45a3-b22c-45865303c483 failed to build and is in ERROR status
Details: {'code': 500, 'created': '2019-05-11T21:14:58Z', 'message': "Cannot access storage file '/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk' (as uid:107, gid:107): Permission denied"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_server_actions.py", line 66, in tearDown
    self.server_check_teardown()
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/base.py", line 206, in server_check_teardown
    cls.server_id)
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 124, in wait_for_server_termination
    raise lib_exc.DeleteErrorException(resource_id=server_id)
tempest.lib.exceptions.DeleteErrorException: Resource e6c2fb9d-a0ae-45a3-b22c-45865303c483 failed to delete and is in ERROR status
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_server_actions.py", line 324, in test_resize_server_confirm
    self._test_resize_server_confirm(self.server_id, stop=False)
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_server_actions.py", line 306, in _test_resize_server_confirm
    'VERIFY_RESIZE')
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server e6c2fb9d-a0ae-45a3-b22c-45865303c483 failed to build and is in ERROR status
Details: {'code': 500, 'created': '2019-05-11T21:14:58Z', 'message': "Cannot access storage file '/var/lib/nova/instances/e6c2fb9d-a0ae-45a3-b22c-45865303c483/disk' (as uid:107, gid:107): Permission denied"}

Expected results:


Additional info: