Bug 1250081 - test_minimum_basic scenario failed to run on rdo-manager [NEEDINFO]
test_minimum_basic scenario failed to run on rdo-manager
Status: CLOSED EOL
Product: RDO
Classification: Community
Component: tempest (Show other bugs)
trunk
Unspecified Unspecified
medium Severity medium
: RC
: trunk
Assigned To: David Kranz
tkammer
: Automation
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-04 09:28 EDT by mathieu bultel
Modified: 2017-06-18 02:36 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-18 02:36:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
snecklifter: needinfo? (mbultel)


Attachments (Terms of Use)
tempest.log (942.54 KB, text/plain)
2015-08-04 09:28 EDT, mathieu bultel
no flags Details

  None (edit)
Description mathieu bultel 2015-08-04 09:28:52 EDT
Created attachment 1059093 [details]
tempest.log

Description of problem:

When I run test_minimal_basic scenario on rdo-manager / centos
the test failed at the step :
self.nova_volume_attach()

(see attachment for more information)

When launching an instance manually and attaching a volume to the instance, there is no issue.
tempest.conf:

[DEFAULT]
debug = true
use_stderr = false
log_file = tempest.log

[auth]
tempest_roles = _member_

[compute]
image_ssh_user = fedora
allow_tenant_isolation = true
build_timeout = 900
ssh_user = fedora
flavor_ref = 2
flavor_ref_alt = 2
image_ref = 2e5e8ce2d2c043dea4754e56c0fc22df
image_ref_alt = 2e5e8ce2d2c043dea4754e56c0fc22df

[identity]
username = demo
tenant_name = demo
password = secrete
alt_username = alt_demo
alt_tenant_name = alt_demo
alt_password = secrete
admin_username = admin
admin_tenant_name = admin
admin_domain_name = Default
disable_ssl_certificate_validation = false
uri = http://192.0.2.6:5000/v2.0/
admin_password = aee8208bef006c91feec73fa4a3e61041a664079
uri_v3 = http://192.0.2.6:5000/v3/
admin_tenant_id = 2e5e8ce2d2c043dea4754e56c0fc22df

[object-storage]
operator_role = swiftoperator

[data_processing]
catalog_type = data-processing

[orchestration]

[scenario]
img_dir = etc
ssh_user = fedora

[oslo_concurrency]
lock_path = /tmp

[volume-feature-enabled]
bootable = true
api_extensions = OS-SCH-HNT,os-hosts,os-vol-tenant-attr,os-quota-sets,os-availability-zone,os-volume-encryption-metadata,os-volume-replication,backups,os-snapshot-actions,cgsnapshots,os-used-limits,os-volume-type-access,consistencygroups,os-vol-host-attr,encryption,os-types-manage,qos-specs,os-volume-actions,os-types-extra-specs,os-vol-mig-status-attr,os-volume-unmanage,os-image-create,os-extended-services,os-extended-snapshot-attributes,os-vol-image-meta,os-quota-class-sets,os-volume-transfer,os-volume-manage,os-admin-actions,os-services,scheduler-stats
api_v1 = True
api_v2 = True

[compute-feature-enabled]
live_migrate_paused_instances = true
preserve_ports = true
api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-EXT-VIF-NET,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-admin-actions,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes

[network-feature-enabled]
ipv6_subnet_attributes = true
api_extensions = security-group,l3_agent_scheduler,net-mtu,ext-gw-mode,binding,provider,agent,quotas,subnet_allocation,dhcp_agent_scheduler,l3-ha,multi-provider,external-net,router,allowed-address-pairs,extraroute,extra_dhcp_opt,dvr

[network]
build_timeout = 500
public_network_id = c88c6cc1-aee6-443e-b48e-94c49abcfaff

[volume]
build_timeout = 500

[service_available]
glance = True
cinder = True
swift = True
nova = True
neutron = True
trove = False
ceilometer = True
sahara = False
ironic = False
heat = True
zaqar = False
horizon = True

[object-storage-feature-enabled]
discoverable_apis = formpost,staticweb,ratelimit,tempurl

[image-feature-enabled]
api_v1 = True
api_v2 = True

[identity-feature-enabled]
api_v2 = True
api_v3 = True

[boto]
ec2_url = http://192.0.2.9:8773/services/Cloud

[dashboard]
dashboard_url = http://192.0.2.6/dashboard/
login_url = http://192.0.2.6/dashboard/auth/login/
Comment 3 mathieu bultel 2015-08-04 09:47:00 EDT
            Body: {"server": {"status": "ACTIVE", "updated": "2015-08-04T13:45:45Z", "hostId": "6bd6a42ab88e97641305170fd759b30ad14583a6362f7cde3f230617", "addresses": {"TestMinimumBasicScenario-677603733-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:fc:a2:e4", "version": 4, "addr": "10.100.0.3", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://192.0.2.6:8774/v2/0fa854fb9aaa4d10862bd07b5a8dfcc3/servers/0780670b-21f8-463d-8279-c3de433acbea", "rel": "self"}, {"href": "http://192.0.2.6:8774/0fa854fb9aaa4d10862bd07b5a8dfcc3/servers/0780670b-21f8-463d-8279-c3de433acbea", "rel": "bookmark"}], "key_name": "TestMinimumBasicScenario-2106708450", "image": {"id": "80aaedc9-b90e-4d73-8205-0f37865b7dc8", "links": [{"href": "http://192.0.2.6:8774/0fa854fb9aaa4d10862bd07b5a8dfcc3/images/80aaedc9-b90e-4d73-8205-0f37865b7dc8", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2015-08-04T13:37:38.000000", "flavor": {"id": "2", "links": [{"href": "http://192.0.2.6:8774/0fa854fb9aaa4d10862bd07b5a8dfcc3/flavors/2", "rel": "bookmark"}]}, "id": "0780670b-21f8-463d-8279-c3de433acbea", "security_groups": [{"name": "default"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "6caef445d5cd41cca518982d6a66e1f3", "name": "TestMinimumBasicScenario-945558213", "created": "2015-08-04T13:37:28Z", "tenant_id": "0fa854fb9aaa4d10862bd07b5a8dfcc3", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [{"id": "f49755c3-9b58-4560-8652-c195210a6291"}], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {}}}
    2015-08-04 13:45:49,628 24546 INFO     [tempest_lib.common.rest_client] Request (TestMinimumBasicScenario:_run_cleanups): 200 GET http://192.0.2.6:8774/v2/0fa854fb9aaa4d10862bd07b5a8dfcc3/servers/0780670b-21f8-463d-8279-c3de433acbea 0.141s
    2015-08-04 13:45:49,629 24546 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '200', 'content-length': '1468', 'content-location': 'http://192.0.2.6:8774/v2/0fa854fb9aaa4d10862bd07b5a8dfcc3/servers/0780670b-21f8-463d-8279-c3de433acbea', 'x-compute-request-id': 'req-e0aba47d-63c4-4417-9743-c8c3d99e3115', 'connection': 'close', 'date': 'Tue, 04 Aug 2015 13:45:49 GMT', 'content-type': 'application/json'}
            Body: {"server": {"status": "ACTIVE", "updated": "2015-08-04T13:45:45Z", "hostId": "6bd6a42ab88e97641305170fd759b30ad14583a6362f7cde3f230617", "addresses": {}, "links": [{"href": "http://192.0.2.6:8774/v2/0fa854fb9aaa4d10862bd07b5a8dfcc3/servers/0780670b-21f8-463d-8279-c3de433acbea", "rel": "self"}, {"href": "http://192.0.2.6:8774/0fa854fb9aaa4d10862bd07b5a8dfcc3/servers/0780670b-21f8-463d-8279-c3de433acbea", "rel": "bookmark"}], "key_name": "TestMinimumBasicScenario-2106708450", "image": {"id": "80aaedc9-b90e-4d73-8205-0f37865b7dc8", "links": [{"href": "http://192.0.2.6:8774/0fa854fb9aaa4d10862bd07b5a8dfcc3/images/80aaedc9-b90e-4d73-8205-0f37865b7dc8", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2015-08-04T13:37:38.000000", "flavor": {"id": "2", "links": [{"href": "http://192.0.2.6:8774/0fa854fb9aaa4d10862bd07b5a8dfcc3/flavors/2", "rel": "bookmark"}]}, "id": "0780670b-21f8-463d-8279-c3de433acbea", "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "6caef445d5cd41cca518982d6a66e1f3", "name": "TestMinimumBasicScenario-945558213", "created": "2015-08-04T13:37:28Z", "tenant_id": "0fa854fb9aaa4d10862bd07b5a8dfcc3", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [{"id": "f49755c3-9b58-4560-8652-c195210a6291"}], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {}}}
    2015-08-04 13:45:50,702 24546 INFO     [tempest_lib.common.rest_client] Request (TestMinimumBasicScenario:_run_cleanups): 404 GET http://192.0.2.6:8774/v2/0fa854fb9aaa4d10862bd07b5a8dfcc3/servers/0780670b-21f8-463d-8279-c3de433acbea 0.053s
    2015-08-04 13:45:50,702 24546 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '404', 'content-length': '111', 'x-compute-request-id': 'req-54d033ca-650c-49d4-b0df-874d3b591616', 'connection': 'close', 'date': 'Tue, 04 Aug 2015 13:45:50 GMT', 'content-type': 'application/json; charset=UTF-8'}
            Body: {"itemNotFound": {"message": "Instance 0780670b-21f8-463d-8279-c3de433acbea could not be found.", "code": 404}}
    2015-08-04 13:45:50,717 24546 INFO     [tempest_lib.common.rest_client] Request (TestMinimumBasicScenario:_run_cleanups): 202 DELETE http://192.0.2.6:8774/v2/0fa854fb9aaa4d10862bd07b5a8dfcc3/os-keypairs/TestMinimumBasicScenario-2106708450 0.014s
    2015-08-04 13:45:50,717 24546 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '202', 'content-length': '0', 'x-compute-request-id': 'req-4ae37d7f-e35b-4a18-b923-9edf7b4dc8e4', 'connection': 'close', 'date': 'Tue, 04 Aug 2015 13:45:50 GMT', 'content-type': 'text/html; charset=UTF-8'}
            Body: 
    2015-08-04 13:45:51,485 24546 INFO     [tempest_lib.common.rest_client] Request (TestMinimumBasicScenario:_run_cleanups): 200 DELETE http://192.0.2.6:9292/v1/images/80aaedc9-b90e-4d73-8205-0f37865b7dc8 0.768s
    2015-08-04 13:45:51,486 24546 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '200', 'content-length': '0', 'connection': 'close', 'date': 'Tue, 04 Aug 2015 13:45:51 GMT', 'content-type': 'text/html; charset=UTF-8', 'x-openstack-request-id': 'req-req-89dc3a4a-e2ec-443b-ae07-b76f9b704b19'}
            Body: 
    2015-08-04 13:45:51,532 24546 INFO     [tempest_lib.common.rest_client] Request (TestMinimumBasicScenario:_run_cleanups): 404 GET http://192.0.2.6:8774/v2/0fa854fb9aaa4d10862bd07b5a8dfcc3/servers/0780670b-21f8-463d-8279-c3de433acbea 0.046s
    2015-08-04 13:45:51,533 24546 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '404', 'content-length': '111', 'x-compute-request-id': 'req-0a087ed7-7b4e-442c-90c9-86077b66f97d', 'connection': 'close', 'date': 'Tue, 04 Aug 2015 13:45:51 GMT', 'content-type': 'application/json; charset=UTF-8'}
            Body: {"itemNotFound": {"message": "Instance 0780670b-21f8-463d-8279-c3de433acbea could not be found.", "code": 404}}
Comment 4 mathieu bultel 2015-08-04 10:09:34 EDT
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/stack/tempest/tempest/scenario/manager.py", line 89, in delete_wrapper
        delete_thing(*args, **kwargs)
      File "/home/stack/tempest/tempest/services/volume/json/volumes_client.py", line 96, in delete_volume
        resp, body = self.delete("volumes/%s" % str(volume_id))
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 283, in delete
        return self.request('DELETE', url, extra_headers, headers, body)
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 629, in request
        resp, resp_body)
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 680, in _error_checker
        raise exceptions.BadRequest(resp_body)
    BadRequest: Bad request
    Details: {u'message': u'Volume ae225be1-d03f-4616-a286-9211fbd6f318 is still attached, detach volume first.', u'code': 400}
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stderr

    pythonlogging:'': {{{
    2015-08-04 13:58:01,436 26979 DEBUG    [tempest.scenario.manager] paths: img: etc/cirros-0.3.1-x86_64-disk.img, container_fomat: bare, disk_format: qcow2, properties: None, ami: etc/cirros-0.3.1-x86_64-blank.img, ari: etc/cirros-0.3.1-x86_64-initrd, aki: etc/cirros-0.3.1-x86_64-vmlinuz
    2015-08-04 13:58:01,641 26979 INFO     [tempest_lib.common.rest_client] Request (TestMinimumBasicScenario:test_minimum_basic_scenario): 200 POST http://192.0.2.6:5000/v2.0/tokens
    2015-08-04 13:58:01,642 26979 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {}
            Body: None
        Response - Headers: {'status': '200', 'content-length': '4109', 'vary': 'X-Auth-Token', 'connection': 'close', 'date': 'Tue, 04 Aug 2015 13:58:01 GMT', 'content-type': 'application/json', 'x-openstack-request-id': 'req-a939f65f-be1d-4cd7-99ad-30d0dde644a2'}
            Body: None
Comment 5 mathieu bultel 2015-08-06 10:02:25 EDT
This issue is :
100% reproducible on RDO

The test passed on OSP-D on several config :
HA
with and without network isolation
BM
gre & ml2-vxlan

It appear with a basic virt setup with RDO (kilo) on ml2
like this job on centosci gate:

https://ci.centos.org/job/rdo_manager-gate_khaleesi-rdo-kilo/1128/console
Comment 6 David Kranz 2015-08-06 16:48:37 EDT
The following is what is happening as can be seen in the tempest.log:

A server and volume are created, and the volume is attached to the server.
There is a failed call to create a floating ip due to a misconfiguration issue.
The cleanup code in tempest detaches the volume and then deletes it, polling to see if the volume has been deleted. It never is, and eventually the test times out.

Oddly, I was able to reproduce this running tempest manually, but after the run I was able to delete the volume from the command line.
Comment 7 Christopher Brown 2017-06-17 12:40:50 EDT
Suggest this bug is stale can should be closed.

Note You need to log in before you can comment on or make changes to this bug.