Bug 1271232

Summary: tempest_lib.exceptions.Conflict: An object with that identifier already exists
Product: [Community] RDO Reporter: Ola Pavlenko <opavlenk>
Component: rdo-managerAssignee: Hugh Brock <hbrock>
Status: CLOSED EOL QA Contact: Shai Revivo <srevivo>
Severity: high Docs Contact:
Priority: unspecified    
Version: LibertyCC: bkopilov, jtrowbri, mburns, opavlenk
Target Milestone: ---   
Target Release: Kilo   
Hardware: x86_64   
OS: Other   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-19 15:32:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
tempest.log
none
tempest_new.log none

Description Ola Pavlenko 2015-10-13 12:45:07 UTC 
Created attachment 1082407 [details]
tempest.log

Description of problem:
GETTING  tempest_lib.exceptions.Conflict: An object with that identifier already exists tempest results


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.deploy overcloud ( 1 compue + 1 controller + 1 ceph) on Centos Virt env using guide: https://repos.fedorapeople.org/repos/openstack-m/rdo-manager-docs/liberty/environments/virtual.html
2. Configure overcloud network prior running tempest :
neutron net-create nova --router:external --provider:network_type vlan \
  --provider:physical_network datacentre --provider:segmentation_id 195
neutron subnet-create --name nova --disable-dhcp \
  --allocation-pool start=172.16.23.140,end=172.16.23.240 \
  --gateway 172.16.23.251 nova 172.16.23.128/25

3.Install and configure tempest: 
 sudo yum install openstack-tempest
     mkdir ~/tempest
     cd ~/tempest
     /usr/share/openstack-tempest-3.0.1/tools/configure-tempest-directory
     source ~/overcloudrc

 tools/config_tempest.py  --deployer-input ~/tempest-deployer-input.conf --debug --create identity.uri $OS_AUTH_URL identity.admin_password $OS_PASSWORD

4. run tempest:
  tools/run-tests.sh .*smoke


Actual results:

Tempest result:
tearDownClass (tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2)
---------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/stack/tempest/tempest/test.py", line 312, in tearDownClass
        six.reraise(etype, value, trace)
      File "/home/stack/tempest/tempest/test.py", line 295, in tearDownClass
        teardown()
      File "/home/stack/tempest/tempest/test.py", line 523, in clear_isolated_creds
        cls._creds_provider.clear_isolated_creds()
      File "/home/stack/tempest/tempest/common/isolated_creds.py", line 323, in clear_isolated_creds
        self._clear_isolated_net_resources()
      File "/home/stack/tempest/tempest/common/isolated_creds.py", line 314, in _clear_isolated_net_resources
        creds.subnet['name'])
      File "/home/stack/tempest/tempest/common/isolated_creds.py", line 265, in _clear_isolated_subnet
        net_client.delete_subnet(subnet_id)
      File "/home/stack/tempest/tempest/services/network/json/network_client.py", line 53, in delete_subnet
        return self.delete_resource(uri)
      File "/home/stack/tempest/tempest/services/network/json/base.py", line 40, in delete_resource
        resp, body = self.delete(req_uri)
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 290, in delete
        return self.request('DELETE', url, extra_headers, headers, body)
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 639, in request
        resp, resp_body)
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 702, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest_lib.exceptions.Conflict: An object with that identifier already exists
    Details: {u'message': u'Unable to complete operation on subnet 23fd4ecc-88a7-43b7-bddc-bdfd900969f9. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}
    


======
Totals
======
Ran: 131 tests in 1684.0000 sec.
 - Passed: 92
 - Skipped: 19
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 20
Sum of execute time for each test: 1004.4454 sec.

==============
Worker Balance
==============
 - Worker 0 (131 tests) => 0:27:42.844375


Expected results:
No Tracebacks

Additional info:
tempest log attched
Comment 1 John Trowbridge 2015-10-13 13:20:49 UTC 
Does this network actually match your environment?

2. Configure overcloud network prior running tempest :
neutron net-create nova --router:external --provider:network_type vlan \
  --provider:physical_network datacentre --provider:segmentation_id 195
neutron subnet-create --name nova --disable-dhcp \
  --allocation-pool start=172.16.23.140,end=172.16.23.240 \
  --gateway 172.16.23.251 nova 172.16.23.128/25

By default for virt, that would be 192.0.2.0/24, and not on a vlan.
Comment 2 Ola Pavlenko 2015-10-13 15:07:27 UTC 
(In reply to John Trowbridge from comment #1)
> Does this network actually match your environment?
> 
> 2. Configure overcloud network prior running tempest :
> neutron net-create nova --router:external --provider:network_type vlan \
>   --provider:physical_network datacentre --provider:segmentation_id 195
> neutron subnet-create --name nova --disable-dhcp \
>   --allocation-pool start=172.16.23.140,end=172.16.23.240 \
>   --gateway 172.16.23.251 nova 172.16.23.128/25
> 
> By default for virt, that would be 192.0.2.0/24, and not on a vlan.

I've used the doc. And wasn't sure about it... I will try to change and re-run tempest
Comment 3 Ola Pavlenko 2015-10-13 16:26:59 UTC 
I've delete nova net and subnet, and created with correct values as following: 

neutron net-create nova --provider:network_type flat --provider:physical_network datacentre       --router:external

 neutron subnet-create nova 192.0.2.0/24 --gateway 192.0.2.1 --enable_dhcp=False --allocation-pool start=192.0.2.200,end=192.0.2.210

and re-ran tempest, the results as following:

(the error I got now is a bit different )

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/stack/tempest/tempest/test.py", line 312, in tearDownClass
        six.reraise(etype, value, trace)
      File "/home/stack/tempest/tempest/test.py", line 295, in tearDownClass
        teardown()
      File "/home/stack/tempest/tempest/test.py", line 523, in clear_isolated_creds
        cls._creds_provider.clear_isolated_creds()
      File "/home/stack/tempest/tempest/common/isolated_creds.py", line 323, in clear_isolated_creds
        self._clear_isolated_net_resources()
      File "/home/stack/tempest/tempest/common/isolated_creds.py", line 314, in _clear_isolated_net_resources
        creds.subnet['name'])
      File "/home/stack/tempest/tempest/common/isolated_creds.py", line 265, in _clear_isolated_subnet
        net_client.delete_subnet(subnet_id)
      File "/home/stack/tempest/tempest/services/network/json/network_client.py", line 53, in delete_subnet
        return self.delete_resource(uri)
      File "/home/stack/tempest/tempest/services/network/json/base.py", line 40, in delete_resource
        resp, body = self.delete(req_uri)
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 290, in delete
        return self.request('DELETE', url, extra_headers, headers, body)
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 639, in request
        resp, resp_body)
      File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 702, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest_lib.exceptions.Conflict: An object with that identifier already exists
    Details: {u'message': u'Unable to complete operation on subnet 0291fbc2-b976-4e24-a498-482eeda8a9df. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}
Comment 4 Ola Pavlenko 2015-10-13 16:29:22 UTC 
Created attachment 1082501 [details]
tempest_new.log
Comment 5 Ola Pavlenko 2015-10-13 16:30:17 UTC 
$ neutron net-list
+--------------------------------------+-----------------------------------------------------+----------------------------------------------------+
| id                                   | name                                                | subnets                                            |
+--------------------------------------+-----------------------------------------------------+----------------------------------------------------+
| 93af8475-fae4-4d5e-abce-0c68cc77d622 | tempest-ServersTestManualDisk-1944172414-network    | 9b290a3a-a632-4bad-9c6d-20c4328f183d 10.100.0.0/28 |
| 07aa440e-0699-4bb2-a860-ce6ec6bb995b | tempest-ServersTestJSON-1300967235-network          | 63842461-7447-4fda-bd9e-5ccab4910178 10.100.0.0/28 |
| 398e449b-fdee-4034-9701-1359a9f01479 | tempest-ServerAddressesTestJSON-782247326-network   | 8470e70b-80e6-4838-8b5c-932675e24480 10.100.0.0/28 |
| 91eb59e8-7ca5-4964-a39d-aae9da2190cf | tempest-ServerAddressesTestJSON-1449725513-network  | 5b6c9539-e179-4a4e-9b76-69c66221ae36 10.100.0.0/28 |
| 342e67bb-bd2b-4e59-b5c9-b02921c2c04b | tempest-TestVolumeBootPattern-1432522566-network    | 039938ae-44c8-4b8b-84ff-aa00dc232346 10.100.0.0/28 |
| ef14c230-2e6b-4ff9-a8ae-334755f78694 | tempest-AttachInterfacesTestJSON-2001350369-network | d32ee64d-01cc-4246-8947-f47efec66f4c 10.100.0.0/28 |
| 6679127d-86dc-4356-8031-93e280185047 | tempest-ServersTestJSON-1644488754-network          | d6ebbaa1-ddca-40c3-87e1-417051fb1f5c 10.100.0.0/28 |
| 2a5786c8-ee6b-4df1-80fe-5426bce37572 | nova                                                | f8c044dd-881c-465e-a05d-fe58b086f053 192.0.2.0/24  |
| e1c8caa9-3fcc-4b26-ba34-b4d6d1adce12 | tempest-AttachInterfacesTestJSON-77919250-network   | 6fd79cd8-7b39-4630-a271-2375b50aa5ad 10.100.0.0/28 |
| 636ad262-0da6-4a37-bdc3-49ff7de685db | tempest-TestVolumeBootPatternV2-1552685182-network  | 0291fbc2-b976-4e24-a498-482eeda8a9df 10.100.0.0/28 |
| 2869590f-b4cb-4137-9cca-216bd5464b55 | tempest-network-smoke--1237148653                   | b94554b5-4fca-46cd-896d-6223569f6718 10.100.0.0/28 |
| d992d363-7216-4b89-82a5-0a9bc4aa1200 | tempest-TestServerBasicOps-1569476594-network       | b8e3d458-ca66-43af-a16d-f239c811984a 10.100.0.0/28 |
| 54779516-ec06-4fba-86ad-0252ee53b342 | tempest-ServerActionsTestJSON-13469931-network      | 8e1f4cca-e490-44c8-a5b2-c219ed50445e 10.100.0.0/28 |
| d21e1cc6-1fb6-4b82-8ccd-564b6dc966e9 | tempest-ServerActionsTestJSON-1275817983-network    | e884ac7b-412b-4b15-a993-8ef3ca7d31aa 10.100.0.0/28 |
| 210f9fa0-097e-4ec7-abd1-0828c1b57ef3 | tempest-TestVolumeBootPattern-1751498654-network    | 0463589d-59ac-4b91-8a48-0f7b7502af31 10.100.0.0/28 |
| 4b2215ac-21c0-4b4e-9edf-7ed9e8f5e5b3 | tempest-network-smoke--2094707612                   | 74826fbc-84a6-4308-b555-f8fb45696454 10.100.0.0/28 |
| bd5ac303-2528-46b4-b207-7d686c96eb20 | tempest-TestServerBasicOps-267121015-network        | 1f9dc68a-df12-4408-aa72-9a0b7a165859 10.100.0.0/28 |
| 13d56e60-9675-4c68-b782-38564835bc81 | tempest-TestVolumeBootPatternV2-772934026-network   | 23fd4ecc-88a7-43b7-bddc-bdfd900969f9 10.100.0.0/28 |
| 3eb72ebe-72e0-41ba-ae0b-b6a9a57d5e4c | tempest-ServersTestManualDisk-2094843042-network    | 130b1b53-3447-426d-bc93-831918930569 10.100.0.0/28 |
+--------------------------------------+-----------------------------------------------------+----------------------------------------------------+


$ neutron subnet-list
+--------------------------------------+----------------------------------------------------+---------------+------------------------------------------------+
| id                                   | name                                               | cidr          | allocation_pools                               |
+--------------------------------------+----------------------------------------------------+---------------+------------------------------------------------+
| 5b6c9539-e179-4a4e-9b76-69c66221ae36 | tempest-ServerAddressesTestJSON-1449725513-subnet  | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 9b290a3a-a632-4bad-9c6d-20c4328f183d | tempest-ServersTestManualDisk-1944172414-subnet    | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 23fd4ecc-88a7-43b7-bddc-bdfd900969f9 | tempest-TestVolumeBootPatternV2-772934026-subnet   | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| f8c044dd-881c-465e-a05d-fe58b086f053 |                                                    | 192.0.2.0/24  | {"start": "192.0.2.200", "end": "192.0.2.210"} |
| 8e1f4cca-e490-44c8-a5b2-c219ed50445e | tempest-ServerActionsTestJSON-13469931-subnet      | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 1f9dc68a-df12-4408-aa72-9a0b7a165859 | tempest-TestServerBasicOps-267121015-subnet        | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| e884ac7b-412b-4b15-a993-8ef3ca7d31aa | tempest-ServerActionsTestJSON-1275817983-subnet    | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 8470e70b-80e6-4838-8b5c-932675e24480 | tempest-ServerAddressesTestJSON-782247326-subnet   | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 130b1b53-3447-426d-bc93-831918930569 | tempest-ServersTestManualDisk-2094843042-subnet    | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| d6ebbaa1-ddca-40c3-87e1-417051fb1f5c | tempest-ServersTestJSON-1644488754-subnet          | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 74826fbc-84a6-4308-b555-f8fb45696454 | tempest-subnet-smoke-1389455511                    | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| d32ee64d-01cc-4246-8947-f47efec66f4c | tempest-AttachInterfacesTestJSON-2001350369-subnet | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 0463589d-59ac-4b91-8a48-0f7b7502af31 | tempest-TestVolumeBootPattern-1751498654-subnet    | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 6fd79cd8-7b39-4630-a271-2375b50aa5ad | tempest-AttachInterfacesTestJSON-77919250-subnet   | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| b94554b5-4fca-46cd-896d-6223569f6718 | tempest-subnet-smoke-2142783551                    | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 039938ae-44c8-4b8b-84ff-aa00dc232346 | tempest-TestVolumeBootPattern-1432522566-subnet    | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 63842461-7447-4fda-bd9e-5ccab4910178 | tempest-ServersTestJSON-1300967235-subnet          | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| b8e3d458-ca66-43af-a16d-f239c811984a | tempest-TestServerBasicOps-1569476594-subnet       | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
| 0291fbc2-b976-4e24-a498-482eeda8a9df | tempest-TestVolumeBootPatternV2-1552685182-subnet  | 10.100.0.0/28 | {"start": "10.100.0.2", "end": "10.100.0.14"}  |
+--------------------------------------+----------------------------------------------------+---------------+------------------------------------------------+
Comment 6 bkopilov 2015-10-13 18:45:55 UTC 
Hi Ola, 
You should deploy the overcloud with "--libvirt-type qemu"  flag.

in your compute logs we hit on : 
libvirtError: internal error: no supported architecture for os type 'hvm'

This is the reason for instnce boot failures .

Thanks,
Benny
Comment 7 Ola Pavlenko 2015-10-14 17:04:20 UTC 
Ok, so I have re-installed from scratch and redeployed. then re-ran tempest. 

The result is as following:
======
Totals
======
Ran: 130 tests in 2081.0000 sec.
 - Passed: 104
 - Skipped: 19
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 7
Sum of execute time for each test: 1268.3879 sec.

==============
Worker Balance
==============
 - Worker 0 (130 tests) => 0:34:18.506613


Steps:

0.  openstack overcloud deploy --ceph-storage-scale 1 --templates -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml --libvirt-type qemu 
1. . overcloudrc 
2. neutron net-create nova --provider:network_type flat --provider:physical_network datacentre       --router:external
3. neutron subnet-create nova 192.0.2.0/24 --gateway 192.0.2.1 --enable_dhcp=False --allocation-pool start=192.0.2.200,end=192.0.2.210
4. mkdir ~/tempest
5. cd ~/tempest
6. sudo yum install openstack-tempest
7./usr/share/openstack-tempest-3.0.1/tools/configure-tempest-directory
8. tools/config_tempest.py  --deployer-input ~/tempest-deployer-input.conf --debug --create identity.uri $OS_AUTH_URL identity.admin_password $OS_PASSWORD
9. tools/run-tests.sh .*smoke
Comment 11 Chandan Kumar 2016-05-19 15:32:48 UTC 
This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.