Bug 1440897 - Frequent failure at Subnet deletion with 3 controller
Summary: Frequent failure at Subnet deletion with 3 controller
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 11.0 (Ocata)
Assignee: Ihar Hrachyshka
QA Contact: Alexander Stafeyev
URL:
Whiteboard:
Depends On:
Blocks: 1440901 1440908
TreeView+ depends on / blocked
 
Reported: 2017-04-10 17:32 UTC by Attila Fazekas
Modified: 2017-08-23 08:14 UTC (History)
8 users (show)

Fixed In Version: openstack-neutron-10.0.0-16.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1440901 1440908 (view as bug list)
Environment:
Last Closed: 2017-05-17 20:19:46 UTC
afazekas: needinfo-


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC
Launchpad 1680912 None None None 2017-04-10 17:35:36 UTC

Description Attila Fazekas 2017-04-10 17:32:35 UTC
Description of problem:
The 4 thread tempest runs frequently fails with subnet deletion.
 
The tempest log contains multiple connection retry to the neutron server:

2017-04-07 02:01:40,246 24529 WARNING  [urllib3.connectionpool] Retrying (Retry(total=9, connect=None, read=None, redirect=5)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='10.0.0.110', port=9696): Read timed out. (read timeout=60)",)': /v2.0/subnets/9197a378-4a60-4e60-a0e0-1df71e26a35e


The exception at the cleanup phase:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet
    return self.delete_resource(uri)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 307, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 665, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 762, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Subnet 9197a378-4a60-4e60-a0e0-1df71e26a35e could not be found.', u'type': u'SubnetNotFound', u'detail': u''}



Version-Release number of selected component (if applicable):
python-neutron-lbaas-10.0.0-5.el7ost.noarch
python-neutronclient-6.1.0-1.el7ost.noarch
puppet-neutron-10.3.0-2.el7ost.noarch
openstack-neutron-common-10.0.0-13.el7ost.noarch
openstack-neutron-lbaas-10.0.0-5.el7ost.noarch
openstack-neutron-openvswitch-10.0.0-13.el7ost.noarch
python-neutron-10.0.0-13.el7ost.noarch
openstack-neutron-10.0.0-13.el7ost.noarch
openstack-neutron-metering-agent-10.0.0-13.el7ost.noarch
python-neutron-lib-1.1.0-1.el7ost.noarch
openstack-neutron-sriov-nic-agent-10.0.0-13.el7ost.noarch
openstack-neutron-ml2-10.0.0-13.el7ost.noarch


How reproducible:
It is reproducible even without a full tempest run.

25% chance with  ostestr -r PortsIpV6TestJSON

Comment 1 Ihar Hrachyshka 2017-04-10 17:34:50 UTC
This is a known regression in Ocata and earlier tracked in upstream as: https://bugs.launchpad.net/neutron/+bug/1680912

Comment 2 Assaf Muller 2017-04-11 21:00:14 UTC
Backport is posted d/s, should be in any day now. Setting as GA blocker.

Comment 5 Alexander Stafeyev 2017-04-27 15:11:16 UTC
What is that means ?> 

Which test exaclty? 

I need reproduction steps please.

Comment 6 Assaf Muller 2017-04-27 15:22:34 UTC
Attila mentions "25% chance with  ostestr -r PortsIpV6TestJSON" in the original bug report, if that passes reliably in a loop then I'd suggest that we're good.

Comment 7 Alexander Stafeyev 2017-04-27 15:45:27 UTC
openstack-neutron-10.0.1-1.el7ost.noarch

Ran the following 4 times - 100% success  

ostestr -r PortsIpV6TestJSON 



==============
Worker Balance
==============
 - Worker 0 (14 tests) => 0:02:00.718520

Slowest Tests:

Test id                                                                                                                                                   Runtime (s)
--------------------------------------------------------------------------------------------------------------------------------------------------------  -----------
tempest.api.network.test_ports.PortsIpV6TestJSON.test_port_list_filter_by_router_id[id-5ad01ed0-0e6e-4c5d-8194-232801b15c72]                              27.051
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_update_port_with_second_ip[id-63aeadd4-3b49-427f-a3b1-19ca81f06270]                          20.893
tempest.api.network.test_ports.PortsIpV6TestJSON.test_update_port_with_two_security_groups_and_extra_attributes[id-edf6766d-3d40-4621-bc6e-2521a44c257d]  14.518
tempest.api.network.test_ports.PortsIpV6TestJSON.test_update_port_with_security_group_and_extra_attributes[id-58091b66-4ff4-4cc1-a549-05d60c7acd1a]       12.271
tempest.api.network.test_ports.PortsIpV6TestJSON.test_port_list_filter_by_ip[id-e7fe260b-1e79-4dd3-86d9-bec6a7959fc5]                                     12.247
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups[id-4179dcb9-1382-4ced-84fe-1b91c54f5735,smoke]                   11.316
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools[id-0435f278-40ae-48cb-a404-b8a087bc09b1,smoke]               7.614
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_bulk_port[id-67f1b811-f8db-43e2-86bd-72c074d4a42c]                                            7.372
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_show_delete_port_user_defined_mac[id-13e95171-6cbd-489c-9d7c-3f9c58215c18]                    4.071
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_update_delete_port[id-c72c1c0c-2193-4aca-aaa4-b1442640f51c,smoke]                             2.601

Comment 8 errata-xmlrpc 2017-05-17 20:19:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.