Bug 1575696
| Summary: | neutron-openvswitch-agent eventually fails listing high count of ports on network node. | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Miguel Angel Ajo <majopela> | |
| Component: | openstack-neutron | Assignee: | Daniel Alvarez Sanchez <dalvarez> | |
| Status: | CLOSED EOL | QA Contact: | Toni Freger <tfreger> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 11.0 (Ocata) | CC: | akaris, amuller, bcafarel, bjarolim, chrisw, dalvarez, gkumar, jraju, lmiccini, nyechiel, pmorey, ragiman, skaplons, srevivo | |
| Target Milestone: | zstream | Keywords: | Reopened, Triaged, ZStream | |
| Target Release: | 11.0 (Ocata) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-neutron-10.0.5-6.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1579300 (view as bug list) | Environment: | ||
| Last Closed: | 2018-07-18 08:34:44 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1558336, 1579300, 1579352 | |||
|
Description
Miguel Angel Ajo
2018-05-07 16:49:36 UTC
Note, this is also likely to be an issue with OSP12 too, may be other earlier versions too? I tested this on an OSP11 setup with the following puddle: core_puddle_version 2018-02-08.3 OVS agent: openstack-neutron-openvswitch-10.0.5-1.el7ost.noarch * Created 2000 internal OVS ports: [root@controller-0 neutron]# ovs-vsctl list interface | grep uuid| wc -l 2018 * Restarted OVS agent and we can easily see timeouts when listing the ports: 2018-05-09 07:44:50.370 746921 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change do_commit /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py:124 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands [req-fe482d7b-276e-45fb-a6f4-be82a04ccbdc - - - - -] Error executing command 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands Traceback (most recent call last): 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py", line 36, in execute 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands txn.add(self) 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py", line 79, in __exit__ 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands self.result = self.commit() 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 73, in commit 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands 'timeout': self.timeout}) 2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands TimeoutException: Commands [DbListCommand(if_exists=True, records=[u'tap-919', u'tap-1263', u'tap-439', u'tap-803', u'tap-320', u'tap-741', u'tap-87', u'tap-764', u'tap-729', u'tap-1623', u'tap-1024', u'tap-47', u'tap-526',.... ] exceeded timeout 10 seconds * Now I apply this patch https://code.engineering.redhat.com/gerrit/#/c/138055/ . Restart OVS agent and look for DbListCommand traces in logs: 2018-05-09 07:50:32.914 769810 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbListCommand(if_exists=True, records=[u'tap-919', u'tap-1263', u'tap-439', u'ta p-803', u'tap-320', u'tap-741', u'tap-87', u'tap-764', u'tap-729', u'tap-1623', ... , table=Interface, columns=['name', 'external_ids', 'ofport']) do_commit /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py:100 2018-05-09 07:50:33.079 769810 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change do_commit /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py:124 Conclusion: The patch linked made the DbListCommand operation to take from >10 seconds to around 0.15 seconds. This clearly improves performance but also avoids other side effects of the timeout of this command. OSP11 is now retired, see details at https://access.redhat.com/errata/product/191/ver=11/rhel---7/x86_64/RHBA-2018:1828 |