Bug 1575696 - neutron-openvswitch-agent eventually fails listing high count of ports on network node.
Summary: neutron-openvswitch-agent eventually fails listing high count of ports on net...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: zstream
: 11.0 (Ocata)
Assignee: Daniel Alvarez Sanchez
QA Contact: Toni Freger
URL:
Whiteboard:
Depends On:
Blocks: 1558336 1579300 1579352
TreeView+ depends on / blocked
 
Reported: 2018-05-07 16:49 UTC by Miguel Angel Ajo
Modified: 2020-10-26 11:59 UTC (History)
14 users (show)

Fixed In Version: openstack-neutron-10.0.5-6.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1579300 (view as bug list)
Environment:
Last Closed: 2018-07-18 08:34:44 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 566912 0 None MERGED Improve DbListCommand operation from O(n^2) to O(n) 2021-02-03 11:38:12 UTC

Description Miguel Angel Ajo 2018-05-07 16:49:36 UTC
Description of problem:

On normal deployments with 3 network nodes and l3ha, when the network nodes go beyond a certain threshold of ports, neutron-openvswitch-agent starts timing out on ovsdb access, which makes the agent unable to make progress and tag new ports.


/var/log/neutron/openvswitch-agent.log:2018-03-05 10:14:37.026 290346 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent TimeoutException: Commands [DbListCommand(if_exists=True, records=[u'tapb8abcb46-31', u'ha-0c2da070-16', u'qr-71af5638-5f', u'ha-17bc25d4-bb', u'qg-2ce9750f-72', u'qr-343d5f52-4b', u'tap4023bd45-91', u'ha-1071bc54-3a', u'tap70e27cfe-10', u'qr-df34d1b9-62', u'qr-c6bc0c0b-88', u'tapdfdd7639-72', u'ha-c73431e3-62', u'qg-ffb2c765-cd', u'qr-02b730e4-db', u'ha-2160efce-8e', u'tapc3027a86-7e', u'tap1fd4c936-db', u'qr-500805e9-83', u'qg-2c0ac496-c2', u'ha-1b4bc3aa-cb', u'tap879a5ddd-ea', u'qg-f95cef28-b0', u'qg-d616a4e8-3f', u'tape2252204-0f', u'qg-07c15093-cb', u'qg-77f30b3d-d5', u'qr-1b5fc273-40', u'tap75fdaeaa-5d', u'tap35c9b883-a2', u'tapf565414d-ac', u'tap2e495414-c6', u'qg-dc934b13-4c', u'qr-586db496-37', u'qr-ff46a78b-c8', u'qg-5cf1780d-b9', u'tap337ad3d7-2c', u'tap2cf588b2-a1', u'tap5cff235e-08', u'qg-52be0ea1-65', u'ha-1060ce4f-b5', u'ha-e36b0f92-5b', u'ha-a4af1b77-8e', u'qr-bb3bfae9-df', u'qr-cdd71cf0-65', u'qg-cc0b2f19-fe', u'qr-fc87ff2b-b4', u'tapbf6795c3-c2', u'qr-6a46ebad-8b', u'ha-008fee9b-ae', u'tapefee8c02-79', u'tapd16a1440-2a', u'qr-0b0be302-82', u'tap58819f37-33', u'qg-635022ab-51', u'tap8f4a0112-3b', u'ha-78b5ea12-28', u'ha-3af299b1-51', u'tap72d424e1-15', u'qg-99fb128e-f4', u'tapbd738a56-fd', u'qg-18365403-87', u'qr-b52525ed-8f', u'qg-a8605e1b-ad', u'ha-dcbaca51-21', u'qr-5a20f8ae-1b', u'tap6010f799-d6', u'ha-27817a67-57', u'qr-e5c4dc81-49', u'tapcf3546be-67', u'ha-70d6ee20-4b', u'ha-5624549e-93', u'ha-adc937cb-5e', u'qr-d49c5f88-7d', u'ha-71debbe6-72', u'ha-e28d7db9-49', u'tap3a16cdb7-16', u'qr-37037348-23', u'qg-a4747698-85', u'ha-b29e7a59-44', u'ha-9f5da4d8-59', u'qr-82af4d32-08', u'ha-b0b995a6-ad', u'qr-e94e38d8-a2', u'tapf271e762-7a', u'ha-5e82070c-b8', u'ha-dea6c4fa-cd', u'tapfe96dc9b-c5', u'qg-57b508ef-cf', u'tapfe75ad65-88', u'ha-925bde21-96', u'qr-5e615c28-9c', u'qr-ddfbceb9-8b', u'tape0fbf372-97', u'tapf707c03f-14', u'qg-9ae4a23f-b0', u'qg-1f696b34-bb', u'qr-b888a2de-8c', u'qg-004e9530-d8', u'qg-e8dbe1a1-b9', u'tapdd16fa68-52', u'tapadbdfdd7-c7', u'ha-3913b84d-02', u'tap57f0426c-f5', u'qg-39ba4958-71', u'ha-06594765-c4', u'ha-6b05f153-17', u'ha-fbe90a82-7a', u'tapfce3bcb4-e6', u'tapf4868c81-c1', u'qr-6ac5c2f9-9e', u'qr-6d897610-3f', u'qg-d866fa7e-61', u'tap821c1641-7d', u'qr-adc844c3-ec', u'qr-5bc7495e-a3', u'qg-500d1cd8-d2', u'qg-74993dba-c3', u'tapd8e0462f-1b', u'qr-27724ca6-37', u'tap6678b161-6e', u'tap82c07649-79', u'qg-1a046de8-78', u'tap56978e6e-d7', u'qg-71d87be3-cd', u'qg-af274a25-e6', u'qg-fa1ff07c-d7', u'ha-7f6f97ee-11', u'qr-08a821b9-59', u'tap82ef6f01-b9', u'tapcc04d41d-c7', u'qg-e50a64db-4c', u'ha-f26b98ea-9a', u'qr-e4ede30d-b4', u'ha-050fce08-63', u'qr-99c6dff5-39', u'qg-3c0285b6-01', u'qr-92eecf47-9a', u'tap578b379e-90', u'qr-fddd2b99-0f', u'qr-03340cfe-8b', u'qg-dbc25476-29', u'qr-afb62fb7-2e', u'qg-55d7ccf9-3e', u'ha-28977e7e-99', u'qg-50b15ddf-ce', u'tapcc75a176-98', u'ha-f0a6e8cb-61', u'ha-6a52abc3-2b', u'tapf7f00fdc-d1', u'ha-cf0dd5f2-33', u'qr-a753c2e3-a6', u'qr-154052a4-e3', u'ha-e415c361-75', u'qr-94566863-9c', u'qr-b1372ebb-59', u'tap41735fde-8c', u'tap60640f58-66', u'tapcdc0ae83-ab', u'qg-e1dd4e59-72', u'qr-b6dfd7b1-95', u'qg-918987c0-84', u'tapfc906c2a-64', u'ha-28da5f95-87', u'qr-f15e81a3-97', u'qr-374ee24c-5f', u'tapf34ae5b5-9f', u'ha-3bcbb2f9-74', u'tapa1cf8271-9c', u'tap8c2de1e4-7c', u'tap4a8f0725-14', u'ha-46081f69-3e', u'qr-fa9a296e-73', u'qg-9d94b8ac-09', u'qr-273da82b-9d', u'tapc84520b9-37', u'qg-b3ebb666-65', u'qr-56da2e06-73', u'qr-df225961-e9', u'qg-f7e05e2f-a3', u'qr-9aa9ad4c-85', u'tap95a7da9e-cd', u'ha-9cc3d2c2-50', u'qg-0d088ac7-82', u'tape10ee1d6-e1', u'tapea94e518-7d', u'qg-0aa83ace-0c', u'tapd28f82bf-fb', u'qg-86225cd9-c6', u'ha-dfedf61f-f7', u'qr-5cc5579b-1e', u'qg-4e3f6e20-12', u'qr-216dc21f-ca', u'ha-963205a4-9c', u'qg-1b797933-1e', u'ha-3b367554-f5', u'qr-8d5bc6a0-44', u'qg-e29c94da-fa', u'qr-81a06351-2d', u'qg-5ad63867-db', u'tap6c34669a-d8', u'qg-e3af69ad-15', u'tapb8ceae6f-be', u'tapef1399b2-b9', u'qg-bf1ab580-fe', u'qr-bc366b41-b3', u'qg-14ba3134-2f', u'qg-f6f80c1b-e3', u'tap4b99c0b0-cd', u'qr-f55f1971-f6', u'qg-463dffba-86', u'qr-cfbdcd6c-4b', u'tap98f8de1d-04', u'qr-014da039-50', u'qr-903f5a5e-70', u'qg-89567997-3f', u'ha-10d81654-09', u'ha-0ccbfdec-c9', u'qg-d8670208-b3', u'qr-7e270565-52', u'tapdee4b894-d8', u'tap6b2bcf4c-a5', u'qr-ebd851b0-ff', u'qr-202151e3-8a', u'qr-ec457d16-50', u'tapb464df02-d9', u'qr-81708e2f-ef', u'qg-013e89b3-fc', u'tap88190af9-27', u'qr-78a8534f-b2', u'qr-420e3bbd-d6', u'tap9373cdba-f9', u'qg-f33eb166-ae', u'qg-fd898d5f-9d', u'tap5f83ec2f-a8', u'qg-5dd037c9-cf', u'tap881fc7b7-29', u'tap874c94f7-45', u'qr-806d1221-24', u'qg-b96733fa-a0', u'tap33644482-4a', u'tapb20d819a-49', u'qr-f4161e97-d9', u'tapc75812b9-01', u'qr-1450dd77-25', u'qr-ef59a869-95', u'qg-dbe7c613-8b', u'qg-d5b3aba7-c7', u'qg-ff8da58f-ed', u'ha-28467daa-b7', u'qr-c024c5c3-d4', u'qr-0c3ef1f2-00', u'qr-f53a90ba-d6', u'qr-91b7e8bd-d9', u'ha-05f29c78-9d', u'tape33e6548-3e', u'qg-492495c8-8e', u'tap4a985893-f5', u'tapeb05e257-35', u'tap0773194d-17', u'ha-0f19974b-35', u'ha-85e2e945-2c', u'qr-0828b088-c3', u'qr-ffe3579f-53', u'tape3836900-b3', u'qg-661dbae2-c7', u'qg-e82643cc-78', u'tapdf795d1e-de', u'qr-72aaaa6f-90', u'qr-260d5b31-94', u'ha-67c22231-29', u'tap22b07aed-c7', u'qg-aafe52f3-1c', u'qr-5666274f-4f', u'qr-1622e607-dd', u'qr-cf6dbbf1-66', u'tapc4420858-27', u'qr-e0f0d2de-c0', u'qg-36c5ced0-88', u'tapba2db030-13', u'tap99bd95d7-a2', u'tap01427dac-a1', u'qg-cb137346-6a', u'qg-b3f7a754-bb', u'tap79b069ff-18', u'ha-52bd95da-55', u'ha-bef6b21a-9f', u'qg-be9aed2d-5d', u'qg-eac3c95a-58', u'ha-f49c5cac-aa', u'tapcf97811d-9e', u'tapd7481c01-60', u'qg-8ab63c9b-0e', u'ha-aecd10e8-af', u'ha-05c3a8e4-02', u'qg-bea6a362-bd', u'qg-3ae51829-35', u'ha-245d8576-20', u'tapa6d47791-52', u'ha-0f702ea2-73', u'qg-883e3a49-e4', u'tap84b793e7-66', u'tapa6b01736-c7', u'ha-4659105c-83', u'ha-0f40c095-6a', u'qr-d063d4dc-ce', u'qr-f346db3a-51', u'qg-09746231-8c', u'tap6beb830a-c5', u'ha-c2b1c1f7-6d', u'qr-7c894408-0b', u'qg-a2526eb7-da', u'tap3731b895-a8', u'qr-565190fd-7a', u'ha-d55d8ff1-3b', u'qr-f7cfacc9-10', u'qg-5c158e60-af', u'qg-ade295f2-1f', u'tape02d1f4e-0e', u'qr-a4c91c8a-4f', u'qg-346889b5-cf', u'ha-ff4cf398-b1', u'qr-8709dbea-43', u'qg-2247e777-b8', u'ha-854aaa26-c3', u'tapefefa302-e2', u'qr-039ed898-99', u'qg-36b9f98a-35', u'tap5dad0624-fe', u'qr-e463bf33-61', u'tap133dfed8-8d', u'ha-9f835048-79', u'ha-323ecf08-4d', u'qg-60dcaecd-79', u'tap6f19ce26-a4', u'ha-baa5114f-67', u'qr-204d2187-78', u'tap8b56cebf-e8', u'qr-d8054194-9c', u'qg-518f70b1-aa', u'qr-8a8f3c92-e0', u'tapc83773b7-37', u'tap31c9d0df-d6', u'tap2e204e94-32', u'qg-3753c19c-be', u'ha-186aaae6-9e', u'ha-57edea5f-e5', u'qg-2e8f2dd9-aa', u'ha-39602179-31', u'qg-ef50132e-67', u'tap6a427fc7-17', u'qg-9dd41778-45', u'qg-73d6fb3b-e3', u'qr-06391ad2-f1', u'tapf28f1d03-f3', u'qg-5daf8908-12', u'qr-3e1aeb73-ab', u'tapb1bd7acf-cc', u'tapac527bad-02', u'ha-7b9bf02c-41', u'qr-822df584-6b', u'ha-b3a7717e-56', u'qg-8c72ce6a-37', u'ha-83dcb474-be', u'ha-c0ec2e73-fc', u'qr-319f89aa-8b', u'qr-262ab449-98', u'qr-c4b82e9c-17', u'qg-7db1e5bb-10', u'tapb53a4468-cb', u'qg-c254cef3-d5', u'ha-a79994e8-6c', u'qr-b71184fe-b7', u'qr-cba3bab6-d2', u'qr-d7be7709-d5', u'ha-7d8eea69-a6', u'tapd3ced7ea-32', u'ha-7681e56b-45', u'ha-7446cd95-07', u'ha-14476ff8-26', u'tapb936f8f8-18', u'tap77b932a5-72', u'tap4979940c-73', u'qr-4d4422a3-47', u'tap97d675d9-47', u'ha-85ed7db3-d9', u'qr-ce8d9287-e2', u'ha-df8d93ab-28', u'tap7d731c1f-23', u'ha-31b696aa-ea', u'tapa1042ba8-b6', u'tapd4491ced-d3', u'tap97c5d6e5-3d', u'ha-007e06e5-b3', u'tap47c66fa2-c4', u'qg-053d3deb-34', u'ha-62b3add2-d8', u'tap72bc0a73-57', u'tapc30361ac-c7', u'qr-10052fed-7b', u'qg-2b9ce25c-5b', u'ha-a9601f0c-ea', u'tap73a736ba-82', u'tapa8023f54-97', u'ha-0edccc44-12', u'qr-99e23015-3d', u'tap9c634d47-15', u'qg-5e425729-98', u'ha-9d78962c-ee', u'tap9276e59f-6a', u'tap3af620a2-91', u'qr-a56fe2dc-92', u'qg-d3c79a2c-5c', u'ha-1cd4323e-32', u'qg-887abc8c-b3', u'tap3e1b693a-2b', u'ha-6dc67817-5a', u'tapf878780c-83', u'tapa96fd071-f9', u'tapaf946194-e7', u'qr-13ed510d-78', u'tapefb6fb95-2f', u'qg-8969a0e0-a6', u'qg-518c018e-87', u'qr-4e831438-a2', u'qg-fa32d20d-ab', u'qr-2b6b615d-71', u'qr-97f3d69d-4b', u'tapc6cc5faf-3c', u'qg-4f00ff4c-76', u'tap9c15f262-09', u'tap8cda620f-9b', u'qg-3d45af3d-0c', u'tap7e9db226-63', u'qg-a188e9e0-74', u'tapfdc0f37c-c9', u'qg-e3cd5b37-a8', u'qr-c40e6383-ce', u'qr-baecc6ee-d7', u'ha-3c4b90c8-b4', u'ha-0e7cc454-a8', u'tapf8411537-77', u'tapf8cbd585-65', u'qg-eb4cf206-c6', u'qr-dead7458-96', u'qr-2717fa3a-b8', u'tape630bd08-ac', u'qr-505f6988-80', u'ha-bd6d33a0-ed', u'ha-4b1e577d-7d', u'ha-6baea9d5-ec', u'tapcaf3c580-de', u'tapdec008c7-9a', u'qr-df7d6daa-33', u'tapbcbbf510-49', u'tap20b8341c-f3', u'ha-ec40e8df-2b', u'qr-a384b6d8-a0', u'ha-5b3f4706-85', u'tapd4be8a1a-ac', u'tape2324dbb-1d', u'ha-b3f133bd-05', u'qg-c7acef20-44', u'ha-ee983ae5-88', u'tap80b86bfa-0b', u'qr-da1f7267-13', u'qg-2e2a6263-59', u'qg-562dbf09-01', u'tap8c4dbaac-ac', u'qg-2ce0fc25-9f', u'tape0c7d412-47', u'ha-50815185-e8', u'tap648f840c-b4', u'ha-877e509d-10', u'qr-5bb72098-52', u'tapdc09710a-0a', u'tapd0536226-42', u'tap62c4e40c-ce', u'qr-1b4348f4-03', u'qr-f1ccce80-8f', u'qg-66512f8c-bf', u'qr-22e1a700-e6', u'tape8e4dfb2-97', u'ha-086e981f-e3', u'qr-283b3cf4-97', u'qg-bce44ea4-ad', u'tap4748f072-fb', u'qg-299309aa-1a', u'ha-95d6aeef-f7', u'ha-5b4ea389-95', u'ha-7dd0a13f-6e', u'ha-0f30620d-ce', u'tapd8afa05a-28', u'qg-0d2cd525-35', u'qg-d40f8261-c4', u'ha-c065496d-b9', u'ha-6da7c7b6-01', u'qg-3d0b782b-b5', u'ha-82a46fb4-75', u'qg-03fc3b24-1c', u'tapde96c913-ad', u'qr-6c63ad3c-cf', u'qr-86a1af27-c5', u'ha-c5f1e384-42', u'qg-bac77d84-4a', u'qg-30aff087-24', u'qg-9e587924-70', u'tap10ced7de-23', u'tapaf576cfc-73', u'ha-8cc0ceab-d4', u'qr-26b74630-bc', u'tape0279984-a2', u'ha-e74292d4-2a', u'ha-66880207-66', u'qr-ebf154e9-2e', u'ha-7867a544-14', u'tapdf80be33-44', u'tap5fb86bf8-04', u'qg-909026c8-01', u'qg-522c1b78-c6', u'qg-167561a2-dc', u'qr-bed2de72-a2', u'qg-3ecfbd90-f2', u'tap7713324a-75', u'ha-74a2fcb2-3f', u'tap0ca44143-92', u'tapcdd77472-ac', u'tap564852c8-4f', u'tap760dc4e3-c6', u'qg-a7b6ab39-76', u'ha-3c0ffe5d-e4', u'tap4988b335-e0', u'tap964b15d6-1a', u'tap872c30c9-79', u'qg-63a89d23-a1', u'tap4c0442e1-28', u'qg-a70955b4-11', u'qg-7919067b-02', u'tapd96517b4-df', u'qg-61675033-30', u'qr-f0153a4b-a7', u'ha-1ca62cfd-02', u'qr-d424a7af-1d', u'qg-19e371f3-d5', u'qg-799a5ddb-15', u'qg-f9921c3a-0a', u'ha-e4ac11fa-45', u'qg-f12434d0-65', u'tap7a67911b-e1', u'qg-27dab0a5-50', u'tapf18a71b0-8e', u'ha-6a1ccb5d-41', u'tapd9cc0d82-ba', u'qg-b11dde45-a1', u'qr-8c195d4c-f3', u'ha-1b2b3843-41', u'tap80d514cc-30', u'qr-8b704aec-5f', u'qg-68d3a8eb-c9', u'tapdecb818e-2c', u'qr-7d230b3f-ac', u'qg-9460792e-eb', u'ha-548b372b-09', u'ha-ae80909f-ee', u'ha-6beb3ffc-f3', u'tap94c72b29-5e', u'tap6981ff73-ed', u'ha-71c0ac28-48', u'ha-10d120f3-43', u'tapf073ddec-4f', u'qr-a489c069-4e', u'qg-b95d1f67-ec', u'tap827251c4-8c', u'tapc9161970-2c', u'ha-47841b61-b4', u'ha-dc2cc428-40', u'qr-8652764c-f6', u'ha-38dd6504-88', u'tap614e1cb4-23', u'ha-67d20abd-6b', u'ha-d33da7fc-87', u'qr-19dc219e-61', u'qr-c0fcc12e-08', u'tap42c2272d-ab', u'tap9869b000-32', u'tapb203539a-d7', u'qr-48044243-3b', u'tap2cdc2ec3-90', u'tapbf6e7a1a-e2', u'qg-76be07cb-a9', u'tapa338ef3a-a5', u'tape5fe0d43-0c', u'qg-9b189608-a7', u'qg-e6e73f13-4d', u'tap3232e9f2-aa', u'qr-13858731-9b', u'qg-269f48c0-18', u'qr-172a667a-0a', u'tapf7563234-66', u'qr-e0dd0bc6-cb', u'qg-4489118a-cb', u'qr-41f2080c-07', u'ha-2ad7f5f6-90', u'ha-27508618-ed', u'tapd276977e-f0', u'tap74e59911-74', u'ha-6a2f1495-84', u'tapdb10a1b2-3f', u'qg-16427fcd-6d', u'tapb2274353-86', u'tapfee729da-9b', u'ha-e9f170e4-98', u'qg-73c4fa87-74', u'qg-14696152-03', u'ha-925da4ae-5b', u'tap605b1752-3d', u'ha-18cd0c14-05', u'qg-de368a13-28', u'qg-8b510324-d5', u'qg-7f8fb44d-68', u'ha-0659f802-86', u'qr-eaf4381e-fe', u'tapee73bac7-3e', u'qr-8898b8f7-ca', u'ha-2d8fca3a-b1', u'tapee7c791b-f7', u'ha-df764eb9-ee', u'ha-003110b4-d4', u'ha-688f746a-50', u'ha-0d4dd8e2-4e', u'qr-e17b92f4-60', u'tapbea6b766-0c', u'tap988747a5-dc', u'qg-2d541e31-8d', u'tapc8163747-fe', u'qr-8ba43e8c-7b', u'qg-ad006937-54', u'qr-b33c8a85-20', u'qr-816d325c-c2', u'tapee478bd6-c8', u'qg-64c95f5a-91', u'qg-a3263633-95', u'tapb4315c78-9f', u'qr-c5689a27-05', u'tap57c92d23-a7', u'qg-7069ea6a-48', u'qg-7c062816-11', u'ha-28ec231a-67', u'qr-5e1285ed-91', u'qg-e73963ff-84', u'qr-0ae3f22e-37', u'qr-98589de7-6e', u'qg-3b6cb618-b6', u'qg-df24c083-db', u'tap8f114b1a-97', u'ha-36a32ac9-48', u'tapbace27ed-31', u'qg-45e162ee-e2', u'tap4cda729b-66', u'tap50459348-39', u'qr-abbe7256-c7', u'qr-87acbf42-63', u'tapaef64a39-05', u'qr-58d47fc6-9b', u'qr-d7b705ec-60', u'tapd6717d86-26', u'ha-29d214ff-b1', u'qr-74a53230-32', u'tapdf7a65b6-f0', u'qr-42a8f906-a1', u'ha-3e7e3c4c-aa', u'tapdff0853f-aa', u'tap8366f504-fa', u'qr-152d5a66-1d', u'qr-ad5ebdf4-98', u'qg-04710968-4d', u'ha-416e2ce2-aa', u'qg-5546e229-5e', u'ha-daf4a37c-78', u'qr-e01e0d07-f6', u'qg-cf9c9b7e-86', u'ha-913984a5-44', u'ha-0e4264f2-d3', u'tap43134c9b-88', u'qg-b227dfba-9f', u'ha-cee077ed-c8', u'tapea211058-c3', u'tapf15a18df-82', u'tapc05baa61-d0', u'tap7973c638-0d', u'qg-0c02e6fa-77', u'tap2160c6c5-26', u'tap8f5713bc-21', u'ha-9e9cc2e7-46', u'qr-a28d9d4a-5b', u'int-br-ex', u'qg-fabef145-87', u'qg-03102420-45', u'tape5d9a886-26', u'tapa47bfecd-f6', u'tape7639f4b-9c', u'tap6836f620-c2', u'qg-d799e982-1e', u'tapbe7fa307-74', u'tapf4f942c9-eb', u'ha-87a5fb82-79', u'ha-e1555ea5-c8', u'tapfbf4ba33-89', u'tap911940da-0d', u'qr-73fb9aa4-71', u'tapc5323817-29', u'tapcea9c4a0-6b', u'ha-ce9a2fe2-ab', u'ha-db9d2181-75', u'qr-75cbe2eb-35', u'tap2039eab3-cb', u'qg-3ec6fa4f-25', u'tapeda179bf-28', u'ha-c039321d-b6', u'tapa73d24f0-a6', u'qg-c7d50571-76', u'ha-8fecba40-07', u'ha-a6889635-8c', u'qr-3bd6c35a-10', u'tapb1e923e8-ce', u'qg-c568bfa3-82', u'qr-dfa93ad4-b7', u'qr-1a0c058e-25', u'qr-ffd7364c-cd', u'ha-43373ab3-48', u'tapf432e2c8-cb', u'tapc7219b50-0b', u'qr-28b14b57-f1', u'tapaf23db46-2e', u'qr-2216588d-f7', u'tapcc9422f2-43', u'qr-f2112d7a-d9', u'tapebad9026-98', u'qr-98d59a3e-22', u'qr-8736d9c6-45', u'ha-39f59b8a-89', u'ha-7910bc10-38', u'tapc0d6826e-c5', u'tap6e3d54a9-3c', u'qr-cf8cb1f6-86', u'ha-431d35e4-8c', u'ha-473ec767-bd', u'qr-76f8d5b3-46', u'qg-f1142251-1b', u'tape8d86564-51', u'ha-afafd8e3-78', u'ha-1924dca3-60', u'qr-6d40c0c2-ad', u'qr-d3a858d1-00', u'qg-7f4f8e32-b7', u'qr-3f2fee05-cf', u'tapca263fd6-7a', u'ha-24dfb4af-2e', u'ha-145fef73-a2', u'tap98e4e0e5-1d', u'qr-091839ef-a1', u'tapa3eae933-1e', u'qr-ab71e49f-84', u'qg-d1a39f21-30', u'tapd0c33d2c-3b', u'ha-2a660c03-b1', u'tapd2bd3682-7e', u'ha-cb64712a-4e', u'tapafeb1bd1-ac', u'ha-d0835129-34', u'ha-8d45ced5-cb', u'qg-38d0ed2c-0c', u'qr-65f3d315-41', u'tapfc0a51d2-bd', u'ha-1acb7228-e5', u'qr-d282ade4-13', u'tapc3c56a3e-20', u'ha-878a22a9-12', u'ha-de2f3b4a-0e', u'tap7eee645d-ba', u'tape9c67667-3b', u'qr-4060c911-ed', u'tapdea6f3d1-74', u'tapb8ddf996-d8', u'ha-cc2a5ae1-41', u'ha-87be50c7-2c', u'tapaed49c65-14', u'tape853db51-27', u'tapc4f3e72c-5a', u'qr-aeb4db8d-55', u'tap6567df5b-66', u'qg-c2c42ce9-46', u'qg-2dbefdf9-8e', u'tape10d6f70-43', u'tape0ac74ca-c3', u'tap760ec3b3-43', u'tap98cd1e5c-c0', u'qr-2f29c796-df', u'qg-ba5c0067-2a', u'tap94ca2a51-07', u'tapaa5a89d1-61', u'tapab85b31e-2c', u'ha-db764523-a5', u'qr-3a1fd94e-6e', u'tapec9cbdc9-dd', u'ha-8c297b78-2b', u'qg-24de00c3-7a', u'tapef72681e-ee', u'qr-cb23f37a-90', u'ha-45f585c3-78', u'tap5fb89351-ce', u'qr-919eb73f-ef', u'tap73103e9d-c5', u'qr-889835bf-01', u'ha-54d9cbae-2f', u'qr-15b44cfe-cc', u'qr-c2e8e132-d9', u'tap3509518c-e4', u'qr-f5813e27-57', u'ha-3efbd57d-44', u'tapb3ebbe76-18', u'qg-343d9980-5d', u'qr-f9dac4b2-6f', u'tap87bafbc1-40', u'ha-f3974557-6a', u'ha-f8a8ded2-c2', u'tapfaa0b48a-4b', u'qr-5d595687-3b', u'ha-9987c93f-63', u'qg-27c8cc40-51', u'qr-8eff6080-22', u'qr-4a638f62-fb', u'tapb47f3be3-2b', u'ha-f80bc502-81', u'qr-7b21d3ca-89', u'qr-f43e684e-3e', u'qg-7a2c935c-fa', u'qr-6421a47e-31', u'qr-5e6b9954-64', u'qr-4a990986-e9', u'ha-12c0baf3-be', u'tapb044793e-b6', u'ha-f263d6ee-e7', u'qr-71febbca-1f', u'qr-dd29a3aa-67', u'tap3ff11699-d1', u'qg-632471a7-a8', u'qr-f7d35ee7-b9', u'tapb3126cdd-1c', u'ha-ea1fc67c-a1', u'tap77b73537-a3', u'qg-7ca38ffe-42', u'tapd5d00dcb-53', u'tap607e38d1-e8', u'tapc7f6df38-06', u'qr-c4af1d41-e6', u'qr-f2b04697-f5', u'tap94e1f3c4-1f', u'tap8657c75d-e7', u'tapd65dfc76-a5', u'ha-b5b296bc-c6', u'qg-db9235dc-38', u'qr-531254f8-9d', u'ha-f4dfb9c5-ff', u'qr-616f4d55-e9', u'tapd54a2880-ba', u'ha-abd56b60-bb', u'qg-12a85376-f0', u'tapb6b0a773-f1', u'tapcc236310-b5', u'qr-c5da4687-dc', u'ha-2fdae2da-e7', u'ha-47646498-06', u'qg-493827e3-1c', u'qg-c5530c7f-86', u'qg-37d5fdd8-53', u'qr-085627c4-d8', u'tap2b771c96-3e', u'qr-2f527767-6f', u'ha-c7d62fe4-f8', u'qg-57bd364c-c9', u'ha-1e596e22-54', u'ha-224e6966-31', u'tapf6e05193-40', u'qr-05e8fe0a-cc', u'qg-3d6f5ac3-43', u'tapbb97d983-4d', u'tapaf3f563d-a5', u'tap432a2f13-37', u'tap7bfb1493-2c', u'tapedcf0615-6c', u'ha-7dc8bfc4-7e', u'ha-d7727d9f-23', u'qg-c86159d2-ee', u'ha-23cec660-ba', u'ha-cdd9619a-9d', u'qg-c062e43c-0a', u'qg-6ab8a3d2-fe', u'ha-4405ecd3-c9', u'qg-5f25feab-88', u'tapa557bb0c-d1', u'ha-26640385-7b', u'qr-cebf3621-d5', u'ha-7982a0e1-80', u'qr-ff4970d7-35', u'ha-1ec21102-5f', u'ha-1a6d1a32-ee', u'tap34322086-29', u'tapc62740ba-de', u'qr-0741548d-a0', u'tap89828637-7c', u'ha-d95934c1-f8', u'ha-fd318772-8f', u'qr-48e76f18-f0', u'qg-fd213fea-11', u'qg-f1a5706c-07', u'qg-58e77f08-ba', u'qg-6003fbeb-8f', u'ha-4ca363c2-b6', u'qg-d3275773-e7', u'ha-241f3028-41', u'qg-c892e805-2c', u'ha-63a5ccf5-f7', u'tapd6395f5a-da', u'qg-067b0688-8e', u'ha-efc4c47e-8b', u'tapcf20ab65-1f', u'qg-6770f5d4-85', u'qr-a8761176-54', u'ha-7b292795-62', u'ha-65408422-5a', u'qg-cde41b4d-6b', u'qg-10b5b37b-82', u'ha-2f592a98-b6', u'qg-1b4c2d2c-62', u'ha-1b9fa1a4-ad', u'ha-6e5b3dca-d5', u'ha-a6efc065-79', u'tapc8be5c19-3c', u'qr-1d94d9db-96', u'ha-fd32e29a-96', u'ha-d184d9ce-ae', u'ha-22f7f20b-5b', u'qr-ad2953bf-d4', u'qr-1be76948-0a', u'qr-f7421672-0f', u'ha-25f824bb-47', u'qr-1d4ac604-0a', u'ha-77c98bc7-d7', u'qg-436577c3-57', u'qr-bd123811-56', u'tapd55cc1f5-c9', u'qg-b63cdff1-89', u'ha-2244e984-96', u'qg-6d92e626-68', u'qr-acb80252-5d', u'tapf4da7480-36', u'qr-2bfd53a9-bd', u'qg-9fcb7d2d-44', u'ha-2a1a9865-16', u'tap34a2cf2e-73', u'ha-71086091-c5', u'qr-7dc49f35-d9', u'ha-113a852d-2b', u'tap84d78424-48', u'ha-e630a0d8-cf', u'qg-fbc269f3-71', u'ha-fe3861e0-6a', u'tap784e34b2-06', u'tapdb4af922-ad', u'tap93470629-b4', u'qg-6399481b-df', u'tap74bf14dd-c6', u'qg-279b7568-0f', u'qr-ff0b750b-dd', u'ha-2e77551f-57', u'qr-7360fa81-13', u'tapcfae0dbe-a1', u'qr-2e98849d-44', u'ha-10c7e57e-7f', u'qr-5b69d9a6-d6', u'ha-f7de8c1d-0d', u'qr-b3e9f17a-83', u'ha-147e8028-ad', u'tap91824a4b-af', u'tap97248ed6-e2', u'ha-7901ca35-be', u'qg-63122f0a-21', u'qr-2f8eec4c-52', u'tapa8a04048-a9', u'ha-3f6689b8-a5', u'ha-19f1d0ef-cd', u'qr-00af5051-72', u'qr-3bfd8dc9-f2', u'tap4cd93445-0e', u'qg-7fab33b1-3b', u'qg-6fc2f96e-ae', u'qr-68dfe075-14', u'ha-3186206f-19', u'tap24b87642-94', u'qg-195d6e21-38', u'qr-8152bad1-ac', u'qg-bed94c60-aa', u'qg-c1417af7-7b', u'tapec84a707-2a', u'tape1daac98-17', u'qr-63c25861-3c', u'qg-9a1a2350-49', u'ha-d9d5329f-5a', u'qr-cfba12ee-3b', u'tapcf958119-02', u'ha-787f3c33-a5', u'tap5f498a39-de', u'tap1d239fb5-52', u'qr-32083891-5c', u'ha-14948d9d-e4', u'ha-ddb07d93-cb', u'tapef4f9255-22', u'qg-c135fb69-44', u'qg-db40b33a-41', u'ha-51b184c4-db', u'tapcc21fed1-57', u'tap91adb685-c6', u'tap8fead629-3b', u'qg-c52d93af-aa', u'qr-454a8d75-cf', u'tap9f2cf581-03', u'qg-3d72b46b-07', u'tapbe5e8c95-88', u'tap742c0067-09', u'tapc209c158-45', u'ha-b3a88b74-cf', u'qg-82d622bb-08', u'qr-f20686d0-8a', u'qr-0b007521-73', u'qg-b8776f4b-39', u'ha-ca3da720-50', u'qr-81b480d1-97', u'tape76d4a39-70', u'tapee25c1fd-51', u'ha-6b89fce0-3a', u'ha-6ff0c5ac-22', u'qg-88f0a4a5-00', u'qg-5da4390d-d2', u'qr-1ab07321-ee', u'qr-09227e20-23', u'qr-d89e5257-28', u'qg-4fd4639f-a0', u'ha-8c01701b-36', u'tapb7aa2b24-7f', u'qr-c9f20f74-00', u'tapb4de85f3-fb', u'tap078dc47f-1a', u'tape2324abd-cd', u'qg-dce06458-02', u'tap430acf03-f2', u'qg-eacab584-70', u'qg-4242552b-ac', u'qg-85880a14-45', u'tapf027cf42-a0', u'qr-1b333a0b-78', u'qg-ecef7ad5-0e', u'qr-5cf0105a-e2', u'qr-bb748180-2b', u'tapd56babec-c1', u'tap554239bc-79', u'tape76fdb73-5e', u'tapfb25ce17-cb', u'qg-af0d8445-a8', u'qr-b4092df3-f4', u'qr-707f7cdd-c2', u'ha-30e5cab9-1a', u'qg-adbf9827-85', u'qr-db3f4677-18', u'ha-dfcb87f3-c3', u'tapdf7438b9-aa', u'qr-b76d49cb-4f', u'ha-75670f58-76', u'ha-b636dde0-bb', u'qg-638d497c-ad', u'tap732022e9-7f', u'qr-582a7444-68', u'tapf2c31377-d5', u'qg-c62ef8d2-c2', u'qg-177ffc23-09', u'ha-ecfcc8b4-85', u'qr-2daddb32-1d', u'qr-42ee6c84-c8', u'qr-438d4a1f-96', u'qr-db6ab05e-97', u'tap10349e24-15', u'ha-ee60f0e9-1d', u'tapbb10a799-d5', u'qr-d3e834ea-32', u'qr-268f0830-db', u'qr-d1d2924a-d4', u'qg-afd313b8-24', u'tapbc3f3102-0d', u'qr-7af77b08-57', u'tap695d0d75-59', u'qr-7c20436d-91', u'qr-4d531d96-1d', u'qr-ef5dcda1-4b', u'qg-56000ae3-13', u'qr-355d4af1-3c', u'ha-947e6ba3-62', u'qg-944e0880-f7', u'qr-1a014831-b9', u'tap3b326421-b5', u'tapd41581a4-12', u'ha-0154fcca-66', u'qg-25d6c8a1-1b', u'qg-7562cdc0-6b', u'ha-7f44a8cd-6d', u'tapfd3cdb92-c3', u'ha-f2030d52-08', u'ha-60d45df0-99', u'qr-0a33c67d-15', u'tap9b408343-c4', u'tapc2ed1a3d-69', u'qr-72e244fc-86', u'qr-1bb7d6d1-cd', u'tap99f7372f-fb', u'qg-7ca04545-9d', u'qg-846232c1-a7', u'tap21025362-fd', u'qg-c2357c8d-c6', u'ha-b247157f-63', u'qr-553588fe-a1', u'qg-31e5e69e-75', u'qr-633f2fc4-b9', u'qr-19bd9ecb-ea', u'tapdb8b73a3-25', u'tapbdea7f37-ad', u'tap727fda2e-5b', u'qg-1ea09a52-72', u'ha-9c6a0085-9a', u'qg-b36002fe-81', u'qg-16e4fdc2-41', u'qg-a5555414-dc', u'tap7451abf6-39', u'ha-b432399b-85', u'tapeb7e454b-9b', u'ha-9c79afd6-7d', u'tap90669746-08', u'tap5860f545-da', u'qg-0326fe70-e7', u'ha-ba974b02-99', u'tap999de4d4-c4', u'qg-e5dad208-30', u'qg-3cd1bdf9-19', u'qg-8cfb0d5e-35', u'tape81cc67f-f1', u'ha-a6c8134a-69', u'qg-7145ec0e-24', u'ha-521e85a4-84', u'tapd0fce789-33', u'ha-7993659c-3e', u'qr-0f8031e6-04', u'tap587b4600-e8', u'tapc4c98ed0-bb', u'qg-660539ce-fc', u'tap36cd94fd-95', u'patch-tun', u'tapf6153953-f9', u'ha-0f74017b-f8', u'tapc04c8746-16', u'qr-cbf3e63f-1b', u'qr-6cdec5bf-84', u'tapc1d7df36-9d', u'tapf52fb5b8-23', u'qg-0c7a386f-33', u'qr-a371d4a8-e7', u'tape3154ca5-48', u'tap7437c9dc-75', u'ha-530acd8d-a1', u'qr-b8c19505-6b', u'tap91f18ba0-22', u'tapabf2ba78-bf', u'tapfb080bc5-3c', u'ha-a36afe30-81', u'ha-68772f82-04', u'qr-ff9d32a5-e0', u'tapd3053e27-ea', u'qr-d38eaa5d-e8', u'qg-91b1b98e-9e', u'ha-10dc123a-51', u'qg-618368e8-1d', u'ha-3782f425-a0', u'tap09b9396a-04', u'qr-f2863558-f6', u'tapf7f7d555-29', u'ha-8c312188-29', u'qr-69786919-c8', u'qr-f54efeab-32', u'tapb693e242-9d', u'tap0077b381-d6', u'qr-058e8c50-f6', u'qr-1bbbb7a3-e7', u'qg-8d18de2d-19', u'tap960a83a3-8c', u'qr-81b1d893-0c', u'tapeb21f332-e3', u'qg-5712489d-a6', u'tap90a85ac0-01', u'tap5a2e6b0a-3c', u'ha-c3fca98e-55', u'ha-ba48a073-94', u'qg-6fd055d2-6d', u'qr-fe622863-c8', u'tapdc5dca6f-1f', u'qr-52e7795e-da', u'qr-ca5b14dc-08', u'qr-056c51e7-1d', u'tapb2d8ffaa-85', u'qg-5fa720f8-8e', u'ha-47de6474-aa', u'tapd06455e8-37', u'ha-de7439a9-78', u'ha-ed24d201-54', u'tap26370c38-f4', u'ha-23a26bba-dd', u'ha-9f596aa2-0b', u'ha-95d257dd-a2', u'qg-b581f25e-31', u'ha-c208b1d1-1d', u'ha-af671567-6f', u'ha-e8d97203-60', u'qr-f6f5ac70-00', u'ha-8fb94815-e2', u'qr-5db3e226-84', u'ha-b4cd1427-96', u'qg-68d81028-61', u'tap2a9ac01a-f0', u'ha-1277c7fb-3b', u'qg-8734e29d-d6', u'qr-21d99183-3d', u'qr-fda658db-08', u'ha-a1a75374-a2', u'tapc50fb07d-bd', u'qg-4073bac5-ea', u'qr-49c4b0e5-fa', u'tap2a1344d2-2c', u'qr-d22db034-3d', u'qr-78b60ebd-c3', u'tapadde1759-56', u'qg-66b9855d-d8', u'qg-ef1395bc-01', u'qg-3aac0a43-49', u'ha-44520c25-1c', u'qr-c49523ff-0d', u'tapc2593ff9-e1', u'qg-d9887277-78', u'ha-26694fc0-8e', u'qr-1db2f9ec-15', u'tapa5190a62-d9', u'tap7d3abc8d-ef', u'ha-6de1944d-f9', u'tap6c8b8003-7d', u'ha-673bb1a0-c4', u'qr-ff4b39e5-d5', u'tap8e8371f4-0f', u'tape4e1733b-c1', u'tape5326b83-e0', u'tapd4133d76-80', u'qg-97c202d3-8b', u'tap55673919-77', u'tap869ee6d4-ee', u'tap24f5d2b8-f9', u'tap6e841d3b-6d', u'tap79dcfee1-36', u'ha-57dd81f9-1a'], table=Interface, columns=['name', 'external_ids', 'ofport'])] exceeded timeout 10 seconds


I don't have the full picture, but Terry Wilson started looking at this, and he provided me with the following details:


-  DbListCommand also passes in the 1600 ports that it is individually listing. 1600 results, but it is probably looking up each individual list. Probably something where we could make a custom command that doesn't really follow what ovs-vsctl did to improve performance. (like with the ovs-cleanup delete command we added)

- DbListCommand is going to be O(n^2). It looks up each record by name for each record passed in.

- It would be better to just grab all of the ports and filter out the ones we don't want.

- and it is even worse, because get_vif_ports is called to get the port list that is passed to get_ports_attributes() and get_vif_ports also calls get_ports_attributes.... ( though get_vif_ports looks the info up via Interface table, and get_ports_attributes in ovs_neutron_agent usually called to look up info based on Port table. ) 

- The whole VifPort object should really be changed to just store all the data from the port.  we pull back the vifs from the db, then use their names to to look up individual info about them

Comment 1 Miguel Angel Ajo 2018-05-07 16:52:56 UTC
Note, this is also likely to be an issue with OSP12 too, may be other earlier versions too?

Comment 3 Daniel Alvarez Sanchez 2018-05-09 07:55:06 UTC
I tested this on an OSP11 setup with the following puddle:
core_puddle_version
2018-02-08.3

OVS agent: openstack-neutron-openvswitch-10.0.5-1.el7ost.noarch

* Created 2000 internal OVS ports:
[root@controller-0 neutron]# ovs-vsctl list interface | grep uuid| wc -l
2018

* Restarted OVS agent and we can easily see timeouts when listing the ports:
2018-05-09 07:44:50.370 746921 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change do_commit /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py:124
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands [req-fe482d7b-276e-45fb-a6f4-be82a04ccbdc - - - - -] Error executing command
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands Traceback (most recent call last):
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py", line 36, in execute
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands     txn.add(self)
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py", line 79, in __exit__
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands     self.result = self.commit()
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 73, in commit
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands     'timeout': self.timeout})
2018-05-09 07:44:50.373 746921 ERROR neutron.agent.ovsdb.native.commands TimeoutException: Commands [DbListCommand(if_exists=True, records=[u'tap-919', u'tap-1263', u'tap-439', u'tap-803', u'tap-320', u'tap-741', u'tap-87', u'tap-764', u'tap-729', u'tap-1623', u'tap-1024', u'tap-47', u'tap-526',....
] exceeded timeout 10 seconds


* Now I apply this patch https://code.engineering.redhat.com/gerrit/#/c/138055/ .
  Restart OVS agent and look for DbListCommand traces in logs:

2018-05-09 07:50:32.914 769810 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbListCommand(if_exists=True, records=[u'tap-919', u'tap-1263', u'tap-439', u'ta
p-803', u'tap-320', u'tap-741', u'tap-87', u'tap-764', u'tap-729', u'tap-1623', ...
, table=Interface, columns=['name', 'external_ids', 'ofport']) do_commit /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py:100
2018-05-09 07:50:33.079 769810 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change do_commit /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py:124


Conclusion:

The patch linked made the DbListCommand operation to take from >10 seconds to around 0.15 seconds. This clearly improves performance but also avoids other side effects of the timeout of this command.

Comment 8 Scott Lewis 2018-06-22 12:36:41 UTC
OSP11 is now retired, see details at https://access.redhat.com/errata/product/191/ver=11/rhel---7/x86_64/RHBA-2018:1828


Note You need to log in before you can comment on or make changes to this bug.