Description of problem:

On normal deployments with 3 network nodes and l3ha, when the network nodes go beyond a certain threshold of ports, neutron-openvswitch-agent starts timing out on ovsdb access, which makes the agent unable to make progress and tag new ports.


/var/log/neutron/openvswitch-agent.log:2018-03-05 10:14:37.026 290346 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent TimeoutException: Commands [DbListCommand(if_exists=True, records=[u'tapb8abcb46-31', u'ha-0c2da070-16', u'qr-71af5638-5f', u'ha-17bc25d4-bb', u'qg-2ce9750f-72', u'qr-343d5f52-4b', u'tap4023bd45-91', u'ha-1071bc54-3a', u'tap70e27cfe-10', u'qr-df34d1b9-62', u'qr-c6bc0c0b-88', u'tapdfdd7639-72', u'ha-c73431e3-62', u'qg-ffb2c765-cd', u'qr-02b730e4-db', u'ha-2160efce-8e', u'tapc3027a86-7e', u'tap1fd4c936-db', u'qr-500805e9-83', u'qg-2c0ac496-c2', u'ha-1b4bc3aa-cb', u'tap879a5ddd-ea', u'qg-f95cef28-b0', u'qg-d616a4e8-3f', u'tape2252204-0f', u'qg-07c15093-cb', u'qg-77f30b3d-d5', u'qr-1b5fc273-40', u'tap75fdaeaa-5d', u'tap35c9b883-a2', u'tapf565414d-ac', u'tap2e495414-c6', u'qg-dc934b13-4c', u'qr-586db496-37', u'qr-ff46a78b-c8', u'qg-5cf1780d-b9', u'tap337ad3d7-2c', u'tap2cf588b2-a1', u'tap5cff235e-08', u'qg-52be0ea1-65', u'ha-1060ce4f-b5', u'ha-e36b0f92-5b', u'ha-a4af1b77-8e', u'qr-bb3bfae9-df', u'qr-cdd71cf0-65', u'qg-cc0b2f19-fe', u'qr-fc87ff2b-b4', u'tapbf6795c3-c2', u'qr-6a46ebad-8b', u'ha-008fee9b-ae', u'tapefee8c02-79', u'tapd16a1440-2a', u'qr-0b0be302-82', u'tap58819f37-33', u'qg-635022ab-51', u'tap8f4a0112-3b', u'ha-78b5ea12-28', u'ha-3af299b1-51', u'tap72d424e1-15', u'qg-99fb128e-f4', u'tapbd738a56-fd', u'qg-18365403-87', u'qr-b52525ed-8f', u'qg-a8605e1b-ad', u'ha-dcbaca51-21', u'qr-5a20f8ae-1b', u'tap6010f799-d6', u'ha-27817a67-57', u'qr-e5c4dc81-49', u'tapcf3546be-67', u'ha-70d6ee20-4b', u'ha-5624549e-93', u'ha-adc937cb-5e', u'qr-d49c5f88-7d', u'ha-71debbe6-72', u'ha-e28d7db9-49', u'tap3a16cdb7-16', u'qr-37037348-23', u'qg-a4747698-85', u'ha-b29e7a59-44', u'ha-9f5da4d8-59', u'qr-82af4d32-08', u'ha-b0b995a6-ad', u'qr-e94e38d8-a2', u'tapf271e762-7a', u'ha-5e82070c-b8', u'ha-dea6c4fa-cd', u'tapfe96dc9b-c5', u'qg-57b508ef-cf', u'tapfe75ad65-88', u'ha-925bde21-96', u'qr-5e615c28-9c', u'qr-ddfbceb9-8b', u'tape0fbf372-97', u'tapf707c03f-14', u'qg-9ae4a23f-b0', u'qg-1f696b34-bb', u'qr-b888a2de-8c', u'qg-004e9530-d8', u'qg-e8dbe1a1-b9', u'tapdd16fa68-52', u'tapadbdfdd7-c7', u'ha-3913b84d-02', u'tap57f0426c-f5', u'qg-39ba4958-71', u'ha-06594765-c4', u'ha-6b05f153-17', u'ha-fbe90a82-7a', u'tapfce3bcb4-e6', u'tapf4868c81-c1', u'qr-6ac5c2f9-9e', u'qr-6d897610-3f', u'qg-d866fa7e-61', u'tap821c1641-7d', u'qr-adc844c3-ec', u'qr-5bc7495e-a3', u'qg-500d1cd8-d2', u'qg-74993dba-c3', u'tapd8e0462f-1b', u'qr-27724ca6-37', u'tap6678b161-6e', u'tap82c07649-79', u'qg-1a046de8-78', u'tap56978e6e-d7', u'qg-71d87be3-cd', u'qg-af274a25-e6', u'qg-fa1ff07c-d7', u'ha-7f6f97ee-11', u'qr-08a821b9-59', u'tap82ef6f01-b9', u'tapcc04d41d-c7', u'qg-e50a64db-4c', u'ha-f26b98ea-9a', u'qr-e4ede30d-b4', u'ha-050fce08-63', u'qr-99c6dff5-39', u'qg-3c0285b6-01', u'qr-92eecf47-9a', u'tap578b379e-90', u'qr-fddd2b99-0f', u'qr-03340cfe-8b', u'qg-dbc25476-29', u'qr-afb62fb7-2e', u'qg-55d7ccf9-3e', u'ha-28977e7e-99', u'qg-50b15ddf-ce', u'tapcc75a176-98', u'ha-f0a6e8cb-61', u'ha-6a52abc3-2b', u'tapf7f00fdc-d1', u'ha-cf0dd5f2-33', u'qr-a753c2e3-a6', u'qr-154052a4-e3', u'ha-e415c361-75', u'qr-94566863-9c', u'qr-b1372ebb-59', u'tap41735fde-8c', u'tap60640f58-66', u'tapcdc0ae83-ab', u'qg-e1dd4e59-72', u'qr-b6dfd7b1-95', u'qg-918987c0-84', u'tapfc906c2a-64', u'ha-28da5f95-87', u'qr-f15e81a3-97', u'qr-374ee24c-5f', u'tapf34ae5b5-9f', u'ha-3bcbb2f9-74', u'tapa1cf8271-9c', u'tap8c2de1e4-7c', u'tap4a8f0725-14', u'ha-46081f69-3e', u'qr-fa9a296e-73', u'qg-9d94b8ac-09', u'qr-273da82b-9d', u'tapc84520b9-37', u'qg-b3ebb666-65', u'qr-56da2e06-73', u'qr-df225961-e9', u'qg-f7e05e2f-a3', u'qr-9aa9ad4c-85', u'tap95a7da9e-cd', u'ha-9cc3d2c2-50', u'qg-0d088ac7-82', u'tape10ee1d6-e1', u'tapea94e518-7d', u'qg-0aa83ace-0c', u'tapd28f82bf-fb', u'qg-86225cd9-c6', u'ha-dfedf61f-f7', u'qr-5cc5579b-1e', u'qg-4e3f6e20-12', u'qr-216dc21f-ca', u'ha-963205a4-9c', u'qg-1b797933-1e', u'ha-3b367554-f5', u'qr-8d5bc6a0-44', u'qg-e29c94da-fa', u'qr-81a06351-2d', u'qg-5ad63867-db', u'tap6c34669a-d8', u'qg-e3af69ad-15', u'tapb8ceae6f-be', u'tapef1399b2-b9', u'qg-bf1ab580-fe', u'qr-bc366b41-b3', u'qg-14ba3134-2f', u'qg-f6f80c1b-e3', u'tap4b99c0b0-cd', u'qr-f55f1971-f6', u'qg-463dffba-86', u'qr-cfbdcd6c-4b', u'tap98f8de1d-04', u'qr-014da039-50', u'qr-903f5a5e-70', u'qg-89567997-3f', u'ha-10d81654-09', u'ha-0ccbfdec-c9', u'qg-d8670208-b3', u'qr-7e270565-52', u'tapdee4b894-d8', u'tap6b2bcf4c-a5', u'qr-ebd851b0-ff', u'qr-202151e3-8a', u'qr-ec457d16-50', u'tapb464df02-d9', u'qr-81708e2f-ef', u'qg-013e89b3-fc', u'tap88190af9-27', u'qr-78a8534f-b2', u'qr-420e3bbd-d6', u'tap9373cdba-f9', u'qg-f33eb166-ae', u'qg-fd898d5f-9d', u'tap5f83ec2f-a8', u'qg-5dd037c9-cf', u'tap881fc7b7-29', u'tap874c94f7-45', u'qr-806d1221-24', u'qg-b96733fa-a0', u'tap33644482-4a', u'tapb20d819a-49', u'qr-f4161e97-d9', u'tapc75812b9-01', u'qr-1450dd77-25', u'qr-ef59a869-95', u'qg-dbe7c613-8b', u'qg-d5b3aba7-c7', u'qg-ff8da58f-ed', u'ha-28467daa-b7', u'qr-c024c5c3-d4', u'qr-0c3ef1f2-00', u'qr-f53a90ba-d6', u'qr-91b7e8bd-d9', u'ha-05f29c78-9d', u'tape33e6548-3e', u'qg-492495c8-8e', u'tap4a985893-f5', u'tapeb05e257-35', u'tap0773194d-17', u'ha-0f19974b-35', u'ha-85e2e945-2c', u'qr-0828b088-c3', u'qr-ffe3579f-53', u'tape3836900-b3', u'qg-661dbae2-c7', u'qg-e82643cc-78', u'tapdf795d1e-de', u'qr-72aaaa6f-90', u'qr-260d5b31-94', u'ha-67c22231-29', u'tap22b07aed-c7', u'qg-aafe52f3-1c', u'qr-5666274f-4f', u'qr-1622e607-dd', u'qr-cf6dbbf1-66', u'tapc4420858-27', u'qr-e0f0d2de-c0', u'qg-36c5ced0-88', u'tapba2db030-13', u'tap99bd95d7-a2', u'tap01427dac-a1', u'qg-cb137346-6a', u'qg-b3f7a754-bb', u'tap79b069ff-18', u'ha-52bd95da-55', u'ha-bef6b21a-9f', u'qg-be9aed2d-5d', u'qg-eac3c95a-58', u'ha-f49c5cac-aa', u'tapcf97811d-9e', u'tapd7481c01-60', u'qg-8ab63c9b-0e', u'ha-aecd10e8-af', u'ha-05c3a8e4-02', u'qg-bea6a362-bd', u'qg-3ae51829-35', u'ha-245d8576-20', u'tapa6d47791-52', u'ha-0f702ea2-73', u'qg-883e3a49-e4', u'tap84b793e7-66', u'tapa6b01736-c7', u'ha-4659105c-83', u'ha-0f40c095-6a', u'qr-d063d4dc-ce', u'qr-f346db3a-51', u'qg-09746231-8c', u'tap6beb830a-c5', u'ha-c2b1c1f7-6d', u'qr-7c894408-0b', u'qg-a2526eb7-da', u'tap3731b895-a8', u'qr-565190fd-7a', u'ha-d55d8ff1-3b', u'qr-f7cfacc9-10', u'qg-5c158e60-af', u'qg-ade295f2-1f', u'tape02d1f4e-0e', u'qr-a4c91c8a-4f', u'qg-346889b5-cf', u'ha-ff4cf398-b1', u'qr-8709dbea-43', u'qg-2247e777-b8', u'ha-854aaa26-c3', u'tapefefa302-e2', u'qr-039ed898-99', u'qg-36b9f98a-35', u'tap5dad0624-fe', u'qr-e463bf33-61', u'tap133dfed8-8d', u'ha-9f835048-79', u'ha-323ecf08-4d', u'qg-60dcaecd-79', u'tap6f19ce26-a4', u'ha-baa5114f-67', u'qr-204d2187-78', u'tap8b56cebf-e8', u'qr-d8054194-9c', u'qg-518f70b1-aa', u'qr-8a8f3c92-e0', u'tapc83773b7-37', u'tap31c9d0df-d6', u'tap2e204e94-32', u'qg-3753c19c-be', u'ha-186aaae6-9e', u'ha-57edea5f-e5', u'qg-2e8f2dd9-aa', u'ha-39602179-31', u'qg-ef50132e-67', u'tap6a427fc7-17', u'qg-9dd41778-45', u'qg-73d6fb3b-e3', u'qr-06391ad2-f1', u'tapf28f1d03-f3', u'qg-5daf8908-12', u'qr-3e1aeb73-ab', u'tapb1bd7acf-cc', u'tapac527bad-02', u'ha-7b9bf02c-41', u'qr-822df584-6b', u'ha-b3a7717e-56', u'qg-8c72ce6a-37', u'ha-83dcb474-be', u'ha-c0ec2e73-fc', u'qr-319f89aa-8b', u'qr-262ab449-98', u'qr-c4b82e9c-17', u'qg-7db1e5bb-10', u'tapb53a4468-cb', u'qg-c254cef3-d5', u'ha-a79994e8-6c', u'qr-b71184fe-b7', u'qr-cba3bab6-d2', u'qr-d7be7709-d5', u'ha-7d8eea69-a6', u'tapd3ced7ea-32', u'ha-7681e56b-45', u'ha-7446cd95-07', u'ha-14476ff8-26', u'tapb936f8f8-18', u'tap77b932a5-72', u'tap4979940c-73', u'qr-4d4422a3-47', u'tap97d675d9-47', u'ha-85ed7db3-d9', u'qr-ce8d9287-e2', u'ha-df8d93ab-28', u'tap7d731c1f-23', u'ha-31b696aa-ea', u'tapa1042ba8-b6', u'tapd4491ced-d3', u'tap97c5d6e5-3d', u'ha-007e06e5-b3', u'tap47c66fa2-c4', u'qg-053d3deb-34', u'ha-62b3add2-d8', u'tap72bc0a73-57', u'tapc30361ac-c7', u'qr-10052fed-7b', u'qg-2b9ce25c-5b', u'ha-a9601f0c-ea', u'tap73a736ba-82', u'tapa8023f54-97', u'ha-0edccc44-12', u'qr-99e23015-3d', u'tap9c634d47-15', u'qg-5e425729-98', u'ha-9d78962c-ee', u'tap9276e59f-6a', u'tap3af620a2-91', u'qr-a56fe2dc-92', u'qg-d3c79a2c-5c', u'ha-1cd4323e-32', u'qg-887abc8c-b3', u'tap3e1b693a-2b', u'ha-6dc67817-5a', u'tapf878780c-83', u'tapa96fd071-f9', u'tapaf946194-e7', u'qr-13ed510d-78', u'tapefb6fb95-2f', u'qg-8969a0e0-a6', u'qg-518c018e-87', u'qr-4e831438-a2', u'qg-fa32d20d-ab', u'qr-2b6b615d-71', u'qr-97f3d69d-4b', u'tapc6cc5faf-3c', u'qg-4f00ff4c-76', u'tap9c15f262-09', u'tap8cda620f-9b', u'qg-3d45af3d-0c', u'tap7e9db226-63', u'qg-a188e9e0-74', u'tapfdc0f37c-c9', u'qg-e3cd5b37-a8', u'qr-c40e6383-ce', u'qr-baecc6ee-d7', u'ha-3c4b90c8-b4', u'ha-0e7cc454-a8', u'tapf8411537-77', u'tapf8cbd585-65', u'qg-eb4cf206-c6', u'qr-dead7458-96', u'qr-2717fa3a-b8', u'tape630bd08-ac', u'qr-505f6988-80', u'ha-bd6d33a0-ed', u'ha-4b1e577d-7d', u'ha-6baea9d5-ec', u'tapcaf3c580-de', u'tapdec008c7-9a', u'qr-df7d6daa-33', u'tapbcbbf510-49', u'tap20b8341c-f3', u'ha-ec40e8df-2b', u'qr-a384b6d8-a0', u'ha-5b3f4706-85', u'tapd4be8a1a-ac', u'tape2324dbb-1d', u'ha-b3f133bd-05', u'qg-c7acef20-44', u'ha-ee983ae5-88', u'tap80b86bfa-0b', u'qr-da1f7267-13', u'qg-2e2a6263-59', u'qg-562dbf09-01', u'tap8c4dbaac-ac', u'qg-2ce0fc25-9f', u'tape0c7d412-47', u'ha-50815185-e8', u'tap648f840c-b4', u'ha-877e509d-10', u'qr-5bb72098-52', u'tapdc09710a-0a', u'tapd0536226-42', u'tap62c4e40c-ce', u'qr-1b4348f4-03', u'qr-f1ccce80-8f', u'qg-66512f8c-bf', u'qr-22e1a700-e6', u'tape8e4dfb2-97', u'ha-086e981f-e3', u'qr-283b3cf4-97', u'qg-bce44ea4-ad', u'tap4748f072-fb', u'qg-299309aa-1a', u'ha-95d6aeef-f7', u'ha-5b4ea389-95', u'ha-7dd0a13f-6e', u'ha-0f30620d-ce', u'tapd8afa05a-28', u'qg-0d2cd525-35', u'qg-d40f8261-c4', u'ha-c065496d-b9', u'ha-6da7c7b6-01', u'qg-3d0b782b-b5', u'ha-82a46fb4-75', u'qg-03fc3b24-1c', u'tapde96c913-ad', u'qr-6c63ad3c-cf', u'qr-86a1af27-c5', u'ha-c5f1e384-42', u'qg-bac77d84-4a', u'qg-30aff087-24', u'qg-9e587924-70', u'tap10ced7de-23', u'tapaf576cfc-73', u'ha-8cc0ceab-d4', u'qr-26b74630-bc', u'tape0279984-a2', u'ha-e74292d4-2a', u'ha-66880207-66', u'qr-ebf154e9-2e', u'ha-7867a544-14', u'tapdf80be33-44', u'tap5fb86bf8-04', u'qg-909026c8-01', u'qg-522c1b78-c6', u'qg-167561a2-dc', u'qr-bed2de72-a2', u'qg-3ecfbd90-f2', u'tap7713324a-75', u'ha-74a2fcb2-3f', u'tap0ca44143-92', u'tapcdd77472-ac', u'tap564852c8-4f', u'tap760dc4e3-c6', u'qg-a7b6ab39-76', u'ha-3c0ffe5d-e4', u'tap4988b335-e0', u'tap964b15d6-1a', u'tap872c30c9-79', u'qg-63a89d23-a1', u'tap4c0442e1-28', u'qg-a70955b4-11', u'qg-7919067b-02', u'tapd96517b4-df', u'qg-61675033-30', u'qr-f0153a4b-a7', u'ha-1ca62cfd-02', u'qr-d424a7af-1d', u'qg-19e371f3-d5', u'qg-799a5ddb-15', u'qg-f9921c3a-0a', u'ha-e4ac11fa-45', u'qg-f12434d0-65', u'tap7a67911b-e1', u'qg-27dab0a5-50', u'tapf18a71b0-8e', u'ha-6a1ccb5d-41', u'tapd9cc0d82-ba', u'qg-b11dde45-a1', u'qr-8c195d4c-f3', u'ha-1b2b3843-41', u'tap80d514cc-30', u'qr-8b704aec-5f', u'qg-68d3a8eb-c9', u'tapdecb818e-2c', u'qr-7d230b3f-ac', u'qg-9460792e-eb', u'ha-548b372b-09', u'ha-ae80909f-ee', u'ha-6beb3ffc-f3', u'tap94c72b29-5e', u'tap6981ff73-ed', u'ha-71c0ac28-48', u'ha-10d120f3-43', u'tapf073ddec-4f', u'qr-a489c069-4e', u'qg-b95d1f67-ec', u'tap827251c4-8c', u'tapc9161970-2c', u'ha-47841b61-b4', u'ha-dc2cc428-40', u'qr-8652764c-f6', u'ha-38dd6504-88', u'tap614e1cb4-23', u'ha-67d20abd-6b', u'ha-d33da7fc-87', u'qr-19dc219e-61', u'qr-c0fcc12e-08', u'tap42c2272d-ab', u'tap9869b000-32', u'tapb203539a-d7', u'qr-48044243-3b', u'tap2cdc2ec3-90', u'tapbf6e7a1a-e2', u'qg-76be07cb-a9', u'tapa338ef3a-a5', u'tape5fe0d43-0c', u'qg-9b189608-a7', u'qg-e6e73f13-4d', u'tap3232e9f2-aa', u'qr-13858731-9b', u'qg-269f48c0-18', u'qr-172a667a-0a', u'tapf7563234-66', u'qr-e0dd0bc6-cb', u'qg-4489118a-cb', u'qr-41f2080c-07', u'ha-2ad7f5f6-90', u'ha-27508618-ed', u'tapd276977e-f0', u'tap74e59911-74', u'ha-6a2f1495-84', u'tapdb10a1b2-3f', u'qg-16427fcd-6d', u'tapb2274353-86', u'tapfee729da-9b', u'ha-e9f170e4-98', u'qg-73c4fa87-74', u'qg-14696152-03', u'ha-925da4ae-5b', u'tap605b1752-3d', u'ha-18cd0c14-05', u'qg-de368a13-28', u'qg-8b510324-d5', u'qg-7f8fb44d-68', u'ha-0659f802-86', u'qr-eaf4381e-fe', u'tapee73bac7-3e', u'qr-8898b8f7-ca', u'ha-2d8fca3a-b1', u'tapee7c791b-f7', u'ha-df764eb9-ee', u'ha-003110b4-d4', u'ha-688f746a-50', u'ha-0d4dd8e2-4e', u'qr-e17b92f4-60', u'tapbea6b766-0c', u'tap988747a5-dc', u'qg-2d541e31-8d', u'tapc8163747-fe', u'qr-8ba43e8c-7b', u'qg-ad006937-54', u'qr-b33c8a85-20', u'qr-816d325c-c2', u'tapee478bd6-c8', u'qg-64c95f5a-91', u'qg-a3263633-95', u'tapb4315c78-9f', u'qr-c5689a27-05', u'tap57c92d23-a7', u'qg-7069ea6a-48', u'qg-7c062816-11', u'ha-28ec231a-67', u'qr-5e1285ed-91', u'qg-e73963ff-84', u'qr-0ae3f22e-37', u'qr-98589de7-6e', u'qg-3b6cb618-b6', u'qg-df24c083-db', u'tap8f114b1a-97', u'ha-36a32ac9-48', u'tapbace27ed-31', u'qg-45e162ee-e2', u'tap4cda729b-66', u'tap50459348-39', u'qr-abbe7256-c7', u'qr-87acbf42-63', u'tapaef64a39-05', u'qr-58d47fc6-9b', u'qr-d7b705ec-60', u'tapd6717d86-26', u'ha-29d214ff-b1', u'qr-74a53230-32', u'tapdf7a65b6-f0', u'qr-42a8f906-a1', u'ha-3e7e3c4c-aa', u'tapdff0853f-aa', u'tap8366f504-fa', u'qr-152d5a66-1d', u'qr-ad5ebdf4-98', u'qg-04710968-4d', u'ha-416e2ce2-aa', u'qg-5546e229-5e', u'ha-daf4a37c-78', u'qr-e01e0d07-f6', u'qg-cf9c9b7e-86', u'ha-913984a5-44', u'ha-0e4264f2-d3', u'tap43134c9b-88', u'qg-b227dfba-9f', u'ha-cee077ed-c8', u'tapea211058-c3', u'tapf15a18df-82', u'tapc05baa61-d0', u'tap7973c638-0d', u'qg-0c02e6fa-77', u'tap2160c6c5-26', u'tap8f5713bc-21', u'ha-9e9cc2e7-46', u'qr-a28d9d4a-5b', u'int-br-ex', u'qg-fabef145-87', u'qg-03102420-45', u'tape5d9a886-26', u'tapa47bfecd-f6', u'tape7639f4b-9c', u'tap6836f620-c2', u'qg-d799e982-1e', u'tapbe7fa307-74', u'tapf4f942c9-eb', u'ha-87a5fb82-79', u'ha-e1555ea5-c8', u'tapfbf4ba33-89', u'tap911940da-0d', u'qr-73fb9aa4-71', u'tapc5323817-29', u'tapcea9c4a0-6b', u'ha-ce9a2fe2-ab', u'ha-db9d2181-75', u'qr-75cbe2eb-35', u'tap2039eab3-cb', u'qg-3ec6fa4f-25', u'tapeda179bf-28', u'ha-c039321d-b6', u'tapa73d24f0-a6', u'qg-c7d50571-76', u'ha-8fecba40-07', u'ha-a6889635-8c', u'qr-3bd6c35a-10', u'tapb1e923e8-ce', u'qg-c568bfa3-82', u'qr-dfa93ad4-b7', u'qr-1a0c058e-25', u'qr-ffd7364c-cd', u'ha-43373ab3-48', u'tapf432e2c8-cb', u'tapc7219b50-0b', u'qr-28b14b57-f1', u'tapaf23db46-2e', u'qr-2216588d-f7', u'tapcc9422f2-43', u'qr-f2112d7a-d9', u'tapebad9026-98', u'qr-98d59a3e-22', u'qr-8736d9c6-45', u'ha-39f59b8a-89', u'ha-7910bc10-38', u'tapc0d6826e-c5', u'tap6e3d54a9-3c', u'qr-cf8cb1f6-86', u'ha-431d35e4-8c', u'ha-473ec767-bd', u'qr-76f8d5b3-46', u'qg-f1142251-1b', u'tape8d86564-51', u'ha-afafd8e3-78', u'ha-1924dca3-60', u'qr-6d40c0c2-ad', u'qr-d3a858d1-00', u'qg-7f4f8e32-b7', u'qr-3f2fee05-cf', u'tapca263fd6-7a', u'ha-24dfb4af-2e', u'ha-145fef73-a2', u'tap98e4e0e5-1d', u'qr-091839ef-a1', u'tapa3eae933-1e', u'qr-ab71e49f-84', u'qg-d1a39f21-30', u'tapd0c33d2c-3b', u'ha-2a660c03-b1', u'tapd2bd3682-7e', u'ha-cb64712a-4e', u'tapafeb1bd1-ac', u'ha-d0835129-34', u'ha-8d45ced5-cb', u'qg-38d0ed2c-0c', u'qr-65f3d315-41', u'tapfc0a51d2-bd', u'ha-1acb7228-e5', u'qr-d282ade4-13', u'tapc3c56a3e-20', u'ha-878a22a9-12', u'ha-de2f3b4a-0e', u'tap7eee645d-ba', u'tape9c67667-3b', u'qr-4060c911-ed', u'tapdea6f3d1-74', u'tapb8ddf996-d8', u'ha-cc2a5ae1-41', u'ha-87be50c7-2c', u'tapaed49c65-14', u'tape853db51-27', u'tapc4f3e72c-5a', u'qr-aeb4db8d-55', u'tap6567df5b-66', u'qg-c2c42ce9-46', u'qg-2dbefdf9-8e', u'tape10d6f70-43', u'tape0ac74ca-c3', u'tap760ec3b3-43', u'tap98cd1e5c-c0', u'qr-2f29c796-df', u'qg-ba5c0067-2a', u'tap94ca2a51-07', u'tapaa5a89d1-61', u'tapab85b31e-2c', u'ha-db764523-a5', u'qr-3a1fd94e-6e', u'tapec9cbdc9-dd', u'ha-8c297b78-2b', u'qg-24de00c3-7a', u'tapef72681e-ee', u'qr-cb23f37a-90', u'ha-45f585c3-78', u'tap5fb89351-ce', u'qr-919eb73f-ef', u'tap73103e9d-c5', u'qr-889835bf-01', u'ha-54d9cbae-2f', u'qr-15b44cfe-cc', u'qr-c2e8e132-d9', u'tap3509518c-e4', u'qr-f5813e27-57', u'ha-3efbd57d-44', u'tapb3ebbe76-18', u'qg-343d9980-5d', u'qr-f9dac4b2-6f', u'tap87bafbc1-40', u'ha-f3974557-6a', u'ha-f8a8ded2-c2', u'tapfaa0b48a-4b', u'qr-5d595687-3b', u'ha-9987c93f-63', u'qg-27c8cc40-51', u'qr-8eff6080-22', u'qr-4a638f62-fb', u'tapb47f3be3-2b', u'ha-f80bc502-81', u'qr-7b21d3ca-89', u'qr-f43e684e-3e', u'qg-7a2c935c-fa', u'qr-6421a47e-31', u'qr-5e6b9954-64', u'qr-4a990986-e9', u'ha-12c0baf3-be', u'tapb044793e-b6', u'ha-f263d6ee-e7', u'qr-71febbca-1f', u'qr-dd29a3aa-67', u'tap3ff11699-d1', u'qg-632471a7-a8', u'qr-f7d35ee7-b9', u'tapb3126cdd-1c', u'ha-ea1fc67c-a1', u'tap77b73537-a3', u'qg-7ca38ffe-42', u'tapd5d00dcb-53', u'tap607e38d1-e8', u'tapc7f6df38-06', u'qr-c4af1d41-e6', u'qr-f2b04697-f5', u'tap94e1f3c4-1f', u'tap8657c75d-e7', u'tapd65dfc76-a5', u'ha-b5b296bc-c6', u'qg-db9235dc-38', u'qr-531254f8-9d', u'ha-f4dfb9c5-ff', u'qr-616f4d55-e9', u'tapd54a2880-ba', u'ha-abd56b60-bb', u'qg-12a85376-f0', u'tapb6b0a773-f1', u'tapcc236310-b5', u'qr-c5da4687-dc', u'ha-2fdae2da-e7', u'ha-47646498-06', u'qg-493827e3-1c', u'qg-c5530c7f-86', u'qg-37d5fdd8-53', u'qr-085627c4-d8', u'tap2b771c96-3e', u'qr-2f527767-6f', u'ha-c7d62fe4-f8', u'qg-57bd364c-c9', u'ha-1e596e22-54', u'ha-224e6966-31', u'tapf6e05193-40', u'qr-05e8fe0a-cc', u'qg-3d6f5ac3-43', u'tapbb97d983-4d', u'tapaf3f563d-a5', u'tap432a2f13-37', u'tap7bfb1493-2c', u'tapedcf0615-6c', u'ha-7dc8bfc4-7e', u'ha-d7727d9f-23', u'qg-c86159d2-ee', u'ha-23cec660-ba', u'ha-cdd9619a-9d', u'qg-c062e43c-0a', u'qg-6ab8a3d2-fe', u'ha-4405ecd3-c9', u'qg-5f25feab-88', u'tapa557bb0c-d1', u'ha-26640385-7b', u'qr-cebf3621-d5', u'ha-7982a0e1-80', u'qr-ff4970d7-35', u'ha-1ec21102-5f', u'ha-1a6d1a32-ee', u'tap34322086-29', u'tapc62740ba-de', u'qr-0741548d-a0', u'tap89828637-7c', u'ha-d95934c1-f8', u'ha-fd318772-8f', u'qr-48e76f18-f0', u'qg-fd213fea-11', u'qg-f1a5706c-07', u'qg-58e77f08-ba', u'qg-6003fbeb-8f', u'ha-4ca363c2-b6', u'qg-d3275773-e7', u'ha-241f3028-41', u'qg-c892e805-2c', u'ha-63a5ccf5-f7', u'tapd6395f5a-da', u'qg-067b0688-8e', u'ha-efc4c47e-8b', u'tapcf20ab65-1f', u'qg-6770f5d4-85', u'qr-a8761176-54', u'ha-7b292795-62', u'ha-65408422-5a', u'qg-cde41b4d-6b', u'qg-10b5b37b-82', u'ha-2f592a98-b6', u'qg-1b4c2d2c-62', u'ha-1b9fa1a4-ad', u'ha-6e5b3dca-d5', u'ha-a6efc065-79', u'tapc8be5c19-3c', u'qr-1d94d9db-96', u'ha-fd32e29a-96', u'ha-d184d9ce-ae', u'ha-22f7f20b-5b', u'qr-ad2953bf-d4', u'qr-1be76948-0a', u'qr-f7421672-0f', u'ha-25f824bb-47', u'qr-1d4ac604-0a', u'ha-77c98bc7-d7', u'qg-436577c3-57', u'qr-bd123811-56', u'tapd55cc1f5-c9', u'qg-b63cdff1-89', u'ha-2244e984-96', u'qg-6d92e626-68', u'qr-acb80252-5d', u'tapf4da7480-36', u'qr-2bfd53a9-bd', u'qg-9fcb7d2d-44', u'ha-2a1a9865-16', u'tap34a2cf2e-73', u'ha-71086091-c5', u'qr-7dc49f35-d9', u'ha-113a852d-2b', u'tap84d78424-48', u'ha-e630a0d8-cf', u'qg-fbc269f3-71', u'ha-fe3861e0-6a', u'tap784e34b2-06', u'tapdb4af922-ad', u'tap93470629-b4', u'qg-6399481b-df', u'tap74bf14dd-c6', u'qg-279b7568-0f', u'qr-ff0b750b-dd', u'ha-2e77551f-57', u'qr-7360fa81-13', u'tapcfae0dbe-a1', u'qr-2e98849d-44', u'ha-10c7e57e-7f', u'qr-5b69d9a6-d6', u'ha-f7de8c1d-0d', u'qr-b3e9f17a-83', u'ha-147e8028-ad', u'tap91824a4b-af', u'tap97248ed6-e2', u'ha-7901ca35-be', u'qg-63122f0a-21', u'qr-2f8eec4c-52', u'tapa8a04048-a9', u'ha-3f6689b8-a5', u'ha-19f1d0ef-cd', u'qr-00af5051-72', u'qr-3bfd8dc9-f2', u'tap4cd93445-0e', u'qg-7fab33b1-3b', u'qg-6fc2f96e-ae', u'qr-68dfe075-14', u'ha-3186206f-19', u'tap24b87642-94', u'qg-195d6e21-38', u'qr-8152bad1-ac', u'qg-bed94c60-aa', u'qg-c1417af7-7b', u'tapec84a707-2a', u'tape1daac98-17', u'qr-63c25861-3c', u'qg-9a1a2350-49', u'ha-d9d5329f-5a', u'qr-cfba12ee-3b', u'tapcf958119-02', u'ha-787f3c33-a5', u'tap5f498a39-de', u'tap1d239fb5-52', u'qr-32083891-5c', u'ha-14948d9d-e4', u'ha-ddb07d93-cb', u'tapef4f9255-22', u'qg-c135fb69-44', u'qg-db40b33a-41', u'ha-51b184c4-db', u'tapcc21fed1-57', u'tap91adb685-c6', u'tap8fead629-3b', u'qg-c52d93af-aa', u'qr-454a8d75-cf', u'tap9f2cf581-03', u'qg-3d72b46b-07', u'tapbe5e8c95-88', u'tap742c0067-09', u'tapc209c158-45', u'ha-b3a88b74-cf', u'qg-82d622bb-08', u'qr-f20686d0-8a', u'qr-0b007521-73', u'qg-b8776f4b-39', u'ha-ca3da720-50', u'qr-81b480d1-97', u'tape76d4a39-70', u'tapee25c1fd-51', u'ha-6b89fce0-3a', u'ha-6ff0c5ac-22', u'qg-88f0a4a5-00', u'qg-5da4390d-d2', u'qr-1ab07321-ee', u'qr-09227e20-23', u'qr-d89e5257-28', u'qg-4fd4639f-a0', u'ha-8c01701b-36', u'tapb7aa2b24-7f', u'qr-c9f20f74-00', u'tapb4de85f3-fb', u'tap078dc47f-1a', u'tape2324abd-cd', u'qg-dce06458-02', u'tap430acf03-f2', u'qg-eacab584-70', u'qg-4242552b-ac', u'qg-85880a14-45', u'tapf027cf42-a0', u'qr-1b333a0b-78', u'qg-ecef7ad5-0e', u'qr-5cf0105a-e2', u'qr-bb748180-2b', u'tapd56babec-c1', u'tap554239bc-79', u'tape76fdb73-5e', u'tapfb25ce17-cb', u'qg-af0d8445-a8', u'qr-b4092df3-f4', u'qr-707f7cdd-c2', u'ha-30e5cab9-1a', u'qg-adbf9827-85', u'qr-db3f4677-18', u'ha-dfcb87f3-c3', u'tapdf7438b9-aa', u'qr-b76d49cb-4f', u'ha-75670f58-76', u'ha-b636dde0-bb', u'qg-638d497c-ad', u'tap732022e9-7f', u'qr-582a7444-68', u'tapf2c31377-d5', u'qg-c62ef8d2-c2', u'qg-177ffc23-09', u'ha-ecfcc8b4-85', u'qr-2daddb32-1d', u'qr-42ee6c84-c8', u'qr-438d4a1f-96', u'qr-db6ab05e-97', u'tap10349e24-15', u'ha-ee60f0e9-1d', u'tapbb10a799-d5', u'qr-d3e834ea-32', u'qr-268f0830-db', u'qr-d1d2924a-d4', u'qg-afd313b8-24', u'tapbc3f3102-0d', u'qr-7af77b08-57', u'tap695d0d75-59', u'qr-7c20436d-91', u'qr-4d531d96-1d', u'qr-ef5dcda1-4b', u'qg-56000ae3-13', u'qr-355d4af1-3c', u'ha-947e6ba3-62', u'qg-944e0880-f7', u'qr-1a014831-b9', u'tap3b326421-b5', u'tapd41581a4-12', u'ha-0154fcca-66', u'qg-25d6c8a1-1b', u'qg-7562cdc0-6b', u'ha-7f44a8cd-6d', u'tapfd3cdb92-c3', u'ha-f2030d52-08', u'ha-60d45df0-99', u'qr-0a33c67d-15', u'tap9b408343-c4', u'tapc2ed1a3d-69', u'qr-72e244fc-86', u'qr-1bb7d6d1-cd', u'tap99f7372f-fb', u'qg-7ca04545-9d', u'qg-846232c1-a7', u'tap21025362-fd', u'qg-c2357c8d-c6', u'ha-b247157f-63', u'qr-553588fe-a1', u'qg-31e5e69e-75', u'qr-633f2fc4-b9', u'qr-19bd9ecb-ea', u'tapdb8b73a3-25', u'tapbdea7f37-ad', u'tap727fda2e-5b', u'qg-1ea09a52-72', u'ha-9c6a0085-9a', u'qg-b36002fe-81', u'qg-16e4fdc2-41', u'qg-a5555414-dc', u'tap7451abf6-39', u'ha-b432399b-85', u'tapeb7e454b-9b', u'ha-9c79afd6-7d', u'tap90669746-08', u'tap5860f545-da', u'qg-0326fe70-e7', u'ha-ba974b02-99', u'tap999de4d4-c4', u'qg-e5dad208-30', u'qg-3cd1bdf9-19', u'qg-8cfb0d5e-35', u'tape81cc67f-f1', u'ha-a6c8134a-69', u'qg-7145ec0e-24', u'ha-521e85a4-84', u'tapd0fce789-33', u'ha-7993659c-3e', u'qr-0f8031e6-04', u'tap587b4600-e8', u'tapc4c98ed0-bb', u'qg-660539ce-fc', u'tap36cd94fd-95', u'patch-tun', u'tapf6153953-f9', u'ha-0f74017b-f8', u'tapc04c8746-16', u'qr-cbf3e63f-1b', u'qr-6cdec5bf-84', u'tapc1d7df36-9d', u'tapf52fb5b8-23', u'qg-0c7a386f-33', u'qr-a371d4a8-e7', u'tape3154ca5-48', u'tap7437c9dc-75', u'ha-530acd8d-a1', u'qr-b8c19505-6b', u'tap91f18ba0-22', u'tapabf2ba78-bf', u'tapfb080bc5-3c', u'ha-a36afe30-81', u'ha-68772f82-04', u'qr-ff9d32a5-e0', u'tapd3053e27-ea', u'qr-d38eaa5d-e8', u'qg-91b1b98e-9e', u'ha-10dc123a-51', u'qg-618368e8-1d', u'ha-3782f425-a0', u'tap09b9396a-04', u'qr-f2863558-f6', u'tapf7f7d555-29', u'ha-8c312188-29', u'qr-69786919-c8', u'qr-f54efeab-32', u'tapb693e242-9d', u'tap0077b381-d6', u'qr-058e8c50-f6', u'qr-1bbbb7a3-e7', u'qg-8d18de2d-19', u'tap960a83a3-8c', u'qr-81b1d893-0c', u'tapeb21f332-e3', u'qg-5712489d-a6', u'tap90a85ac0-01', u'tap5a2e6b0a-3c', u'ha-c3fca98e-55', u'ha-ba48a073-94', u'qg-6fd055d2-6d', u'qr-fe622863-c8', u'tapdc5dca6f-1f', u'qr-52e7795e-da', u'qr-ca5b14dc-08', u'qr-056c51e7-1d', u'tapb2d8ffaa-85', u'qg-5fa720f8-8e', u'ha-47de6474-aa', u'tapd06455e8-37', u'ha-de7439a9-78', u'ha-ed24d201-54', u'tap26370c38-f4', u'ha-23a26bba-dd', u'ha-9f596aa2-0b', u'ha-95d257dd-a2', u'qg-b581f25e-31', u'ha-c208b1d1-1d', u'ha-af671567-6f', u'ha-e8d97203-60', u'qr-f6f5ac70-00', u'ha-8fb94815-e2', u'qr-5db3e226-84', u'ha-b4cd1427-96', u'qg-68d81028-61', u'tap2a9ac01a-f0', u'ha-1277c7fb-3b', u'qg-8734e29d-d6', u'qr-21d99183-3d', u'qr-fda658db-08', u'ha-a1a75374-a2', u'tapc50fb07d-bd', u'qg-4073bac5-ea', u'qr-49c4b0e5-fa', u'tap2a1344d2-2c', u'qr-d22db034-3d', u'qr-78b60ebd-c3', u'tapadde1759-56', u'qg-66b9855d-d8', u'qg-ef1395bc-01', u'qg-3aac0a43-49', u'ha-44520c25-1c', u'qr-c49523ff-0d', u'tapc2593ff9-e1', u'qg-d9887277-78', u'ha-26694fc0-8e', u'qr-1db2f9ec-15', u'tapa5190a62-d9', u'tap7d3abc8d-ef', u'ha-6de1944d-f9', u'tap6c8b8003-7d', u'ha-673bb1a0-c4', u'qr-ff4b39e5-d5', u'tap8e8371f4-0f', u'tape4e1733b-c1', u'tape5326b83-e0', u'tapd4133d76-80', u'qg-97c202d3-8b', u'tap55673919-77', u'tap869ee6d4-ee', u'tap24f5d2b8-f9', u'tap6e841d3b-6d', u'tap79dcfee1-36', u'ha-57dd81f9-1a'], table=Interface, columns=['name', 'external_ids', 'ofport'])] exceeded timeout 10 seconds


I don't have the full picture, but Terry Wilson started looking at this, and he provided me with the following details:


-  DbListCommand also passes in the 1600 ports that it is individually listing. 1600 results, but it is probably looking up each individual list. Probably something where we could make a custom command that doesn't really follow what ovs-vsctl did to improve performance. (like with the ovs-cleanup delete command we added)

- DbListCommand is going to be O(n^2). It looks up each record by name for each record passed in.

- It would be better to just grab all of the ports and filter out the ones we don't want.

- and it is even worse, because get_vif_ports is called to get the port list that is passed to get_ports_attributes() and get_vif_ports also calls get_ports_attributes.... ( though get_vif_ports looks the info up via Interface table, and get_ports_attributes in ovs_neutron_agent usually called to look up info based on Port table. ) 

- The whole VifPort object should really be changed to just store all the data from the port.  we pull back the vifs from the db, then use their names to to look up individual info about them