Description of problem:
When virtual routers were failed over for the Mitaka upgrade there was a time-out on rabbitmq ( probably due to high number of routers which took a while to failover):
l3-agent.log:
2017-02-11 08:05:17.994 107895 ERROR neutron.common.rpc [req-4f9653ba-9c63-4cb9-97b4-6ed52f28341b - - - - -] Timeout in RPC method sync_routers. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.
2017-02-11 08:05:17.996 107895 WARNING neutron.common.rpc [req-4f9653ba-9c63-4cb9-97b4-6ed52f28341b - - - - -] Increasing timeout for sync_routers calls to 120 seconds. Restart the agent to restore it to the default value.
This caused that some of the routers were on standby on all l3-agents:
# neutron l3-agent-list-hosting-router 09a20aeb-8f24-4104-a91d-1d3f270b1343
+--------------------------------------+-----------------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+-----------------------------------+----------------+-------+----------+
| 4b750613-1870-426c-9a5b-aa92c1bb7d9c | host51a | True | :-) | standby |
| 5a7700dc-4b39-4783-b960-e0dbaa015a4f | host52a | True | :-) | standby |
| b194d5a3-21c3-4c37-9dbb-4b3d28131cec | host51b | True | :-) | standby |
+--------------------------------------+-----------------------------------+----------------+-------+----------+
Current workaround is to trigger the retry manually:
# neutron l3-agent-router-remove 4b750613-1870-426c-9a5b-aa92c1bb7d9c 09a20aeb-8f24-4104-a91d-1d3f270b1343; sleep 5; neutron l3-agent-router-add 4b750613-1870-426c-9a5b-aa92c1bb7d9c 09a20aeb-8f24-4104-a91d-1d3f270b1343;
# neutron l3-agent-list-hosting-router 09a20aeb-8f24-4104-a91d-1d3f270b1343
+--------------------------------------+-----------------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+-----------------------------------+----------------+-------+----------+
| 4b750613-1870-426c-9a5b-aa92c1bb7d9c | host51a | True | :-) | active |
| 5a7700dc-4b39-4783-b960-e0dbaa015a4f | host52a | True | :-) | standby |
| b194d5a3-21c3-4c37-9dbb-4b3d28131cec | host51b | True | :-) | standby |
+--------------------------------------+-----------------------------------+----------------+-------+----------+
This bug seems to be fixed uptream:
https://bugs.launchpad.net/neutron/+bug/1648242
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. trigger failover of high number of virtual routers
2. l3-agents report standby state on some routers
3.
Actual results:
Expected results:
all routers move smoothly and report correct status
Additional info:
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2018:2851