Bug 1281254 - [Backport] Race conditions of L3 HA Scheduler
[Backport] Race conditions of L3 HA Scheduler
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: ---
: 7.0 (Kilo)
Assigned To: John Schwarz
Eran Kuris
: Automation, AutomationBlocker, Triaged, ZStream
: 1296708 (view as bug list)
Depends On:
Blocks: 1360352
  Show dependency treegraph
 
Reported: 2015-11-12 02:43 EST by yfried
Modified: 2016-08-31 13:38 EDT (History)
13 users (show)

See Also:
Fixed In Version: openstack-neutron-2015.1.4-3.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1360352 (view as bug list)
Environment:
Last Closed: 2016-08-31 13:38:55 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 227821 None None None Never
OpenStack gerrit 230481 None None None Never
OpenStack gerrit 238123 None None None Never
OpenStack gerrit 257059 None None None 2016-02-14 04:27 EST

  None (edit)
Description yfried 2015-11-12 02:43:58 EST
Description of problem:

I'm able to create the router with GW, and I can unset a GW from a router.
I can't set a GW after router is created.

$ neutron --debug router-gateway-set router1 10.8.184.0/22
DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://10.8.188.11:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG: keystoneclient.session RESP: [200] content-length: 337 vary: X-Auth-Token connection: keep-alive date: Wed, 11 Nov 2015 08:13:40 GMT content-type: application/json x-openstack-request-id: req-4d3ebc76-f842-4124-8200-a9a5f3b3895d 
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.8.188.11:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG: neutronclient.neutron.v2_0.router.SetGatewayRouter run(Namespace(disable_snat=False, external_network=u'10.8.184.0/22', request_format='json', router=u'router1'))
DEBUG: keystoneclient.auth.identity.v2 Making authentication request to http://10.8.188.11:5000/v2.0/tokens
DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://10.8.188.11:9696/v2.0/routers.json?fields=id&name=router1 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}005593b7df9a564b4a3f22a800515ef21034b0c1"
DEBUG: keystoneclient.session RESP: [200] date: Wed, 11 Nov 2015 08:13:42 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 61 x-openstack-request-id: req-2e771a30-eb40-4256-868a-32d6d611cd64 
RESP BODY: {"routers": [{"id": "dd6c40aa-e0ff-4538-933e-0d6ea29d9130"}]}

DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://10.8.188.11:9696/v2.0/networks.json?fields=id&name=10.8.184.0%2F22 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}005593b7df9a564b4a3f22a800515ef21034b0c1"
DEBUG: keystoneclient.session RESP: [200] date: Wed, 11 Nov 2015 08:13:42 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 62 x-openstack-request-id: req-172e749c-b132-45f1-8f4a-d2c234cbef56 
RESP BODY: {"networks": [{"id": "8b4e5869-4990-4f82-beee-18799e9ea8bf"}]}

DEBUG: keystoneclient.session REQ: curl -g -i -X PUT http://10.8.188.11:9696/v2.0/routers/dd6c40aa-e0ff-4538-933e-0d6ea29d9130.json -H "User-Agent: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}005593b7df9a564b4a3f22a800515ef21034b0c1" -d '{"router": {"external_gateway_info": {"network_id": "8b4e5869-4990-4f82-beee-18799e9ea8bf"}}}'
DEBUG: keystoneclient.session RESP:
DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}
ERROR: neutronclient.shell Request Failed: internal server error while processing your request.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 766, in run_subcommand
    return run_command(cmd, cmd_parser, sub_argv)
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 101, in run_command
    return cmd.run(known_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/router.py", line 200, in run
    neutron_client.add_gateway_router(_router_id, router_dict)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 678, in add_gateway_router
    body={'router': {'external_gateway_info': body}})
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20
    status_code=status_code)
InternalServerError: Request Failed: internal server error while processing your request.
Comment 3 Assaf Muller 2015-11-12 08:35:15 EST
Since this is on OSP 7 I removed blocker for OSP 8. We're working on what I think is the fix, but I wanted to ask to be sure:

If you create another, new HA router, and add a gateway for that one, does it work?
Comment 4 Assaf Muller 2015-12-01 12:14:47 EST
Can we have neutron-server logs showing the TRACE?
Comment 5 yfried 2015-12-02 05:40:14 EST
The setup was scrapped so I no longer have access to it. Sorry
Comment 7 Assaf Muller 2015-12-04 11:06:51 EST
Not sure the linked upstream bug has anything to do with the error reported. I asked for more info in the Jira ticket.
Comment 8 Assaf Muller 2015-12-04 11:12:19 EST
Sorry for my ignorance, how did you "create a router with a GW"?
Comment 9 Assaf Muller 2015-12-04 11:12:24 EST
Sorry for my ignorance, how did you "create a router with a GW"?
Comment 10 yfried 2015-12-13 01:37:50 EST
Via Horizon - allows you to choose GW on router creation.
Python SDK allows you to do that as well...
Comment 15 Nir Magnezi 2016-02-14 04:27:11 EST
https://review.openstack.org/#/c/238122/ Won't get backported since it involves db migtation.

Adding https://review.openstack.org/#/c/257059/ to the list, which will be backported as soon as it's ready.
Comment 16 Nir Magnezi 2016-05-08 09:43:32 EDT
Backport is pending on: https://review.openstack.org/#/c/305774/
Comment 17 Assaf Muller 2016-06-03 22:49:30 EDT
*** Bug 1296708 has been marked as a duplicate of this bug. ***
Comment 18 Ihar Hrachyshka 2016-06-13 03:20:57 EDT
Why is upstream backport blocking us from making progress in OSP?
Comment 19 John Schwarz 2016-07-03 07:56:19 EDT
I've added 3 downstream-only patches that deal with backporting these patches, as to stop blocking OSP progress.
Comment 21 Eran Kuris 2016-08-24 02:53:36 EDT
Verified on openstack-neutron-2015.1.4-3.el7ost   
RHOS -7 HA ENV 
Run Rally : 
rally task start samples/tasks/scenarios/neutron/create-and-delete-routers.json
Comment 23 errata-xmlrpc 2016-08-31 13:38:55 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1789.html

Note You need to log in before you can comment on or make changes to this bug.