Bug 1281254

Summary: [Backport] Race conditions of L3 HA Scheduler
Product: Red Hat OpenStack Reporter: yfried
Component: openstack-neutronAssignee: John Schwarz <jschwarz>
Status: CLOSED ERRATA QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: amuller, chrisw, dcadzow, ggillies, ihrachys, jlibosva, nyechiel, srevivo, swaite, tfreger, tkammer, yfried
Target Milestone: ---Keywords: Automation, AutomationBlocker, FeatureBackport, Triaged, ZStream
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-2015.1.4-3.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1360352 (view as bug list) Environment:
Last Closed: 2016-08-31 17:38:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1360352    

Description yfried 2015-11-12 07:43:58 UTC
Description of problem:

I'm able to create the router with GW, and I can unset a GW from a router.
I can't set a GW after router is created.

$ neutron --debug router-gateway-set router1 10.8.184.0/22
DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://10.8.188.11:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG: keystoneclient.session RESP: [200] content-length: 337 vary: X-Auth-Token connection: keep-alive date: Wed, 11 Nov 2015 08:13:40 GMT content-type: application/json x-openstack-request-id: req-4d3ebc76-f842-4124-8200-a9a5f3b3895d 
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.8.188.11:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG: neutronclient.neutron.v2_0.router.SetGatewayRouter run(Namespace(disable_snat=False, external_network=u'10.8.184.0/22', request_format='json', router=u'router1'))
DEBUG: keystoneclient.auth.identity.v2 Making authentication request to http://10.8.188.11:5000/v2.0/tokens
DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://10.8.188.11:9696/v2.0/routers.json?fields=id&name=router1 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}005593b7df9a564b4a3f22a800515ef21034b0c1"
DEBUG: keystoneclient.session RESP: [200] date: Wed, 11 Nov 2015 08:13:42 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 61 x-openstack-request-id: req-2e771a30-eb40-4256-868a-32d6d611cd64 
RESP BODY: {"routers": [{"id": "dd6c40aa-e0ff-4538-933e-0d6ea29d9130"}]}

DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://10.8.188.11:9696/v2.0/networks.json?fields=id&name=10.8.184.0%2F22 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}005593b7df9a564b4a3f22a800515ef21034b0c1"
DEBUG: keystoneclient.session RESP: [200] date: Wed, 11 Nov 2015 08:13:42 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 62 x-openstack-request-id: req-172e749c-b132-45f1-8f4a-d2c234cbef56 
RESP BODY: {"networks": [{"id": "8b4e5869-4990-4f82-beee-18799e9ea8bf"}]}

DEBUG: keystoneclient.session REQ: curl -g -i -X PUT http://10.8.188.11:9696/v2.0/routers/dd6c40aa-e0ff-4538-933e-0d6ea29d9130.json -H "User-Agent: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}005593b7df9a564b4a3f22a800515ef21034b0c1" -d '{"router": {"external_gateway_info": {"network_id": "8b4e5869-4990-4f82-beee-18799e9ea8bf"}}}'
DEBUG: keystoneclient.session RESP:
DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}
ERROR: neutronclient.shell Request Failed: internal server error while processing your request.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 766, in run_subcommand
    return run_command(cmd, cmd_parser, sub_argv)
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 101, in run_command
    return cmd.run(known_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/router.py", line 200, in run
    neutron_client.add_gateway_router(_router_id, router_dict)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 678, in add_gateway_router
    body={'router': {'external_gateway_info': body}})
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20
    status_code=status_code)
InternalServerError: Request Failed: internal server error while processing your request.

Comment 3 Assaf Muller 2015-11-12 13:35:15 UTC
Since this is on OSP 7 I removed blocker for OSP 8. We're working on what I think is the fix, but I wanted to ask to be sure:

If you create another, new HA router, and add a gateway for that one, does it work?

Comment 4 Assaf Muller 2015-12-01 17:14:47 UTC
Can we have neutron-server logs showing the TRACE?

Comment 5 yfried 2015-12-02 10:40:14 UTC
The setup was scrapped so I no longer have access to it. Sorry

Comment 7 Assaf Muller 2015-12-04 16:06:51 UTC
Not sure the linked upstream bug has anything to do with the error reported. I asked for more info in the Jira ticket.

Comment 8 Assaf Muller 2015-12-04 16:12:19 UTC
Sorry for my ignorance, how did you "create a router with a GW"?

Comment 9 Assaf Muller 2015-12-04 16:12:24 UTC
Sorry for my ignorance, how did you "create a router with a GW"?

Comment 10 yfried 2015-12-13 06:37:50 UTC
Via Horizon - allows you to choose GW on router creation.
Python SDK allows you to do that as well...

Comment 15 Nir Magnezi 2016-02-14 09:27:11 UTC
https://review.openstack.org/#/c/238122/ Won't get backported since it involves db migtation.

Adding https://review.openstack.org/#/c/257059/ to the list, which will be backported as soon as it's ready.

Comment 16 Nir Magnezi 2016-05-08 13:43:32 UTC
Backport is pending on: https://review.openstack.org/#/c/305774/

Comment 17 Assaf Muller 2016-06-04 02:49:30 UTC
*** Bug 1296708 has been marked as a duplicate of this bug. ***

Comment 18 Ihar Hrachyshka 2016-06-13 07:20:57 UTC
Why is upstream backport blocking us from making progress in OSP?

Comment 19 John Schwarz 2016-07-03 11:56:19 UTC
I've added 3 downstream-only patches that deal with backporting these patches, as to stop blocking OSP progress.

Comment 21 Eran Kuris 2016-08-24 06:53:36 UTC
Verified on openstack-neutron-2015.1.4-3.el7ost   
RHOS -7 HA ENV 
Run Rally : 
rally task start samples/tasks/scenarios/neutron/create-and-delete-routers.json

Comment 23 errata-xmlrpc 2016-08-31 17:38:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1789.html