1855172 – [OSP16.1] redeploying a number of times will eventually lead to heat.common.exception.RequestLimitExceeded: Request limit exceeded: JSON body size (4396634 bytes) exceeds maximum allowed size (4194304 bytes) on the undercloud

Bug 1855172 - [OSP16.1] redeploying a number of times will eventually lead to heat.common.exception.RequestLimitExceeded: Request limit exceeded: JSON body size (4396634 bytes) exceeds maximum allowed size (4194304 bytes) on the undercloud

Summary: [OSP16.1] redeploying a number of times will eventually lead to heat.common.e...

Keywords:
Status:	CLOSED DUPLICATE of bug 1842919
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-tripleoclient
Sub Component:
Version:	16.1 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	RHOS Maint
QA Contact:	David Rosenfeld
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-07-09 07:04 UTC by Michele Baldessari
Modified:	2020-07-09 13:47 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-07-09 13:47:28 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Michele Baldessari 2020-07-09 07:04:25 UTC

Description of problem:
So last night I was testing a fix for a race condition and I ran a redeployment for 100 times (no parameters changes). This morning I woke up to pretty much all of them having failed with the following:
  File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
    return request.get_response(self.application)
  File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
    application, catch_exc_info=False)
  File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/heat/common/wsgi.py", line 921, in __call__
    raise translate_exception(err, request.best_match_language())
  File "/usr/lib/python3.6/site-packages/heat/common/wsgi.py", line 884, in __call__
    action, request)
  File "/usr/lib/python3.6/site-packages/heat/common/wsgi.py", line 965, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/heat/common/wsgi.py", line 830, in default
    return {'body': self.from_json(request.body)}
  File "/usr/lib/python3.6/site-packages/heat/common/wsgi.py", line 823, in from_json
    raise exception.RequestLimitExceeded(message=msg)
heat.common.exception.RequestLimitExceeded: Request limit exceeded: JSON body size (4396634 bytes) exceeds maximum allowed size (4194304 bytes).

Now I can increase that value, restart heat_api container and everything is fine *but* the question is why did this work for the initial deployment and a few subsequent redeploys but stopped working later?

While I could not find anything obiously wrong in the plan, I did spot this in mistral log which might explain things:
mistral/executor.log:{"stack": {"stack_name": "overcloud", "id": "e7a5114b-3ff1-49d7-a020-4931e6c3f7c6", "links": [{"href": "https://192.168.24.2:13004/v1/4862fcfb52494b4f90168f859bdb14ab/stacks/overcloud/e7a5114b-3ff1-49d7-a020-4931e6c3f7c6", "rel": "self"}], "creation_time": "2020-07-08T12:38:39Z", "updated_time": "2020-07-09T05:40:14Z", "deletion_time": null, "notification_topics": [], "parameters": {"OS::stack_id": "e7a5114b-3ff1-49d7-a020-4931e6c3f7c6", "OS::project_id": "4862fcfb52494b4f90168f859bdb14ab", "OS::stack_name": "overcloud", "CloudNameStorage": "overcloud.storage.redhat.local", "StorageVirtualFixedIPs": "[{\"ip_address\": \"172.17.3.150\"}]", "CloudNameStorageManagement": "overcloud.storagemgmt.redhat.local", "StorageMgmtVirtualFixedIPs": "[{\"ip_address\": \"172.17.4.150\"}]", "CloudNameInternal": "overcloud.internalapi.redhat.local", "InternalApiVirtualFixedIPs": "[{\"ip_address\": \"172.17.1.150\"}]", "CloudName": "overcloud.redhat.local", "PublicVirtualFixedIPs": "[{\"ip_address\": \"10.0.0.150\"}]", "ExternalVirtualFixedIPs": "[]", "CloudNameCtlplane": "overcloud.ctlplane.redhat.local", "ExtraHostFileEntries": "1.2.3.4 stillhavemythumb", "UndercloudHostsEntries": "192.168.24.1 undercloud-0.ctlplane.redhat.local undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane undercloud-0.ctlplane ... 
(the repetition goes on and on).

Note You need to log in before you can comment on or make changes to this bug.