Bug 1712278 - [OSP13 Backport] Message collection size is too large for Zaqar
Summary: [OSP13 Backport] Message collection size is too large for Zaqar
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z9
: 13.0 (Queens)
Assignee: Adriano Petrich
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On: 1664055
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-21 08:34 UTC by Adriano Petrich
Modified: 2023-12-15 16:34 UTC (History)
23 users (show)

Fixed In Version: python-tripleoclient-9.3.1-1.el7ost openstack-tripleo-common-8.7.1-1.el7ost
Doc Type: No Doc Update
Doc Text:
Clone Of: 1664055
Environment:
Last Closed: 2019-11-07 14:01:17 UTC
Target Upstream Version:
Embargoed:
apetrich: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 663688 0 None MERGED Remove execution from workflow message send 2020-12-07 17:12:14 UTC
OpenStack gerrit 680688 0 None MERGED Stop sending execution object via Zaqar message 2020-12-07 17:12:14 UTC
Red Hat Issue Tracker OSP-30784 0 None None None 2023-12-15 16:34:07 UTC
Red Hat Knowledge Base (Solution) 4396231 0 None None None 2019-09-06 02:31:06 UTC
Red Hat Product Errata RHBA-2019:3794 0 None None None 2019-11-07 14:01:36 UTC

Comment 2 Adriano Petrich 2019-09-11 08:36:35 UTC
Everything has been backported upstream

Comment 3 August Simonelli 2019-09-26 04:42:43 UTC
Hello. Is there a target milestone for this backport to land in OSP 13?

Comment 5 Adriano Petrich 2019-10-04 14:41:26 UTC
It depends on QE avaiability but besides that I don't see why not.

Comment 7 Alex McLeod 2019-10-31 11:28:56 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Comment 8 Jad Haj Yahya 2019-10-31 12:30:29 UTC
No regression issues found

Comment 10 errata-xmlrpc 2019-11-07 14:01:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3794

Comment 11 Chris Smart 2020-03-24 23:28:38 UTC
I just hit this again, deploying z11 update at converge step.

~~~
queue_post failed: Error response from Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too large. Max size 1048576.\\n\"}, \"result\": \"Failure caused by error in tasks: send_message\\n\\n  send_message [task_ex_id=2df6a3f6-0aa0-4b0f-9546-f8efb9c36d0b] -> Workflow failed due to message status\\n    [wf_ex_id=5ca8156c-cc5c-4e74-b057-7971cc773f71, idx=0]: Workflow failed due to message status\\n\", \"deployment_status\": \"DEPLOY_FAILED\"}", "input": "{\"run_validations\": false, \"skip_deploy_identifier\": false, \"container\": \"overcloud\", \"queue_name\": \"tripleo\", \"timeout\": 480}", "created_at": "2020-03-24 22:43:39", "project_id": "6829dc9650ef411ca67cd1654d5774df", "id": "6e069679-612c-4bab-a7c6-00a84651c322"}
~~~


Director is at z11 and has:

openstack-tripleo-common-8.7.1-12.el7ost.noarch
python-tripleoclient-9.3.1-7.el7ost.noarch


For now I'm applying my previous work around to increase the sizes even more and trying again.

Comment 12 Chris Smart 2020-03-27 01:45:52 UTC
Just to follow up, increasing the size further worked.

~~~
sudo crudini --set /etc/zaqar/zaqar.conf transport max_messages_post_size 2097152
sudo crudini --set /etc/zaqar/zaqar.conf oslo_messaging_kafka producer_batch_size 32768
sudo crudini --set /etc/mistral/mistral.conf engine execution_field_size_limit_kb 32768
sudo reboot
~~~

Comment 13 Shravan Kumar Tiwari 2020-08-05 10:50:52 UTC
Seems like the issue is not fixed completely.(env RHOSP13z12)
We have hit the issue during scale out of the compute.

'ZaqarAction.queue_post failed: Error response from Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too large. Max size 1048576.: ActionException: ZaqarAction.queue_post failed: Error response f      rom Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too large. Max size 1048576.
68477 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor Traceback (most recent call last):
68478 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor.py", line 114, in run_action
68479 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
68480 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/mistral/actions/openstack/base.py", line 130, in run
68481 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor     (self.__class__.__name__, self.client_method_name, str(e))
68482 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor ActionException: ZaqarAction.queue_post failed: Error response from Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too larg      e. Max size 1048576.'


Note You need to log in before you can comment on or make changes to this bug.