Bug 1712278

Summary: [OSP13 Backport] Message collection size is too large for Zaqar
Product: Red Hat OpenStack Reporter: Adriano Petrich <apetrich>
Component: openstack-tripleo-commonAssignee: Adriano Petrich <apetrich>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: apetrich, apevec, asimonel, beth.white, bshephar, chrisbro, chris.smart, cory.bannister, dvd, jbiao, jhajyahy, jrist, jschluet, jtomasek, lhh, mburns, rrasouli, sasha, shtiwari, slinaber, sputhenp, uemit.seren, ukalifon
Target Milestone: z9Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)Flags: apetrich: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-tripleoclient-9.3.1-1.el7ost openstack-tripleo-common-8.7.1-1.el7ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1664055 Environment:
Last Closed: 2019-11-07 14:01:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1664055    
Bug Blocks:    

Comment 2 Adriano Petrich 2019-09-11 08:36:35 UTC
Everything has been backported upstream

Comment 3 August Simonelli 2019-09-26 04:42:43 UTC
Hello. Is there a target milestone for this backport to land in OSP 13?

Comment 5 Adriano Petrich 2019-10-04 14:41:26 UTC
It depends on QE avaiability but besides that I don't see why not.

Comment 7 Alex McLeod 2019-10-31 11:28:56 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Comment 8 Jad Haj Yahya 2019-10-31 12:30:29 UTC
No regression issues found

Comment 10 errata-xmlrpc 2019-11-07 14:01:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3794

Comment 11 Chris Smart 2020-03-24 23:28:38 UTC
I just hit this again, deploying z11 update at converge step.

~~~
queue_post failed: Error response from Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too large. Max size 1048576.\\n\"}, \"result\": \"Failure caused by error in tasks: send_message\\n\\n  send_message [task_ex_id=2df6a3f6-0aa0-4b0f-9546-f8efb9c36d0b] -> Workflow failed due to message status\\n    [wf_ex_id=5ca8156c-cc5c-4e74-b057-7971cc773f71, idx=0]: Workflow failed due to message status\\n\", \"deployment_status\": \"DEPLOY_FAILED\"}", "input": "{\"run_validations\": false, \"skip_deploy_identifier\": false, \"container\": \"overcloud\", \"queue_name\": \"tripleo\", \"timeout\": 480}", "created_at": "2020-03-24 22:43:39", "project_id": "6829dc9650ef411ca67cd1654d5774df", "id": "6e069679-612c-4bab-a7c6-00a84651c322"}
~~~


Director is at z11 and has:

openstack-tripleo-common-8.7.1-12.el7ost.noarch
python-tripleoclient-9.3.1-7.el7ost.noarch


For now I'm applying my previous work around to increase the sizes even more and trying again.

Comment 12 Chris Smart 2020-03-27 01:45:52 UTC
Just to follow up, increasing the size further worked.

~~~
sudo crudini --set /etc/zaqar/zaqar.conf transport max_messages_post_size 2097152
sudo crudini --set /etc/zaqar/zaqar.conf oslo_messaging_kafka producer_batch_size 32768
sudo crudini --set /etc/mistral/mistral.conf engine execution_field_size_limit_kb 32768
sudo reboot
~~~

Comment 13 Shravan Kumar Tiwari 2020-08-05 10:50:52 UTC
Seems like the issue is not fixed completely.(env RHOSP13z12)
We have hit the issue during scale out of the compute.

'ZaqarAction.queue_post failed: Error response from Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too large. Max size 1048576.: ActionException: ZaqarAction.queue_post failed: Error response f      rom Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too large. Max size 1048576.
68477 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor Traceback (most recent call last):
68478 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor.py", line 114, in run_action
68479 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
68480 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/mistral/actions/openstack/base.py", line 130, in run
68481 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor     (self.__class__.__name__, self.client_method_name, str(e))
68482 2020-07-29 15:19:26.303 2884 ERROR mistral.executors.default_executor ActionException: ZaqarAction.queue_post failed: Error response from Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too larg      e. Max size 1048576.'