Bug 1435262 - Upgrade 10=>11 failed with "JSON body size exceeds maximum allowed size"
Summary: Upgrade 10=>11 failed with "JSON body size exceeds maximum allowed size"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: rc
: 11.0 (Ocata)
Assignee: mathieu bultel
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks: 1441758 1452410
TreeView+ depends on / blocked
 
Reported: 2017-03-23 13:31 UTC by Udi Kalifon
Modified: 2017-05-18 22:20 UTC (History)
10 users (show)

Fixed In Version: instack-undercloud-6.0.0-5.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1441758 (view as bug list)
Environment:
Last Closed: 2017-05-17 20:12:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Full error message (12.14 KB, text/plain)
2017-03-23 13:31 UTC, Udi Kalifon
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 450769 0 None None None 2017-03-28 13:00:09 UTC
Red Hat Product Errata RHEA-2017:1245 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC

Description Udi Kalifon 2017-03-23 13:31:41 UTC
Created attachment 1265749 [details]
Full error message

Description of problem:
The size of the new templates seems to exceed the allowed maximum. I had an OSP10 deployment on virtual machines with 3 controllers + 2 computes + 1 ceph, with customized roles for keystone and a ServiceApi role.

After upgrading the undercloud successfully I continued to run the upgrade command of the overcloud:

openstack overcloud deploy --templates -r roles_data.yaml --ntp-server clock.redhat.com -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml --no-cleanup -e fernet.yaml -e init-repo.yaml

The upgrade failed right on the beginning with: "ERROR: Request limit exceeded: JSON body size (1119641 bytes) exceeds maximum allowed size (1048576 bytes)".

See attached text file with the full error message.


Version-Release number of selected component (if applicable):
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
openstack-heat-engine-8.0.0-4.el7ost.noarch
python-heat-agent-json-file-1.0.0-0.20170224185834.8e6dbb1.el7ost.noarch
openstack-heat-common-8.0.0-4.el7ost.noarch
openstack-tripleo-heat-templates-6.0.0-0.3.0rc2.el7ost.noarch


Additional info:
There is a bug for it upstream (from 2015!): https://bugs.launchpad.net/heat/+bug/1499379

Comment 1 mathieu bultel 2017-03-24 12:56:23 UTC
Hi Udi,

Have you been able to reproduce it or is it 100% reproducible with the deploy command above ?

Comment 2 Marius Cornea 2017-03-25 22:10:12 UTC
I've also hit this issue on a different environment/scenario:

 u'message': u"Failed to run action [action_ex_id=726cebca-9980-4973-a939-0fc0f85e1a0a, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'container': u'overcloud', u'timeout': 240}']\n ERROR: Request limit exceeded: JSON body size (1050160 bytes) exceeds maximum allowed size (1048576 bytes).",

To workaround it I had to adjust max_json_body_size in heat.conf (>1048576). I think we might need to adjust the default value to avoid such conditions. 

I hit this message when running the following deploy command:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-r ~/openstack_deployment/roles/roles_data.yaml \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e $THT/environments/storage-environment.yaml \
-e $THT/environments/manila-cephfsnative-config.yaml \
-e $THT/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml \
-e ~/repo.yaml \
--log-file overcloud_deployment.log &> overcloud_install.log

Environment files:

http://paste.openstack.org/show/604155/

Comment 3 mathieu bultel 2017-03-27 07:08:49 UTC
Ack thank you Marius.
I'm trying to reproduce it as well and i'll fix it, thank you

Comment 4 Thomas Hervé 2017-03-27 08:51:59 UTC
Updating the Heat configuration is good workaround for now.

But, if we need to send more than 1M of template data to Heat, there is a design issue: it's not meant to handle that much, and it's going to be hard to manage memory properly. We need to identify what's in that payload.

Comment 5 Keith Schincke 2017-03-27 15:31:34 UTC
The external patch update resolve this issue for me.

Keith

Comment 6 Sofer Athlan-Guyot 2017-03-28 13:00:09 UTC
Hi,

I've cherry picked upstream to stable/ocata and updated the gerrit reference.

Thanks Keith for the pointer.

Comment 9 Udi Kalifon 2017-04-24 13:12:25 UTC
Verified when upgrading from 10.0.z (2017-04-10) to OSP11 (2017-04-20).

Comment 10 errata-xmlrpc 2017-05-17 20:12:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.