Bug 1435262

Summary: Upgrade 10=>11 failed with "JSON body size exceeds maximum allowed size"
Product: Red Hat OpenStack Reporter: Udi Kalifon <ukalifon>
Component: instack-undercloudAssignee: mathieu bultel <mbultel>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 11.0 (Ocata)CC: jcoufal, jschluet, kschinck, lmiccini, mbultel, mburns, mcornea, rhel-osp-director-maint, sathlang, therve
Target Milestone: rcKeywords: Triaged
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: instack-undercloud-6.0.0-5.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1441758 (view as bug list) Environment:
Last Closed: 2017-05-17 20:12:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1441758, 1452410    
Attachments:
Description Flags
Full error message none

Description Udi Kalifon 2017-03-23 13:31:41 UTC 
Created attachment 1265749 [details]
Full error message

Description of problem:
The size of the new templates seems to exceed the allowed maximum. I had an OSP10 deployment on virtual machines with 3 controllers + 2 computes + 1 ceph, with customized roles for keystone and a ServiceApi role.

After upgrading the undercloud successfully I continued to run the upgrade command of the overcloud:

openstack overcloud deploy --templates -r roles_data.yaml --ntp-server clock.redhat.com -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml --no-cleanup -e fernet.yaml -e init-repo.yaml

The upgrade failed right on the beginning with: "ERROR: Request limit exceeded: JSON body size (1119641 bytes) exceeds maximum allowed size (1048576 bytes)".

See attached text file with the full error message.


Version-Release number of selected component (if applicable):
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
openstack-heat-engine-8.0.0-4.el7ost.noarch
python-heat-agent-json-file-1.0.0-0.20170224185834.8e6dbb1.el7ost.noarch
openstack-heat-common-8.0.0-4.el7ost.noarch
openstack-tripleo-heat-templates-6.0.0-0.3.0rc2.el7ost.noarch


Additional info:
There is a bug for it upstream (from 2015!): https://bugs.launchpad.net/heat/+bug/1499379
Comment 1 mathieu bultel 2017-03-24 12:56:23 UTC 
Hi Udi,

Have you been able to reproduce it or is it 100% reproducible with the deploy command above ?
Comment 2 Marius Cornea 2017-03-25 22:10:12 UTC 
I've also hit this issue on a different environment/scenario:

 u'message': u"Failed to run action [action_ex_id=726cebca-9980-4973-a939-0fc0f85e1a0a, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'container': u'overcloud', u'timeout': 240}']\n ERROR: Request limit exceeded: JSON body size (1050160 bytes) exceeds maximum allowed size (1048576 bytes).",

To workaround it I had to adjust max_json_body_size in heat.conf (>1048576). I think we might need to adjust the default value to avoid such conditions. 

I hit this message when running the following deploy command:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-r ~/openstack_deployment/roles/roles_data.yaml \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e $THT/environments/storage-environment.yaml \
-e $THT/environments/manila-cephfsnative-config.yaml \
-e $THT/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml \
-e ~/repo.yaml \
--log-file overcloud_deployment.log &> overcloud_install.log

Environment files:

http://paste.openstack.org/show/604155/
Comment 3 mathieu bultel 2017-03-27 07:08:49 UTC 
Ack thank you Marius.
I'm trying to reproduce it as well and i'll fix it, thank you
Comment 4 Thomas Hervé 2017-03-27 08:51:59 UTC 
Updating the Heat configuration is good workaround for now.

But, if we need to send more than 1M of template data to Heat, there is a design issue: it's not meant to handle that much, and it's going to be hard to manage memory properly. We need to identify what's in that payload.
Comment 5 Keith Schincke 2017-03-27 15:31:34 UTC 
The external patch update resolve this issue for me.

Keith
Comment 6 Sofer Athlan-Guyot 2017-03-28 13:00:09 UTC 
Hi,

I've cherry picked upstream to stable/ocata and updated the gerrit reference.

Thanks Keith for the pointer.
Comment 9 Udi Kalifon 2017-04-24 13:12:25 UTC 
Verified when upgrading from 10.0.z (2017-04-10) to OSP11 (2017-04-20).
Comment 10 errata-xmlrpc 2017-05-17 20:12:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245